Method and device for estimating using self-motion information

Abstract

A method implemented by a processor includes: estimating short-term self-motion information of one or more sensors based on frame images of consecutive frames acquired from the one or more sensors; estimating long-term self-motion information of the one or more sensors based on the frame images; determining attention information based on the short-term self-motion information and the long-term self-motion information; and determining final long-term self-motion information of a current frame in the consecutive frames based on the long-term self-motion information and the attention information.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority from Korean Patent Application No. 10-2019-0038508 filed on April 2, 2019, in the Korean Intellectual Property Office, the disclosure of which is incorporated herein in its entirety for all purposes by reference. Technical Field

[0003] The following description relates to methods and apparatus utilizing ego-motion estimation. Background Art

[0004] Devices such as autonomous vehicles, vehicles using advanced driver assistance systems (ADAS), mobile robots, and drones can identify the location of the corresponding device itself. For example, the device can determine its location using information collected through the Global Navigation Satellite System (GNSS) and the like. However, in obstructed (e.g., shadowed or underground) areas where GNSS signals are not collected, it may be difficult for the device to accurately determine its location.

[0005] To ensure operation in GNSS shadowed areas, autonomous vehicles may track information associated with the device's own relative position via odometers. Summary of the invention

[0006] This summary is provided to introduce some concepts in a simplified form, which are further described in the detailed description below. This summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used as an aid in determining the scope of the claimed subject matter.

[0007] In a general aspect, a method implemented by a processor includes: estimating short-term self-motion information of one or more sensors based on frame images of consecutive frames acquired from the one or more sensors; estimating long-term self-motion information of the one or more sensors based on the frame images; determining attention information based on the short-term self-motion information and the long-term self-motion information; and determining final long-term self-motion information of a current frame in the consecutive frames based on the long-term self-motion information and the attention information.

[0008] Determining the final long-term self-motion information of the current frame may include: using the attention information to correct the long-term self-motion information.

[0009] Determining the final long-term self-motion information may include determining a change in self-motion of the one or more sensors from a long-term reference frame to the current frame, and determining a frame a predetermined number of frames before the current frame as the long-term reference frame.

[0010] The method may include determining final short-term ego-motion information of the current frame based on the determined final long-term ego-motion information.

[0011] Determining the final short-term self-motion information may include: determining the final short-term self-motion information of the current frame based on: the final long-term self-motion information of the frame before the current frame in the continuous frames, the final short-term self-motion information of the long-term reference frame, and the final long-term self-motion information of the current frame.

[0012] A frame preceding the current frame may be directly adjacent to the current frame in the continuous frames.

[0013] Estimating the short-term self-motion information of the one or more sensors may include: in response to receiving a current frame image of the current frame, using a self-motion model to estimate the short-term self-motion information between the current frame and a short-term reference frame; and estimating the short-term self-motion information of the one or more sensors based on the estimated short-term self-motion information between the current frame and the short-term reference frame.

[0014] Estimating the long-term self-motion information of the one or more sensors may include: in response to receiving a current frame image of the current frame, using a self-motion model to estimate the long-term self-motion information between the current frame and a long-term reference frame; and estimating the long-term self-motion information of the one or more sensors based on the estimated long-term self-motion information between the current frame and the long-term reference frame.

[0015] Estimating the short-term self-motion information may include: using a short-term self-motion model to estimate the short-term self-motion information based on a pair of consecutive frame images in the frame image, estimating the long-term self-motion information may include: using a long-term self-motion model to estimate the long-term self-motion information based on a target frame image in the current frame and a reference frame image in the frame image, and the short-term self-motion model may have a different configuration from the long-term self-motion model, at least having different training parameters respectively.

[0016] Determining the attention information may include: using an attention model to extract the attention information from the short-term self-motion information and the long-term self-motion information.

[0017] Extracting the attention information may include: extracting attention elements from the short-term self-motion information and the long-term self-motion information; extracting attention weights from the short-term self-motion information; and generating the attention information based on the attention elements and the attention weights.

[0018] The method may include: in response to the number of collected frame images from the initial frame to the current frame being less than a determined frame number, determining the short-term self-motion information as final short-term self-motion information.

[0019] The method may include: using an ego-motion model to extract short-term ego-motion features of the one or more sensors from a current frame image of the current frame and a frame image before the current frame image in the frame images; using an image deformation model to generate a restored frame image of the current frame by deforming the previous frame image based on the short-term ego-motion features; using the ego-motion model to calculate correction information based on the current frame image and the restored frame image; and generating image-based short-term ego-motion information by applying the correction information to the short-term ego-motion information of the current frame.

[0020] The correction information may correspond to a small posture error between the restored frame image and the current frame image.

[0021] The method may include determining final short-term self-motion information based on attention-based short-term self-motion information calculated from the final long-term self-motion information and the image-based short-term self-motion information.

[0022] The method may include: using a self-motion model to extract short-term self-motion features of the one or more sensors from a current frame image of the current frame and a frame image before the current frame image in the frame images; generating a depth image of a frame before the current frame based on the previous frame image; generating a depth image corresponding to the current frame by deforming the depth image of the previous frame based on the short-term self-motion features; converting the depth image corresponding to the current frame into a restored frame image; using the self-motion model to calculate correction information based on the current frame image and the restored frame image; and generating depth-based short-term self-motion information by applying the correction information to the short-term self-motion information of the current frame.

[0023] The method may include determining final short-term self-motion information based on attention-based short-term self-motion information calculated from the final long-term self-motion information and the depth-based short-term self-motion information.

[0024] The method may include: tracking a travel path of a device equipped with the one or more sensors based on at least one of the final long-term self-motion information and the final short-term self-motion information calculated based on the final long-term self-motion information; and outputting the tracked travel path of the device.

[0025] The method may include: using a depth model to generate temporary depth information of a frame before the current frame based on a training image; using a self-motion model and an attention model to generate temporary long-term self-motion information based on the training image; calculating temporary short-term self-motion information based on the temporary long-term self-motion information; generating a warped image of the current frame based on the temporary short-term self-motion information and the temporary depth information of the previous frame; and training any one of the following models or any combination of any two or more of the following models based on a loss calculated based on the warped image and the current frame image in the training image: the self-motion model, the attention model, and the depth model.

[0026] Generating the distorted image may include: generating a three-dimensional (3D) coordinate image corresponding to the previous frame based on the temporary depth information; restoring the 3D coordinate image corresponding to the current frame by converting the 3D coordinate image corresponding to the previous frame using the temporary short-term self-motion information calculated based on the temporary long-term self-motion information; and generating the distorted image by two-dimensionally projecting the 3D coordinate image corresponding to the current frame, so that the distorted image is two-dimensionally distorted.

[0027] A non-transitory computer-readable storage medium may store instructions that, when executed by one or more processors, cause the one or more processors to perform the method.

[0028] In another general aspect, a method implemented by a processor includes: using a self-motion model to extract short-term self-motion features from a current frame image and a previous frame image; using a deformation model to generate a restored frame image corresponding to a current frame based on the extracted short-term self-motion features according to the previous frame image and at least one of the depth images generated from the previous frame image; calculating correction information based on the restored frame image and the current frame image; and using the self-motion model to calculate final short-term self-motion information by applying the correction information to the estimated short-term self-motion information of the current frame.

[0029] In another general aspect, a device may include: one or more sensors configured to acquire frame images of consecutive frames; one or more processors configured to: estimate short-term self-motion information of the one or more sensors; estimate long-term self-motion information of the one or more sensors based on the frame images; calculate attention information based on the short-term self-motion information and the long-term self-motion information; and determine final long-term self-motion information of the current frame by correcting the long-term self-motion information based on the attention information.

[0030] The one or more sensors may include one or more image sensors, and the one or more processors may be configured to determine a travel path of the one or more image sensors based on the determined final long-term ego-motion information.

[0031] In another general aspect, a method implemented by a processor includes: estimating short-term self-motion information of a device based on continuous frame images, the continuous frame images including a current frame image to a set number of frame images before the current frame image; estimating long-term self-motion information of the device based on the current frame image and previous frame images, wherein the previous frame images are frame images before the set number from the current frame image; determining attention information based on the short-term self-motion information and the long-term self-motion information; and determining final long-term self-motion information of the device by correcting the long-term self-motion information based on the attention information.

[0032] Other features and aspects will become apparent from the following detailed description, drawings, and claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 An example of an ego-motion model is shown.

[0034] Figure 2 An example of a method of estimating ego-motion information is shown.

[0035] Figure 3 An example of a process for estimating ego-motion information is shown.

[0036] Figure 4 An example of an operation of estimating self-motion information using a self-motion model is shown.

[0037] Figure 5 An example of attention operation using an attention model is shown.

[0038] Figure 6 An example of a method for training an ego-motion model and an attention model is shown.

[0039] Figure 7 An example of the process of training the ego-motion model and the attention model is shown.

[0040] Figure 8 An example of a method of estimating ego-motion information is shown.

[0041] Fig. 9 and Fig.10 An example of a process for estimating ego-motion information is shown.

[0042] Fig.11 An example of a training method for a model is shown.

[0043] Fig.12 An example of the process of training a model is shown.

[0044] Fig.13 An example of a process for estimating ego-motion information is shown.

[0045] Fig.14 and Fig.15 An example of an ego-motion information estimation device is shown.

[0046] Fig.16 An example of an ego-motion information estimation device mounted on a vehicle is shown.

[0047] Throughout the drawings and detailed description, unless otherwise described or provided, the same reference numerals should be understood to refer to the same elements, features, and structures. The drawings may not be drawn to scale, and the relative sizes, proportions, and depictions of elements in the drawings may be exaggerated for clarity, illustration, and convenience. DETAILED DESCRIPTION

[0048] The following detailed description is provided to help the reader obtain a comprehensive understanding of the methods, devices and / or systems described herein. However, after understanding the disclosure of the present application, various changes, modifications and equivalents of the methods, devices and / or systems described herein will be apparent. For example, the order of operations described herein is merely an example and is not limited to those order of operations set forth herein, but can be changed significantly after understanding the disclosure of the present application, except for operations that must be performed in a certain order. In addition, for greater clarity and brevity, the description of features known in the art may be omitted.

[0049] The features described herein may be implemented in different forms and are not to be construed as being limited to the examples described herein. Rather, the examples described herein are provided merely to illustrate some of the many possible ways to implement the methods, devices and / or systems described herein, which will be apparent after understanding the disclosure of the present application.

[0050] Although terms such as "first", "second", "third" and the like may be used herein to describe various components, assemblies, regions, layers or parts, these components, assemblies, regions, layers or parts should not be limited by these terms. These terms are only used to distinguish one component, component, region, layer or part from another component, component, region, layer or part. Therefore, without departing from the teachings of the examples described herein, the first component, component, region, layer or part mentioned in the examples may also be referred to as the second component, component, region, layer or part.

[0051] The terms used herein are only used to describe various examples and are not used to limit the present disclosure. Unless the context clearly indicates otherwise, the articles "a", "an" and "the" are also intended to include plural forms. The terms "include", "comprise" and "have" indicate the presence of the described features, numbers, operations, components, elements and / or combinations thereof, but do not exclude the presence or addition of one or more other features, numbers, operations, components, elements and / or combinations thereof.

[0052] Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as those commonly understood by those skilled in the art to which the present disclosure belongs after understanding the disclosure of the present application. Terms (such as those defined in commonly used dictionaries) should be interpreted as having a meaning consistent with that in the context of the relevant technology and the disclosure of the present application, and should not be interpreted as an ideal or overly formal meaning, unless explicitly defined as such herein.

[0053] Figure 1 An example of an ego-motion model is shown.

[0054] As used herein, "model" may refer to a computational model that is actually executed by any one processor or any combination of processors set forth in the present disclosure. The self-motion information estimation device may estimate the self-motion information according to the input image based on the self-motion model and the attention model. The input image may include a frame image corresponding to a plurality of frames. A frame may refer to a unit frame for dividing time. Individual frame images of the input image may be color images. Each pixel of the color image may represent a color value. The type of frame image is not limited to a color image, and therefore images having other formats may also be used.

[0055] The ego-motion information may be information associated with the ego-motion of the sensor, and may indicate the ego-motion change of the sensor from the reference frame to the target frame. For example, the ego-motion information may include the rotation change and translation change of the sensor from the reference frame to the target frame. In other words, the ego-motion information may indicate the degree of rotation and translation of the sensor in the target frame based on the posture and position of the sensor in the reference frame. When the sensor is mounted on a predetermined device (e.g., a vehicle and a head-mounted display (HMD)), the ego-motion information may indicate the ego-motion change of the device. In the following, the ego-motion information may be described as information associated with the sensor, but the embodiment is not limited thereto. Instead of the ego-motion information of the sensor, the ego-motion information may additionally or alternatively be information of the device on which the sensor is mounted.

[0056] The ego-motion information may include long-term ego-motion information and short-term ego-motion information. The long-term ego-motion information may indicate the change in ego-motion of the sensor between a current frame (e.g., the kth frame) and a long-term reference frame (e.g., the knth frame). Here, n is an integer greater than or equal to 1, and k is an integer greater than or equal to n. Examples where k is less than n will be described later in this disclosure. The short-term ego-motion information may indicate the change in ego-motion of the sensor between a current frame (e.g., the kth frame) and a short-term reference frame (e.g., the k-1th frame). However, the short-term reference frame of the short-term ego-motion information is not limited to the previous frame (e.g., the k-1th frame). The number of frames between the short-term reference frame and the current frame may be less than n, which is the number of frames between the long-term reference frame and the current frame. In the following, the long-term ego-motion information is also referred to as the long-term motion information, and the short-term ego-motion information is also referred to as the short-term motion information.

[0057] The target frame is a frame corresponding to the target estimated by the self-motion information. In the following description, the target frame may be a current frame corresponding to the current time point. The reference frame is a frame corresponding to the reference of the target frame and may be a frame before the target frame in time.

[0058] The ego-motion model can be used to estimate initial ego-motion information corresponding to a target frame based on frame images with various frame intervals in the input image. The frame interval includes, for example, a long-term frame interval and a short-term frame interval. The ego-motion model can be trained, for example, to extract feature data from the frame image and output initial ego-motion information based on the extracted feature data. The feature data can be, for example, data obtained by abstracting the image and can be represented, for example, in a vector form or other multidimensional form. The feature data can be data extracted by convolution filtering, which is based on the following reference Figure 1 The described neural network 100 may be implemented using one or more convolutional layers.

[0059] The attention model is a model for estimating attention information (for example, estimating attention information based on initial self-motion information, which is estimated based on the self-motion model). The attention information may be information for indicating or emphasizing data to be paid attention to for a predetermined purpose among data extracted by the layer. In the present disclosure, for example, the attention information may be information extracted from feature data extracted from the initial self-motion information for correcting long-term self-motion information. The attention model may also include information related to the reference Figure 1 The structure of the neural network 100 described above corresponds to that of the neural network 100. In the attention model, this convolution filter can be excluded. Figure 5 An example of using an attention model to extract attention information is described in detail.

[0060] Both the ego-motion model and the attention model may be based on, for example, a machine learning structure and include a corresponding neural network 100 .

[0061] Therefore, the neural network 100 represents various deep neural network (DNN) embodiments. DNNs may include fully connected networks, deep convolutional networks, and recurrent neural networks. The neural network 100 may perform object classification, object recognition, speech recognition, and / or image recognition by mapping input data and output data having a nonlinear relationship to each other based on deep learning. Deep learning is a machine learning technique and may be used to perform image recognition or speech recognition based on large data sets, and may map input data and output data through supervised learning or unsupervised learning.

[0062] refer to Figure 1 , the neural network 100 may include an input layer 110, a hidden layer 120, and an output layer 130. The input layer 110, the hidden layer 120, and the output layer 130 may each include a plurality of artificial nodes.

[0063] For ease of description, Figure 1 Three hidden layers 120 are shown. However, the hidden layers 120 may include various numbers of layers, such as four or more hidden layers 120. In addition, Figure 1 The neural network 100 including a separate input layer for receiving input data is shown. However, in another example, the input data may be directly input into the hidden layer 120 (e.g., into the first layer of the hidden layer 120). In the neural network 100, the nodes of each layer except the output layer 130 are connected to the nodes of the subsequent layer through links to transmit output signals. The number of links may correspond to the number of nodes included in the subsequent layer.

[0064] The activation function outputs of the artificial nodes of the previous hidden layer 120 associated with the weighted inputs of the artificial nodes of the previous hidden layer 120 can be respectively input into the artificial nodes of the subsequent current hidden layer 120, wherein the activation outputs of the artificial nodes of the previous hidden layer 120 of these inputs can be weighted by the corresponding artificial nodes of the previous hidden layer 120, or can be weighted before the artificial nodes of the previous hidden layer 120, for example, as the corresponding weighted connection or link between the previous hidden layer 120 or the input layer 110 and the previous hidden layer 120. Therefore, for each artificial node of the previous hidden layer 120, a weighted result can be obtained by multiplying each corresponding input of the artificial node (from the previous hidden layer 120 or the input layer 110) by a corresponding weight. These weights are called parameters of the neural network 100, for example, training parameters of the neural network 100 depending on the training objectives of the neural network 100. The activation functions implemented respectively may include any one or any combination of sigmoid, hyperbolic tangent (tanh), and rectified linear unit (ReLU), for example, thereby forming nonlinearity in the neural network 100 through the activation function. Therefore, in the example, the weighted inputs of the artificial nodes included in the previous hidden layer 120 are input into the artificial nodes of the current hidden layer 120, wherein such respectively weighted inputs of the artificial nodes included in the previous hidden layer 120 are respectively input into each of the multiple or all artificial nodes of the current hidden layer 120. Figure 1 In the example of , when the current hidden layer 120 is the final hierarchical hidden layer 120, the activation results of the respective artificial nodes of the final hidden layer 120 are similarly weighted for each connection between the artificial nodes of the final hidden layer 120 and each artificial node of the output layer 130, respectively. Here, a number of weights or parameters of the neural network 120 may be personalized for each connection or link between the artificial nodes of the hidden layer and the output layer, for example, such as the following case discussed further below: many weights or parameters have been preset or predetermined through specific training of the neural network 100 for a specific goal.

[0065] Therefore, the self-motion information estimation device can input the output of the previous hidden node included in the previous hidden layer to each subsequent hidden layer (for example, through an edge with such a weighted connection or "connection weight"), and generate the output of the hidden node included in the hidden layer based on the value obtained by applying the corresponding connection weight and the corresponding activation function of the corresponding node to the output of the previous hidden node. In the neural processor (NPU) example, such output activation can be in the form of an electrical pulse or spike, or in the form of a specific timing pulse or spike or a series of pulses or spikes, wherein each hidden node can be configured to trigger or generate an output pulse or spike to transmit the output pulse or spike along the electrical connection to the next hidden node of the subsequent hidden layer or output layer (and / or to itself or another layer along other connections). For example, each hidden node can be configured to propagate the result of the corresponding activation function when it exceeds a set threshold of the current hidden node, for example, wherein the set threshold can depend on the input activation from the respective hidden nodes of the previous hidden layer to the current hidden node and the training connection weights of the corresponding connections from the respective hidden nodes of such a previous layer to the current hidden node. In this example, the node can also be considered to maintain a deactivated state without triggering or propagating a signal to the next node until a predetermined activation threshold strength is reached by an input vector, which is, for example, an input feature vector generated as an activation from one or more previous layers and / or other layers, or an input feature vector generated by the node itself at a previous time. The ego-motion model and the attention model can also be connected, for example, via an output edge of the ego-motion model and an input edge of the attention model. The connection between the ego-motion model and the attention model will refer to Figure 5 Give a description.

[0066] In an example, the self-motion model may include at least one convolution layer as one or more hidden layers 120. The self-motion information estimation device may perform convolution filtering on the frame image input to the self-motion model based on the convolution layer of the self-motion model. The self-motion information estimation device may perform convolution filtering by applying a convolution operation between the mask and the frame image. The shape, size, and structure of the kernel mask may vary based on the training performed to generate the corresponding training self-motion model, noting that there are examples of various shapes, sizes, and structures of the kernel mask. The self-motion information estimation device may perform convolution filtering using the self-motion model to extract feature data from the frame image. Thereafter, the self-motion information estimation device may sequentially propagate the extracted feature data in the remaining layers of the self-motion model, and ultimately generate initial self-motion information (e.g., long-term self-motion information and short-term self-motion information) in the output layer 130 of the self-motion model.

[0067] When the width and depth of the neural network 100 are large enough, the neural network 100 has a capacity sufficient to implement a predetermined function. The neural network 100 can implement, for example, the following functions: outputting the self-motion information of the sensor according to the frame image captured by the sensor. For example, when a neural network having the architecture of the neural network 100 being trained has learned a sufficient amount of training data through an appropriate training process, the resulting trained neural network 100 can achieve optimal estimation performance. For example, the training may include: adjusting the parameters of the neural network being trained until the accuracy and / or inaccuracy meets the predetermined optimal performance. For reference, an unsupervised scheme may be used to train the self-motion model and the attention model, and the reference Figure 6 and Figure 7 It is also noted that in this document, the term "may" is used with respect to examples or embodiments (e.g., with respect to what an example or embodiment may include or implement, or with respect to how an example or embodiment may be configured), which means that there is at least one example or embodiment that includes, implements and / or configures such features, but all examples and embodiments are not limited thereto.

[0068] Figure 2 An example of a method of estimating ego-motion information is shown. Figure 3 An example of a process for estimating ego-motion information is shown.

[0069] Figure 2 and Figure 3 An operation of the ego-motion information estimating device estimating ego-motion information corresponding to a current frame (eg, k-th frame) is shown.

[0070] The self-motion information estimation device obtains a frame image of the current frame. When the sensor is installed toward the front of the vehicle, the sensor can generate a frame image including a scene corresponding to the front field of view of the vehicle. Figure 3 In the example of FIG. 1 , the frame image 301 includes the 0th frame image I 0 To the kth frame image I of the current frame k Here, a case is described in which n is an integer greater than or equal to 1, and k is an integer greater than n (k>n). A case in which k is less than or equal to n will be described later.

[0071] For reference, until the kth frame, the self-motion information estimation device may accumulate frame images 301 corresponding to frames before the kth frame (e.g., the 0th frame to the k-1th frame). As an example, the self-motion information estimation device may accumulate frame images 301 corresponding to the kth frame and n frames before the kth frame (e.g., the knth frame to the kth frame).

[0072] In operation 210, the self-motion information estimation device estimates short-term self-motion information. The self-motion information estimation device may estimate the short-term self-motion information of the sensor that has acquired the frame image 301 of the continuous frame based on the frame image 301. For example, the self-motion information estimation device may estimate the short-term self-motion information based on the frame image 301 based on the self-motion model 310. The self-motion information estimation device estimates the self-motion change of the sensor from the short-term reference frame to the current frame. The self-motion information estimation device estimates the short-term self-motion information based on the frame image corresponding to the short-term reference frame and the frame image corresponding to the current frame based on the self-motion model 310. For example, the self-motion information estimation device may estimate the initial short-term self-motion information SP corresponding to the current frame based on the frame image 301 corresponding to the kth frame as the current frame and the k-1th frame as the previous frame of the current frame based on the self-motion model 310. k，k-1 .

[0073] The ego-motion information estimation device may accumulate the initial short-term ego-motion information estimated up to the previous frame. For example, the ego-motion information estimation device may accumulate the short-term ego-motion information SP corresponding to the k-n+1th frame. k-n+1，k-n , the short-term self-motion information SP corresponding to the k-n+2th frame k-n+2，k-n+1 , until the short-term self-motion information SP corresponding to the k-1th frame k-1，k-2 , and stores the accumulated information. SP i，j is the short-term self-motion information corresponding to the i-th frame, and indicates the short-term self-motion change from the j-th frame to the i-th frame. Here, i and j are integers greater than or equal to 0, and i is greater than j.

[0074] In operation 220, the self-motion information estimation device estimates the long-term self-motion information. The self-motion information estimation device may estimate the long-term self-motion information of the sensor based on the frame image 301. For example, the self-motion information estimation device may estimate the short-term self-motion information based on the frame image 301 based on the self-motion model 310, and the self-motion information estimation device may estimate the self-motion change of the sensor from the long-term reference frame to the current frame. The self-motion information estimation device may estimate the long-term self-motion information based on the frame image corresponding to the long-term reference frame and the frame image corresponding to the current frame based on the self-motion model 310. For example, the self-motion information estimation device estimates the initial long-term self-motion information LP corresponding to the current frame based on the frame image 301 corresponding to the k-th frame and the kn-th frame as the long-term reference frame based on the self-motion model 310. k，k-n LP i，j is the long-term self-motion information corresponding to the i-th frame, and indicates the long-term self-motion change from the j-th frame to the i-th frame.

[0075] Although the structure in which the self-motion model 310 is configured to output self-motion information corresponding to a target frame (e.g., a current frame from two frame images 301) is described with reference to operations 210 and 220, the embodiment is not limited thereto. According to an example, the self-motion model 310 may be configured to output self-motion information corresponding to a target frame from a plurality of frame images 301.

[0076] In operation 230, the self-motion information estimation device calculates attention information based on the short-term self-motion information and the long-term self-motion information. For example, the self-motion information estimation device may calculate attention information based on the short-term self-motion information and the long-term self-motion information based on the attention model 320.

[0077] In operation 240, the self-motion information estimation device corrects the long-term self-motion information based on the attention information. The self-motion information estimation device applies the attention information to the long-term self-motion information, thereby determining the final long-term self-motion information LP′ of the current frame. k，k-n 309. Correction 330 is to the initial long-term self-motion information LP corresponding to the current frame k，k-n Apply the operation of attention information. For example, when the initial long-term self-motion information LP k，k-n When the attention information is calculated as matrix data of the same dimension, the correction 330 can be performed by matrix product calculation or addition operation to the initial long-term self-motion information LP k，k-n The operation of applying attention information. However, the correction 330 is not limited thereto. The correction 330 may be based on the initial long-term self-motion information LP k，k-n The self-motion information estimation device may determine the final self-motion change of the sensor from the long-term reference frame to the current frame by correcting 330. Here, the frame n frames before the current frame may be determined as the long-term reference frame, where n is a predetermined number of frames and is an integer greater than or equal to 1.

[0078] Thereafter, the self-motion information estimation device may determine the final short-term self-motion information of the current frame based on the determined final long-term self-motion information 309. The self-motion information estimation device may determine the final short-term self-motion information of the current frame based on the determined final long-term self-motion information LP′ determined in the previous frame. k-1，k-n-1 , the final short-term self-motion information SP' corresponding to the long-term reference frame (e.g., the kn-th frame) k-n，k-n-1 , and the final long-term self-motion information LP' of the current frame k，k-n 309, determine the final short-term self-motion information SP' of the current frame k，k-1 For example, the final short-term self-motion information SP' k，k-1 It can be expressed as shown in Equation 3.

[0079] Equation 1:

[0080]

[0081] Equation 2:

[0082] P k，k-n =P k-n+1，k-n P k-n+2，k-n+1 …P k-2，k-3 P k-1，k-2 P k，k-1

[0083] =(P k-1，k-n ) k，k-1

[0084] Equation 3:

[0085] SP′ k，k- 1=P k，k-1 =(P k-1，k-n ) -1 P k，k-n

[0086] ={(P k-n，k-n-1 ) -1 P k-1，k-n-1} -1 P k，k-n

[0087] ={(SP′ k-n，k-n-1 ) -1 LP′ k-1，k-n-1} -1 LP′ k，k-n

[0088] Equation 1 may represent a data format of predetermined self-motion information. For example, R(3×3) is a 3×3 dimensional matrix and may represent a rotation change of the sensor on three axes in a three-dimensional (3D) space. In addition, T(3×1) is a 3×1 dimensional matrix and may represent a translation change. Therefore, the self-motion information may be represented by a self-motion matrix (e.g., a matrix in SE3 format, which includes a rotation matrix and a translation matrix). In the following description, for ease of description and calculation, the self-motion information is interpreted as a matrix, and other types of data may also be used.

[0089] Equation 2 can express the ego-motion information P from the knth frame to the kth frame as k，k-n According to Equation 2, the self-motion matrix corresponding to the previous frame is merged with the self-motion matrix corresponding to the subsequent frame of the previous frame by applying the self-motion matrix corresponding to the subsequent frame through matrix product calculation. Therefore, as shown in Equation 2, the self-motion information P from the kn-th frame to the k-th frame is k，k-nThe self-motion information P from the knth frame to the k-1th frame can be k-1，k-n and the self-motion information P from the k-1th frame to the kth frame k，k-1 It can be represented by the matrix product of .

[0090] When both sides of Equation 2 are multiplied by the self-motion information P k-1，k-n The final short-term self-motion information SP' from the k-1th frame to the kth frame is k，k-1 It can be calculated as shown in Equation 3. The self-motion information estimation device has calculated the final short-term self-motion information SP' corresponding to the kn-th frame in the k-1-th frame which is the previous frame. k-n，k-n-1 , the final long-term self-motion information LP' corresponding to the k-1th frame has been calculated k-1，k-n-1 , and the final long-term self-motion information LP′ corresponding to the k-th frame has been calculated in operation 240 of the k-th frame as the current frame. k，k-n 309. Therefore, the self-motion information estimation device can calculate the final short-term self-motion information SP' of the current frame according to Equation 3 k，k-1 .

[0091] The case of k>n has been described. In the following description, the case of k<n and the case of k=n are explained. Table 1 shows information calculated in each operation based on k.

[0092] [Table 1]

[0093]

[0094] In the kth frame (k is greater than n), the final long-term self-motion information LP' corresponding to the k-1th frame is accumulated in the previous frame k-1，k-n-1 In the case of k=n or k<n, the corresponding information is not calculated.

[0095] Therefore, in the kth frame (k is less than n), the self-motion information estimation device calculates the short-term self-motion information SP in response to the number of frame images 301 collected from the initial frame to the current frame being less than n. k，k-1 The final short-term self-motion information is determined as follows. While k is less than n, the self-motion information estimation device repeatedly estimates the short-term self-motion information SP corresponding to the k-th frame in operation 210. k，k-1 .

[0096] In the kth frame (k is equal to n), the self-motion information estimation device initially calculates the final long-term self-motion information 309 corresponding to the nth frame, for example, LP′ n.0 Since the long-term ego-motion information corresponding to the n-1th frame is not estimated, the ego-motion information estimation device uses the short-term ego-motion information SP corresponding to the first frame to the n-1th frame1，0 ,……,SP n-1，n-2 The matrix product of is used to replace the self-motion information P corresponding to the 0th frame to the n-1th frame n-1，0 The self-motion information estimation device calculates the final short-term self-motion information SP' corresponding to the n-th frame in the n-th frame according to Equation 4, for example. n，n-1 .

[0097] Equation 4:

[0098] SP′ n，n-1 =(P n-1，0 ) -1 P n，0

[0099] =(SP 1，0 …SP n-1，n-2 ) -1 LP′ n，0

[0100] The self-motion information estimation device may use one of the final long-term self-motion information and the final short-term self-motion information or both to determine the posture and position of the device in each frame. The position of the device may be a position relative to a reference position, but is not limited thereto. For example, the position of the device may also be an absolute position. The posture of the device may be a posture relative to a reference posture, but is not limited thereto. For example, the posture of the device may also be an absolute posture. The self-motion information estimation device may set a predetermined geographical location or physical location as an origin as a reference position and set a reference posture. The self-motion information estimation device may determine the posture and position of the device in each frame by cumulatively applying the final short-term self-motion information and / or the final long-term self-motion information to the reference posture and reference position. For example, the self-motion information estimation device may determine the posture and position of the current frame by applying the final short-term self-motion information determined in the current frame to the posture and position of the previous frame. Similarly, the self-motion information estimation device may determine the posture and position of the next frame by applying the final short-term self-motion information determined in the next frame to the posture and position of the current frame. In this way, the self-motion information estimation device can repeatedly apply the final short-term self-motion information determined in the current frame to the posture and position of the previous frame and accumulate the results, thereby tracking the posture and position determined in each frame. When only short-term self-motion information is used to track the posture and position of the device, the accumulated error in the tracked posture and position may increase over time. The self-motion information estimation device of one or more embodiments can minimize the accumulated error by applying long-term self-motion information.

[0101] Figure 4 An example of an operation of estimating self-motion information using a self-motion model is shown.

[0102] The self-motion information estimation device can calculate the short-term self-motion information (eg, SP k-n+1，k-n To SP k，k-1 ) and long-term self-motion information (e.g., LP k，k-n ).

[0103] In response to receiving the current frame image of the current frame, the ego-motion information estimating device may estimate the short-term ego-motion information SP between the current frame (e.g., the kth frame) and the short-term reference frame (e.g., the k-1th frame) based on the ego-motion model. k，k-1 . The ego-motion information estimation device may accumulate the short-term ego-motion information calculated for each frame and store the accumulated short-term ego-motion information. The ego-motion information estimation device may archive the short-term ego-motion information of n frames before the current frame (e.g., kn-th frame to k-th frame). The ego-motion information estimation device may archive the information associated with these n frames (e.g., short-term ego-motion information and long-term ego-motion information) and also remove the information associated with the frames before these n frames.

[0104] In response to receiving the current frame image, the self-motion information estimation device can estimate the long-term self-motion information LP between the current frame (e.g., the kth frame) and the long-term reference frame (e.g., the knth frame) based on the self-motion model. k，k-n .

[0105] The short-term self-motion information and the long-term self-motion information can be estimated based on the same self-motion model. In this example, the self-motion information estimation device can sequentially execute the self-motion model with a fixed structure and fixed parameters for each frame to calculate the self-motion information of the corresponding frame according to the frame image 301.

[0106] However, the embodiment is not limited thereto. Figure 4 As shown, in the example, the self-motion information estimation device includes n+1 self-motion models: PoseNet-1 to PoseNet-n and PoseNet-L. Each of the self-motion models PoseNet-1 to PoseNet-n and PoseNet-L can be used to calculate short-term self-motion information and long-term self-motion information corresponding to each frame. In this example, the self-motion models PoseNet-1 to PoseNet-n and PoseNet-L may include different structures and different trained parameters (e.g., connection weights).

[0107] In addition, the self-motion information estimation device can estimate the short-term self-motion information based on the continuous frame image pair in the frame image 301 based on the short-term self-motion model 411. The continuous frame image pair may be two frame images separated from each other in time. The self-motion information estimation device can estimate the long-term self-motion information based on the target frame image and the reference frame image in the frame image 301 based on the long-term self-motion model 412. The parameters of the short-term self-motion model 411 may be different from the parameters of the long-term self-motion model 412.

[0108] Figure 5 An example of attention operation using an attention model is shown.

[0109] The self-motion information estimation device can extract attention information from the short-term self-motion information and the long-term self-motion information based on the attention model 320. For example, the self-motion information estimation device can extract attention elements from the short-term self-motion information and the long-term self-motion information. The attention element can be data in which a higher weight is assigned to key data similar to the query data Q in the key data K, where the query data Q corresponds to the long-term self-motion information and the key data K corresponds to the short-term self-motion information. The self-motion information estimation device can extract attention weights from the short-term self-motion information. The self-motion information estimation device can generate attention information based on the attention element and the attention weight V. The attention model 320 may include pooling, scaling and / or matrix product calculations without convolution filtering.

[0110] For example, Figure 5 As shown, the output of the ego-motion model is connected to the input of the attention model 320. The ego-motion information estimation device can extract key data K from the short-term ego-motion information output by the ego-motion model based on the attention model 320, extract query data Q from the long-term ego-motion information, sequentially apply matrix product calculation 521, scaling operation 522 and softmax operation 523, and extract attention elements. The attention element indicates the information to be paid attention to in the key data and the query data. The key data K may be data extracted from the short-term ego-motion information by pooling 529. The query data Q may be data extracted from the long-term ego-motion information. The ego-motion information estimation device can calculate the attention information by applying the matrix product calculation 524 to the attention weight V and the attention element extracted from the short-term ego-motion information by pooling 529, and applying a linear function 525 to the result thereof. The attention information may be a weighted value in which the proportion of information similar to the query data Q in the key data K is increased.

[0111] Although Figure 5It is shown that the self-motion information is input to the attention model 320, but the embodiment is not limited thereto. For example, feature data (ie, self-motion features) before being output as self-motion information may be input to the attention model 320.

[0112] As described above, the ego-motion information estimation device may determine the final long-term ego-motion information 309 by correction 330 , wherein the correction 330 is performed by applying the attention information to the long-term ego-motion information.

[0113] For reference, in the present disclosure, the following model (e.g., neural network) is described as an example: in this model, the number of nodes of the layer decreases from the input layer toward the output layer (e.g. Figure 5 Long-term self-motion model 511). However, the embodiment is not limited to this.

[0114] Figure 6 An example of a method for training an ego-motion model and an attention model is shown. Figure 7 An example of the process of training the ego-motion model and the attention model is shown.

[0115] In operation 610, the self-motion information estimation apparatus extracts n sequences of frame images from the training image 701. The training image may be an image separate from the input image and may not include the input image, but is not limited thereto. Since the training is an unsupervised learning method performed even when there is no ground truth for a given image, the input image may be used as the training image.

[0116] In operation 620, the self-motion information estimation device generates temporary depth information 749. The self-motion information estimation device may generate temporary depth information 749 of the previous frame based on the depth model 740 and the training image 701. The self-motion information estimation device may generate temporary depth information 749 of the previous frame based on the depth model 740 and the training image 701 corresponding to the k-1th frame (for example, based on I as a training stereo image). k-1，l and I k-1，r ) to generate temporary depth information 749 corresponding to the k-1th frame. The depth model 740 is a model configured to generate a depth image from a stereo image (eg, two color images).

[0117] The depth information may be information associated with the depth of a point included in the field of view (FOV) of the sensor, and may include a depth image. The depth image may be an image showing the distance from the sensor to the point. For example, the pixel value of each pixel in the depth image may be the value of the distance from the sensor to the physical position of the point indicated by the corresponding pixel. The depth image may be an image obtained by converting a stereoscopic image (e.g., a left image and a right image acquired by a pair of camera sensors), but is not limited thereto. For example, the depth image may be an image acquired by a depth sensor.

[0118] The training stereo image is a stereo image. The first training image I corresponding to the k-1th frame k-1，1 is the left image, and the second training image I k-1，r The left image may be an image acquired by a camera sensor disposed on the left side based on the center of the device (e.g., a vehicle between a pair of camera sensors). The right image may be an image acquired by a camera sensor disposed on the right side based on the center of the device. However, stereoscopic images are not limited to the above-mentioned left and right images. A stereoscopic image may be two images acquired by a pair of camera sensors spaced apart from each other.

[0119] The depth model 740 may be Figure 6 The training model is completed before the training process of , and can be a neural network. However, the deep model 740 is not limited thereto, and the deep model 740 can also be referenced Figure 7 Training is performed during the described training process.

[0120] In operation 630, the self-motion information estimation device generates temporary long-term self-motion information 709 based on the self-motion model 310 and the attention model 320, and calculates temporary short-term self-motion information 708. The self-motion information estimation device may generate temporary long-term self-motion information 709 based on the training image 701 based on the self-motion model 310 and the attention model 320, and may calculate temporary short-term self-motion information 708 based on the temporary long-term self-motion information 709. The calculation of the temporary short-term self-motion information 708 may be as described above with reference to Figures 2 to 5 described.

[0121] In operation 640, the self-motion information estimation device generates a warped image I according to the temporary short-term self-motion information 708 and the temporary depth information 749. w '(k). The self-motion information estimation device can generate a distorted image I of the current frame based on the temporary short-term self-motion information 708 and the temporary depth information 749 of the previous frame. w '(k).

[0122] In operation 640, the self-motion information estimation apparatus generates a warped image I by warping 750. w '(k). The self-motion information estimation device can generate a 3D coordinate image Temp corresponding to the previous frame according to the temporary depth information 749 by 3D coordinate generation 751. k-1 The self-motion information estimation device can perform 3D rotation 752 by using the temporary short-term self-motion information 708 calculated based on the temporary long-term self-motion information 709 to estimate the 3D coordinate image Temp corresponding to the previous frame. k-1 Convert to restore the 3D coordinate image Temp' corresponding to the current frame k . 3D coordinate image Temp k-1 and Temp' k Each of the 3D coordinate images Temp may be an image that displays an object or background that appears in an image (eg, a depth image or a color image) based on 3D information. k-1 and Temp' k The pixel value of each pixel in may have a 3D coordinate value, for which a predetermined reference point is set as the origin. The self-motion information estimation device may project a 3D coordinate image Temp' corresponding to the current frame by two-dimensionally projecting the 2D image 753. k To generate a 2D distorted image I w '(k). In this way, the distorted image I w '(k) is an image obtained by converting the image of the k-1th frame into a posture and position corresponding to the kth frame using information estimated in the previous frame (for example, the k-1th frame).

[0123] In operation 650, the self-motion information estimating device estimates the ego-motion information by w '(k) and the frame image to calculate the loss 790, thereby training the model. The self-motion information estimation device can be based on the current frame image in the training image 701 and the distorted image I w '(k) Calculate the loss 790, and train at least one of the self-motion model 310, the attention model 320, and the depth model 740. For reference, the distorted image I w '(k) and the pixel value of each pixel in the training image can be a color value.

[0124] The self-motion information estimation device is based on the training image I corresponding to the kth frame k，r and the distorted image I corresponding to the kth frame w'(k) to calculate the loss 790. For example, the self-motion information estimation device may calculate the value L1 as the pixel value difference between the two images. Therefore, regardless of whether the basic facts are given, the self-motion information estimation device can calculate the loss 790 by using the training image T corresponding to the k-th frame. k，r and the distorted image I corresponding to the kth frame w '(k) Calculate the loss 790 to perform unsupervised learning. The self-motion information estimation device may update the parameters of the self-motion model 310, the attention model 320, and the depth model 740 by back propagation so that the calculated loss 790 is reduced. The self-motion information estimation device may repeatedly update the parameters of each model until the loss 790 converges to a predetermined value. In addition, the self-motion information estimation device may update the parameters of each model a predetermined number of times until the loss 790 is reduced.

[0125] Figure 8 A method for estimating ego-motion information is shown.

[0126] In operation 810, the self-motion information estimation device extracts short-term self-motion features from the current frame image and the previous frame image based on the self-motion model. The short-term self-motion feature may be feature data extracted based on the self-motion model, and may be the output of a layer before the layer outputting the short-term self-motion information in the self-motion model.

[0127] In operation 820, the ego-motion information estimating apparatus generates a restored frame image corresponding to the current frame from at least one of a previous frame image and a depth image generated according to the previous frame based on the extracted short-term ego-motion feature and the deformation model. Fig. 9 An example of generating a restored frame image from a previous frame image is shown, and Fig.10 An example of generating a restored frame image from a depth image is shown. When the frame image of the input image is a color image, the restored frame image may be a color image having the same color system as the input image.

[0128] In operation 830, the self-motion information estimation device calculates correction information based on the restored frame image and the current frame image. For example, the self-motion information estimation device calculates correction information based on the restored frame image and the current frame image based on the self-motion model. The restored frame image is an image restored for the same frame as the current frame image, and therefore can be the same as the current frame image. Therefore, ideally, when the restored frame image and the current frame image are input to the self-motion model, no posture difference (e.g., posture error) is output. However, in practice, an error may occur. Such an error can be used as correction information.

[0129] In operation 840, the ego-motion information estimating apparatus calculates final short-term ego-motion information by applying the correction information to the short-term ego-motion information of the current frame estimated based on the ego-motion model.

[0130] Fig. 9 and Fig.10 The process of estimating self-motion information is shown.

[0131] refer to Fig. 9 The self-motion information estimation device can be based on the self-motion model 911, according to the k-1 frame image I k-1 and the kth frame image I k To calculate the short-term self-motion information SP k，k-1 In this example, the self-motion information estimation device can encode the short-term self-motion feature 929 based on the posture encoder 920, and the short-term self-motion feature 929 is output with the short-term self-motion information SP k，k-1 The output of the previous layer corresponds to the

[0132] The self-motion information estimation device can use the image I from the k-1 frame based on the deformation model 930. k-1 The encoded short-term self-motion feature 929 is used to generate the restored frame image I' corresponding to the kth frame k For example, the self-motion information estimation device can extract the k-1th frame image I through the image encoder of the deformation model 930 k-1 The image features are then transformed into the restored frame image I' using an image deformer. k , which is obtained by concatenating the extracted image features with the encoded short-term ego-motion features 929.

[0133] The self-motion information estimation device can calculate the correction information based on the self-motion model 912, and the correction information corresponds to the restored frame image I' k With the k-th frame image I k As an operation of correction 990, the ego-motion information estimation device applies the correction information to the short-term ego-motion information SP corresponding to the current frame. k，k-1 , thereby calculating the final short-term ego-motion information 909. Here, the ego-motion models 911 and 912 may be the same parameters with the same structure, but are not limited thereto.

[0134] refer to Fig.10 , similar to Fig. 9 In the example of FIG. 1 , the self-motion information estimation device is based on the self-motion model 1011, according to the k-1 frame image I k-1 and the kth frame image I k To calculate the short-term self-motion information SPk，k-1 In this example, the self-motion information estimation device encodes the short-term self-motion feature 1029 based on the posture encoder 1020, and the short-term self-motion feature 1029 is output with the short-term self-motion information SP k，k-1 The output of the previous layer corresponds to the

[0135] The self-motion information estimation device is based on the k-1 frame image I k-1 To generate the depth image D corresponding to the k-1th frame k-1 The self-motion information estimation device uses the short-term self-motion feature 1029 based on the depth image D corresponding to the k-1th frame based on the depth deformation model 1040. k-1 To restore the depth image D corresponding to the kth frame k , and according to the depth image D corresponding to the kth frame k To generate the restored frame image I" corresponding to the kth frame k For example, the self-motion information estimation device extracts the depth image D corresponding to the k-1th frame through the depth encoder of the deformation model 1030 k-1 The depth features of the image are then transformed into a depth image D' k , which is obtained by concatenating the extracted depth features with the encoded short-term ego-motion features 1029.

[0136] The self-motion information estimation device calculates correction information based on the self-motion model 1012, and the correction information corresponds to the restored frame image I" k With the k-th frame image I k As an operation of correction 1090, the ego-motion information estimation device applies the correction information to the short-term ego-motion information SP corresponding to the current frame. k，k-1 , thereby calculating the final short-term ego-motion information 1009. Here, the ego-motion models 1011 and 1012 may be the same parameters with the same structure, but are not limited thereto.

[0137] Fig.11 A model (e.g., with Fig. 9 and Fig.10 A training method for a combined model of structures. Fig.12 A model (e.g., with Fig. 9 and Fig.10 The training process of the model (a combination of structures).

[0138] In operation 1110, the self-motion information estimation device estimates the image from the training image pairs (eg, Tr k-1 and Tr k) to extract short-term self-motion features. The training image pair includes the training image Tr of the k-1th frame k-1 and the training image Tr of the kth frame k .

[0139] In operation 1120, the self-motion information estimation device generates a temporary restoration frame image corresponding to the current frame based on the extracted short-term self-motion features and the deformation model according to at least one of the following: a depth image generated according to the previous frame image; and a depth image generated according to the previous frame image in the training image. The self-motion information estimation device generates a temporary restoration frame image corresponding to the current frame based on the self-motion model 1210 from the training image to the Tr k-1 and Tr k The short-term self-motion features are extracted from the image deformation model 1230, the short-term self-motion features are encoded using the posture encoder 1220, and the encoded short-term self-motion features are forwarded to the image deformation model 1230 and the depth deformation model 1240. The self-motion information estimation device converts the depth image D extracted from the depth model 1250 into the depth image D extracted from the depth model 1250 based on the depth deformation model 1240 and the short-term self-motion features. k-1 Converted to the depth image D of the next frame k , and generate a second temporary restoration frame image I" k Similarly, the self-motion information estimation device generates a first temporary restoration frame image I' of the next frame according to the training image based on the image deformation model 1230 and the short-term self-motion feature. k .

[0140] In operation 1130, the self-motion information estimation apparatus calculates a loss based on the temporary restoration frame image and the current frame image in the training image. The self-motion information estimation apparatus calculates the first temporary restoration frame image I' k With the first training image Tr k The first loss 1291 between the pixels (for example, the loss L1 corresponding to the difference between the pixel values). In addition, the self-motion information estimation device calculates the second temporary restoration frame image I" k With the first training image Tr k A second loss 1292 between (eg, a loss L1 corresponding to the difference between the pixel values).

[0141] In operation 1140, the self-motion information estimation device trains the model based on the calculated loss. For example, the self-motion information estimation device repeatedly updates the parameters of each model until the calculated loss converges to a predetermined value.

[0142] Fig.13 The process of estimating self-motion information is shown.

[0143] A self-motion information estimation device will be described using Figure 3 , Fig. 9 and Fig.10 A combined model of the structure.

[0144] Similar to Figure 3 In the example, the self-motion information estimation device calculates the final long-term self-motion information LP' according to the frame image based on the first self-motion model 1311 and the attention model 1320. k，k-n According to the above equation 3, the self-motion information estimation device calculates the final long-term self-motion information LP' k，k-n , calculate attention-based short-term self-motion information 1381 (e.g. SP k，k-1 ).

[0145] The self-motion information estimation device uses the first self-motion model 1311 to estimate the self-motion information of the previous frame image I in the frame image. k-1 and the current frame image I k The short-term self-motion feature of the sensor is extracted. The extracted short-term self-motion feature is encoded by the image encoder 1330 and transmitted to the image deformation model 1340 and the depth deformation model 1360. The self-motion information estimation device makes the previous frame image I k-1 Deformation to generate the restored frame image I' corresponding to the current frame k The self-motion information estimation device is based on the second self-motion model 1312, according to the current frame image I k And restore frame image I' k The ego-motion information estimation device generates the image-based short-term ego-motion information 1382 by correcting 1345 by applying the first correction information 1341 to the short-term ego-motion information SP of the current frame. k，k-1 And executed.

[0146] The self-motion information estimation device uses the first self-motion model 1311 to estimate the self-motion information of the previous frame image I in the frame image. k-1 and the current frame image I k Extracting the short-term self-motion features SP of the sensor k，k-1 The self-motion information estimation device is based on the depth model 1350, according to the previous frame image I k-1 To generate the depth image D corresponding to the previous frame k-1 The self-motion information estimation device uses the short-term self-motion feature to make the depth image D corresponding to the previous frame image k-1 Deformation to generate the depth image D' corresponding to the current frame k The self-motion information estimation device converts the depth image D' corresponding to the current frame kConvert to restored frame image I" k The self-motion information estimation device is based on the third self-motion model 1313, according to the current frame image I k and restore frame image I' k The ego-motion information estimation apparatus generates depth-based short-term ego-motion information 1383 by correction 1355, which is obtained by applying the second correction information 1351 to the short-term ego-motion information SP corresponding to the current frame. k，k-1 And executed.

[0147] The ego-motion information estimation device determines the final short-term ego-motion information 1390 based on the following information: the image-based short-term ego-motion information 1382, and the attention-based short-term ego-motion information 1381 calculated from the final long-term ego-motion information. The ego-motion information estimation device determines the final short-term ego-motion information 1390 based on the following information: the depth-based short-term ego-motion information 1383, and the attention-based short-term ego-motion information 1381 calculated from the final long-term ego-motion information. The ego-motion information estimation device determines the final short-term ego-motion information 1390 corresponding to the current frame based on the following information: the attention-based short-term ego-motion information 1381, the image-based short-term ego-motion information 1382, and the depth-based short-term ego-motion information 1383.

[0148] Fig.14 and Fig.15 A self-motion information estimation device is shown.

[0149] The ego-motion information estimating apparatus 1400 includes a sensor 1410 , a processor 1420 (ie, one or more processors), and a memory 1430 .

[0150] The sensor 1410 acquires frame images of consecutive frames. For example, the sensor 1410 may be a camera sensor 1410 that receives a color image, but is not limited thereto. The sensor 1410 may acquire a stereoscopic image. The sensor 1410 may also include a depth sensor 1410 that receives a depth image. When the perspective matches so that the pixels of the images acquired by the depth sensor 1410 and the camera sensor 1410 indicate the same point, the depth image acquired by the depth sensor 1410 may be used for the training process. In this example, instead of a depth image generated from a depth model, a depth model acquired by the depth sensor 1410 is used for the training process.

[0151] The processor 1420 estimates the short-term self-motion information of the sensor 1410; estimates the long-term self-motion information of the sensor 1410 according to the frame image; calculates the attention information according to the short-term self-motion information and the long-term self-motion information; corrects the long-term self-motion information based on the attention information; and determines the final long-term self-motion information of the current frame. The processor 1420 is configured to perform the reference in the corresponding embodiment. Figures 1 to 13 Any one, any combination, or all of the operations described.

[0152] The memory 1430 temporarily or permanently stores data for estimating the self-motion information. The memory 1430 stores the frame image, the short-term self-motion information, and the long-term self-motion information acquired by the sensor 1410 for a predetermined interval (eg, n frames).

[0153] refer to Fig.15 , computing device 1500 is a device for estimating the self-motion of a sensor using the self-motion information estimation method described herein. In an example, the computing device corresponds to Fig.14 The computing device 1500 may be, for example, an image processing device, a smart phone, a wearable device, a tablet computer, a netbook, a laptop computer, a desktop computer, a personal digital assistant (PDA), an HMD, etc.

[0154] refer to Fig.15 , the computing device 1500 includes a processor 1510, a storage device 1520, a camera 1530, an input device 1540, an output device 1550, and a network interface 1560. The processor 1510, the storage device 1520, the camera 1530, the input device 1540, the output device 1550, and the network interface 1560 can communicate with each other via a communication bus 1570.

[0155] Processor 1510 executes functions and instructions in computing device 1500. For example, processor 1510 processes instructions stored in storage device 1520. Processor 1510 executes the above referenced instructions. Figures 1 to 14 Describes one or more operations.

[0156] The storage device 1520 stores information and data required for the processor to execute. The storage device 1520 includes a computer-readable storage medium or a computer-readable storage device. The storage device 1520 stores instructions to be executed by the processor 1510 and stores relevant information when the computing device 1500 is executing software or applications.

[0157] The camera 1530 captures an image including a plurality of image frames. For example, the camera 1530 generates a frame image.

[0158] The input device 1540 receives input from the user through tactile, video, audio or touch input. The input device 1540 includes, for example, a keyboard, a mouse, a touch screen, a microphone, and other devices that can detect input from the user and send the detected input.

[0159] Output device 1550 provides output of computing device 1500 to the user through visual, auditory or tactile means. Output device 1550 includes, for example, a display, a touch screen, a speaker, a vibration generator, and other devices that can provide output to the user. Network interface 1560 communicates with external devices through a wired or wireless network. Output device 1550 outputs in Figures 1 to 13 At least one of the final short-term self-motion information, the final long-term self-motion information, the position information, and the posture information calculated in the example of . The output device 1550 provides the final short-term self-motion information, the final long-term self-motion information, the position information, and the posture information to the user using at least one of the visual information, the auditory information, and the tactile information. As a non-limiting example, in various corresponding embodiments, when the computing device 1500 is a mobile user device, a robotic device, or a vehicle, or when the computing device 1500 is installed in / on a mobile user device, a robotic device, or a vehicle, the computing device 1500 is configured to: estimate the position and posture of the mobile user device, the robotic device, or the vehicle based on the final short-term self-motion information and the final long-term self-motion information; generate a graphical representation corresponding to the estimated position and posture (e.g., a user / device or vehicle object on a generated map background), and visualize the graphical representation on a display; and / or use the estimated position and posture of the mobile user device, the robotic device, or the vehicle to control position mapping (e.g., simultaneous localization and mapping (SLAM)), and control the autonomous movement of the robotic device or vehicle based on the same information.

[0160] Fig.16 An example of an apparatus for estimating ego-motion information installed in a vehicle is shown.

[0161] When the self-motion information estimation device is installed in a vehicle, the sensors 1621 and 1622 may be arranged to face the outside from the inside of the vehicle. For example, the optical axis and FOV of the sensor may be set to face the front of the vehicle. Fig.16A situation 1601 is shown in which a vehicle moves from a first position to a second position, for example, turns left. When an object 1690 exists in front of the vehicle, at a first FOV 1631 of a sensor 1621 at a first position 1611, the object 1690 appears in the left portion of a first frame image 1641. In addition, at a second FOV 1632 of a sensor 1622 at a second position 1612, the object 1690 appears in the right portion of a second frame image 1642. In this way, the self-motion information estimation device can estimate the self-motion information of the sensor based on the change of an object or a static background from one frame image to another frame image (for example, from a first frame image 1641 to a second frame image 1642).

[0162] As reference Figures 1 to 15 As described, the self-motion information estimation device can track the travel path of the device including the sensor based on at least one of the final long-term self-motion information and the final short-term self-motion information calculated from the final long-term self-motion information. The self-motion information estimation device can set a predetermined geographical location or physical location as an origin as a reference location, set a reference posture, accumulate the application of the final short-term self-motion information and / or the final long-term self-motion information to the reference location and reference posture, and determine the location and posture of the device in each frame. For example, the self-motion information estimation device can initialize the location and posture of the device in a predetermined initialization frame. The self-motion information estimation device can set the location and posture of the device in the initialization frame as the reference location and reference posture. In the frame after the initialization frame, the self-motion information estimation device can apply the final short-term self-motion information determined in the subsequent frame to the reference location and reference posture in the initialization frame, thereby calculating the location and posture in the subsequent frame. For example, the self-motion information estimation device can obtain a matrix indicating the posture and position of the device in subsequent frames based on the result of a matrix product calculation, which is a matrix product calculation between the matrix indicating the position and posture of the device and a matrix indicating the final short-term self-motion information (for example, a matrix having the same form as Equation 1). Thereafter, similar to the aforementioned operation, for each frame, the self-motion information estimation device can determine the position and posture of the device corresponding to the current frame by applying the short-term self-motion information to the posture and position of the device determined in the previous frame. The self-motion information estimation device can track the posture of the device at each position and the path of travel of the device by accumulating the posture and position determined in the corresponding frame.

[0163] The information associated with the posture and position of the device is not limited to the form of a matrix, and other types of data may also be used. Although the final short-term self-motion information may be applied to the posture and position in the previous frame by matrix product calculation as an example, other operations may also be used. Moreover, although the final short-term self-motion information may be used as an example, the embodiments are not limited thereto. By applying the final long-term self-motion information to the posture and position corresponding to the long-term reference frame of the current frame, the self-motion information estimation device may also determine the posture and position of the device corresponding to the current frame.

[0164] The self-motion information estimation device may output the tracked path of travel of the device. For example, the self-motion information estimation device may use a display (e.g., a display of one or both of the sensor and the vehicle) to output a graphical representation (which represents a geographical map of the terrain around the device), overlay a graphical representation corresponding to the tracked path of travel on the graphical representation corresponding to the geographical map, and output the overlaid result. However, the visualization of the output of the path of travel is not limited to this. The path of travel may also be output as auditory or tactile information.

[0165] In an example, the self-motion information estimation device may be implemented as a head mounted device (HMD) configured to provide augmented or virtual reality information. In this example, the display of the HMD device is disposed toward the user's eyes. In addition, the sensor is disposed toward the opposite side of the display based on the housing of the device. In other words, the sensor is disposed in the same direction as the user's gaze direction. Since the self-motion information estimation device moves with the movement of the user's head, the self-motion information estimation device estimates the movement of the user's head based on the self-motion information of the sensor. The self-motion information estimation device uses a reference Figures 1 to 15 The method described estimates the self-motion information of the sensor and feeds back the estimated self-motion information to the content visualized on the display. For example, the self-motion information estimation device sets the posture and position of the HMD in the initialization frame as the reference posture and reference position. Thereafter, the self-motion information estimation device applies the final short-term self-motion information and / or the final long-term self-motion information accumulation to the reference posture and reference position in subsequent frames, thereby determining the posture and position of the HMD in each frame. In addition, when the movement of the user rotating the head in a predetermined direction (e.g., right) is detected based on the self-motion information of the sensor, the self-motion information estimation device controls the HMD to display virtual content corresponding to the direction.

[0166] apparatus, apparatus 1400, apparatus 1500, processor, processor 1420, processor 1510, sensor, sensor 1410, sensor 1621, sensor 1622, camera 1530, memory, memory 1430, storage device 1520, input device 1540, output device 1550, network interface 1560, bus 1570, unit, module, device and the like Figures 1 to 15Other components described are implemented by hardware components or represent hardware components. Examples of hardware components that can be used to perform the operations described in this application include controllers, sensors, generators, drivers, memories, comparators, arithmetic logic units, adders, subtractors, multipliers, dividers, integrators, and any other electronic components configured to perform the operations described in this application, where appropriate. In other examples, one or more hardware components that perform the operations described in this application are implemented by computing hardware, such as by one or more processors or computers. A processor or computer can be implemented by one or more processing elements, such as logic gate arrays, controllers and arithmetic logic units, digital signal processors, microcomputers, programmable logic controllers, field programmable gate arrays, programmable logic arrays, microprocessors, or any other device or combination of devices configured to respond and execute instructions in a defined manner to achieve desired results. In one example, a processor or computer includes or is connected to one or more memories storing instructions or software executed by the processor or computer. The hardware components implemented by a processor or computer can execute instructions or software (such as an operating system (OS) and one or more software applications running on the OS) to perform the operations described in this application. Hardware components can also access, manipulate, process, create and store data in response to the execution of instructions or software.For simplicity, the singular term "processor" or "computer" can be used in the description of the examples described in this application, but multiple processors or computers can be used in other examples, or processors or computers can include multiple processing elements, or multiple types of processing elements or both.For example, a single hardware component or two or more hardware components can be implemented by a single processor, or two or more processors, or a processor and a controller.One or more hardware components can be implemented by one or more processors, or a processor and a controller, and one or more other hardware components can be implemented by one or more other processors or another processor and another controller.One or more processors or a processor and a controller can implement a single hardware component, or two or more hardware components.Hardware components can have any one or more of different processing configurations, and the example of the processing configuration includes a single processor, an independent processor, a parallel processor, a single instruction single data (SISD) multiprocessing, a single instruction multiple data (SIMD) multiprocessing, a multiple instruction single data (MISD) multiprocessing, and a multiple instruction multiple data (MIMD) multiprocessing.

[0167] Perform the operations described in this application Figures 1 to 15The method shown is performed by computing hardware (e.g., one or more processors or computers) implemented in the manner described above, and the computing hardware executes instructions or software to perform the operations described in the present application performed by each method. For example, a single operation or two or more operations can be performed by a single processor, or two or more processors, or a processor and a controller. One or more operations can be performed by one or more processors or a processor and a controller, and one or more other operations can be performed by one or more other processors or another processor and another controller. One or more processors or a processor and a controller can perform a single operation or two or more operations.

[0168] The instructions or software for controlling computing hardware (e.g., one or more processors or computers) to implement hardware components and perform the methods described above can be written as computer programs, code segments, instructions, or any combination thereof, for individually or collectively instructing or configuring one or more processors or computers to operate as machines or special-purpose computers to perform the operations performed by the above-mentioned hardware components and methods. In one example, the instructions or software include machine codes directly executed by one or more processors or computers, such as machine codes generated by a compiler. In another example, the instructions or software include higher-level codes executed by one or more processors or computers using an interpreter. Instructions or software can be written in any programming language based on the block diagrams and flow charts shown in the accompanying drawings and the corresponding descriptions in the specification (which disclose algorithms for performing the operations performed by the hardware components and the methods described above).

[0169] Instructions or software for controlling computing hardware (e.g., one or more processors or computers) to implement hardware components and perform the methods described above, as well as any associated data, data files, and data structures, may be recorded, stored, or fixed in or on one or more non-transitory computer-readable storage media. Examples of non-transitory computer-readable storage media include: read-only memory (ROM), random access programmable read-only memory (PROM), electrically erasable programmable read-only memory (EEPROM), random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), flash memory, non-volatile memory, CD-ROM, CD-R, CD+R, CD-RW, CD+RW, DVD-ROM, DVD-R, DVD+R, DVD-RW, DVD+RW, DVD-RAM, BD-ROM, BD-R, BD-R LTH, BD-RE, Blu-ray or optical disk storage, hard disk drive (HDD), solid state drive (SSD), flash memory, card-type storage (such as, multimedia card or micro card (for example, secure digital (SD) or extreme digital (XD))), magnetic tape, floppy disk, magneto-optical data storage device, optical data storage device, hard disk, solid state disk and any other device, which is configured to store instructions or software and any associated data, data files and data structures in a non-temporary manner, and provide instructions or software and any associated data, data files and data structures to one or more processors or computers so that the one or more processors or computers can execute the instructions. In one example, the instructions or software and any associated data, data files and data structures are distributed on a networked computer system so that one or more processors or computers store, access and execute the instructions and software and any associated data, data files and data structures in a distributed manner.

[0170] Although the present disclosure includes specific examples, it will be apparent after understanding the disclosure of the present application that various changes in form and detail may be made to these examples without departing from the spirit and scope of the claims and their equivalents. The examples described herein should be considered as descriptive only and not for limiting purposes. The description of the features or aspects in each example should be considered as similar features or aspects that can be used in other examples. If the described techniques are performed in a different order, and / or if the components in the described systems, architectures, devices, or circuits are combined in different ways, and / or replaced or supplemented by other components or their equivalents, suitable results can be achieved. Therefore, the scope of the present disclosure is not limited by the detailed description, but by the attached claims and their equivalents, and all variations within the scope of the claims and their equivalents will be interpreted as being included in the present disclosure.

Claims

1. A method implemented by a processor, the method comprising: estimating short-term self-motion information of the one or more sensors based on frame images of consecutive frames acquired from the one or more sensors, the short-term self-motion information indicating a change in self-motion of the one or more sensors between a current frame and a short-term reference frame in the consecutive frames; estimating long-term self-motion information of the one or more sensors according to the frame image, the long-term self-motion information indicating a self-motion change of the one or more sensors between the current frame and a long-term reference frame in the continuous frames, wherein the number of frames between the long-term reference frame and the current frame is greater than the number of frames between the short-term reference frame and the current frame; determining attention information by extracting attention elements from the short-term self-motion information and the long-term self-motion information using an attention model, wherein the attention elements are data in which a higher weight is assigned to key data similar to query data among key data, wherein the query data corresponds to the long-term self-motion information and the key data corresponds to the short-term self-motion information; and Based on the long-term self-motion information and the attention information, final long-term self-motion information of the current frame in the continuous frames is determined.

2. The method according to claim 1, wherein: Determining the final long-term self-motion information of the current frame includes: using the attention information to correct the long-term self-motion information.

3. The method according to claim 1, wherein: Determining the final long-term ego-motion information includes: determining a change in ego-motion of the one or more sensors from the long-term reference frame to the current frame, and A frame before the current frame by a predetermined number of frames is determined as the long-term reference frame.

4. The method according to claim 1, further comprising: Based on the determined final long-term ego-motion information, final short-term ego-motion information of the current frame is determined.

5. The method according to claim 4, wherein: Determining the final short-term self-motion information includes: The final short-term ego-motion information of the current frame is determined based on: the final long-term ego-motion information of a frame before the current frame in the continuous frames, the final short-term ego-motion information of the long-term reference frame, and the final long-term ego-motion information of the current frame.

6. The method according to claim 5, wherein: The frame preceding the current frame is directly adjacent to the current frame in the continuous frames.

7. The method according to claim 1, wherein: Estimating the short-term ego-motion information of the one or more sensors comprises: In response to receiving a current frame image of the current frame, estimating short-term self-motion information between the current frame and the short-term reference frame using an ego-motion model; and The short-term ego-motion information of the one or more sensors is estimated based on the estimated short-term ego-motion information between the current frame and the short-term reference frame.

8. The method according to claim 1, wherein: Estimating the long term ego-motion information of the one or more sensors comprises: In response to receiving a current frame image of the current frame, estimating long-term self-motion information between the current frame and the long-term reference frame using an ego-motion model; and The long term ego-motion information of the one or more sensors is estimated based on the estimated long term ego-motion information between the current frame and the long term reference frame.

9. The method according to claim 1, wherein: The estimating the short-term self-motion information comprises: using a short-term self-motion model to estimate the short-term self-motion information according to a pair of consecutive frame images in the frame images; Estimating the long-term self-motion information comprises: estimating the long-term self-motion information according to the current frame image of the current frame and the reference frame image of the long-term reference frame using a long-term self-motion model, and The short-term self-motion model has a different configuration than the long-term self-motion model, at least having different training parameters.

10. The method according to claim 1, wherein: Determining the attention information includes: Extracting attention weights from the short-term ego-motion information; and The attention information is generated based on the attention element and the attention weight.

11. The method according to claim 1, further comprising: In response to the number of frame images collected from the initial frame to the current frame being less than the determined number of frames, the short-term self-motion information is determined as final short-term self-motion information.

12. The method according to claim 1, further comprising: Extracting short-term self-motion features of the one or more sensors from a current frame image of the current frame and a frame image before the current frame image in the frame images using a self-motion model; Using an image deformation model, generating a restored frame image of the current frame by deforming the previous frame image based on the short-term self-motion feature; Using the self-motion model, calculating correction information according to the current frame image and the restored frame image; as well as Image-based short-term ego-motion information is generated by applying the correction information to the short-term ego-motion information of the current frame.

13. The method according to claim 12, wherein: The correction information corresponds to a small posture error between the restored frame image and the current frame image.

14. The method according to claim 12, further comprising: Final short-term self-motion information is determined based on the attention-based short-term self-motion information calculated from the final long-term self-motion information and the image-based short-term self-motion information.

15. The method according to claim 1, further comprising: Extracting short-term self-motion features of the one or more sensors from a current frame image of the current frame and a frame image before the current frame image in the frame images using a self-motion model; Generating a depth image of a frame before the current frame according to the previous frame image; generating a depth image corresponding to the current frame by deforming the depth image of the previous frame based on the short-term ego-motion feature; Converting the depth image corresponding to the current frame into a restored frame image; Using the self-motion model, calculating correction information according to the current frame image and the restored frame image; as well as Depth-based short-term ego-motion information is generated by applying the correction information to the short-term ego-motion information of the current frame.

16. The method according to claim 14, further comprising: Final short-term self-motion information is determined based on the attention-based short-term self-motion information calculated according to the final long-term self-motion information and the depth-based short-term self-motion information.

17. The method according to claim 1, further comprising: tracking a travel path of a device equipped with the one or more sensors based on at least one of the final long-term self-motion information and the final short-term self-motion information calculated based on the final long-term self-motion information; as well as The tracked travel path of the device is output.

18. The method of claim 1, further comprising: Using a depth model, generating temporary depth information of a frame before the current frame according to a training image; generating temporal long-term self-motion information based on the training images using the self-motion model and the attention model; calculating temporary short-term self-motion information based on the temporary long-term self-motion information; generating a warped image of a current frame based on the temporary short-term ego-motion information and the temporary depth information of the previous frame; as well as Based on the loss calculated based on the warped image and the current frame image in the training image, any one of the following models or any combination of any two or more of the following models is trained: the self-motion model, the attention model, and the depth model.

19. The method according to claim 18, wherein: Generating the distorted image comprises: Generating a three-dimensional 3D coordinate image corresponding to the previous frame according to the temporary depth information; restoring a 3D coordinate image corresponding to a current frame by converting the 3D coordinate image corresponding to the previous frame using the temporary short-term ego-motion information calculated based on the temporary long-term ego-motion information; and The distorted image is generated by two-dimensionally projecting a 3D coordinate image corresponding to the current frame, so that the distorted image is two-dimensionally distorted.

20. A non-transitory computer-readable storage medium storing instructions that, when executed by one or more processors, cause the one or more processors to perform the method of claim 1.

21. A method implemented by a processor, the method comprising: Using the self-motion model, extracting feature data from a current frame image and a previous frame image as a short-term self-motion feature, wherein the current frame image is a frame image of a current frame in frame images of continuous frames acquired from one or more sensors, and the previous frame image is a frame image of a frame before the current frame in the continuous frames; Generate a restored frame image corresponding to the current frame based on the extracted short-term self-motion features and the previous frame image and at least one of the depth images generated from the previous frame image using a deformation model; Calculating correction information according to the restored frame image and the current frame image; and Final short-term ego-motion information is calculated by applying the correction information to the estimated short-term ego-motion information of the current frame using the ego-motion model, the short-term ego-motion information indicating a change in ego-motion between the current frame and the previous frame.

22. A self-motion information estimation device, the self-motion information estimation device comprising: One or more sensors configured to acquire consecutive frames of frame images; as well as One or more processors configured to: estimating short-term self-motion information of the one or more sensors according to the frame image, the short-term self-motion information indicating a change in self-motion of the one or more sensors between a current frame and a short-term reference frame in the continuous frames; estimating long-term self-motion information of the one or more sensors according to the frame image, the long-term self-motion information indicating a self-motion change of the one or more sensors between the current frame and a long-term reference frame in the continuous frames, wherein the number of frames between the long-term reference frame and the current frame is greater than the number of frames between the short-term reference frame and the current frame; calculating attention information by extracting attention elements from the short-term self-motion information and the long-term self-motion information using an attention model, wherein the attention elements are data in which a higher weight is assigned to key data similar to query data among key data, wherein the query data corresponds to the long-term self-motion information and the key data corresponds to the short-term self-motion information; and Final long-term self-motion information of the current frame is determined by correcting the long-term self-motion information based on the attention information.

23. The ego-motion information estimating device according to claim 22, wherein: The one or more sensors include one or more image sensors, and The one or more processors are further configured to determine a path of travel of the one or more image sensors based on the determined final long-term ego-motion information.

Images