Method for estimating object motion based on vision sensor and object motion estimating device using the same

Abstract

An object motion estimation device is provided that estimates the motion of an object based on a vision sensor. [Solution] The method includes the steps of sequentially acquiring vision sensor data detected through a detection model from images captured by a camera installed in a vehicle and vehicle system data detected from the vehicle's system, inputting the t-th vision sensor data and the t-th vehicle system data corresponding to the current image frame, the t-th image frame, into a data pre-processing module for pre-processing to generate a t-th object motion estimation vector, and inputting the object motion estimation vector into a sequential regression network and performing recurrent learning calculations on the t-th object motion estimation vector to generate the t-th predicted object motion data.

Description

[Technical Field]

[0001] The present invention relates to a method for estimating the motion of an object based on a vision sensor and an object motion estimation device using the same. [Background technology]

[0002] Generally, ADAS (Advanced Driving Assistance System) is a system that assists and supports driving to make automobiles safer and more convenient.

[0003] Automobiles are equipped with various recognition sensors such as radar, lidar, cameras, and ultrasonic waves, and these sensors are used to detect approaching vehicles, pedestrians, and obstacles, and safety technologies are applied that either warn the driver of danger in advance or actively help the driver avoid accidents.

[0004] For example, autonomous vehicles are equipped with a Forward Collision Warning system (FCW), which monitors the vehicle's speed and the relative speed with the vehicle ahead and warns the driver of the risk of a collision if the distance to the vehicle ahead becomes too close; a Lane Departure Warning system (LDW), which monitors the driving situation ahead and detects when the vehicle is leaving its lane and warns the driver; a Blind Spot Warning system (BSW), which detects other vehicles in the blind spots on the sides of the vehicle that are difficult to see from the driver's seat using the left and right side mirrors, and notifies or warns the driver; an Automatic Emergency Braking system (AEB), which automatically applies the vehicle's brakes to prevent a collision or reduce the impact of an accident if a risk of collision with a vehicle ahead is detected while driving; and a Lane Keeping Assist system (LKAS), which warns the driver in advance to prevent the vehicle from leaving its normal lane while driving and helps the vehicle return to its normal lane if it deviates from its lane. The vehicle is equipped with a variety of safety technologies, including an Adaptive Cruise Control (ACC) system that helps the vehicle stay in the center of the lane while maintaining a safe distance from the vehicle in front according to the maximum speed set by the driver, a Parking Collision-Avoidance Assist (PCAA) system that warns the driver of a potential collision with another vehicle, pedestrian, or obstacle behind the vehicle when reversing at low speed and assists with emergency braking, and a Smart Parking Assist (SPA) system that uses cameras and ultrasonic sensors to search for parking spaces and automatically controls steering, gears, and vehicle speed based on the results to help the driver park and exit the parking space.

[0005] Meanwhile, in recent years, the ADAS market has seen a growing demand for vision-based perception systems that can operate in real time even on lean platforms with limited resources and computing resources.

[0006] In particular, efforts to realize safety and convenience functions such as automatic emergency braking systems and adaptive cruise control using only vision sensors are gaining momentum, and improving the recognition performance based on a single camera has become an important issue.

[0007] However, conventional filtering methods alone have technical limitations when it comes to estimating higher-order physical quantities such as velocity and acceleration based on distance information estimated from a single camera.

[0008] In other words, minute errors and noise in the estimated distance to an object through analysis of images taken by a single camera are amplified in calculations of speed, acceleration, etc., making it difficult to ensure the stable physical quantities required for ADAS control. Therefore, the present applicant proposes a technique that enables stable estimation of physical quantities such as speed and acceleration from distance information acquired by a vision sensor. Summary of the Invention [Problem to be solved by the invention]

[0009] An object of the present invention is to solve all of the problems of the prior art described above.

[0010] Another object of the present invention is to enable stable estimation of physical quantities such as velocity and acceleration from distance information of an object detected by a vision sensor.

[0011] Furthermore, another object of the present invention is to be able to provide stable motion data of an object required for ADAS control from distance information of the object acquired based on a vision sensor. [Means for solving the problem]

[0012] According to one embodiment of the present invention, a method for estimating the motion of an object based on a vision sensor includes: (a) sequentially acquiring vision sensor data, including an original object relative distance, which is a distance from the vehicle to an object, and an object heading angle, which is a heading angle of the object, detected through a detection model based on deep learning from at least one image captured by at least one camera installed in a vehicle, and vehicle system data detected from a system of the vehicle; and an object motion estimation device inputs the t-th vision sensor data and the t-th vehicle system data corresponding to a t-th image frame, which is a current image frame, into a data pre-processing module, and then performs the data pre-processing. The method includes: (a) using a pre-processing module to pre-process the t vision sensor data and the t vehicle system data to generate a t-th object motion estimation vector for estimating the motion of the object; and (b) using the object motion estimation device to input the t object motion estimation vector to a deep learning-based sequential regression network, and using the sequential regression network to perform a recurrent learning operation on the t object motion estimation vector to generate t-th predicted object motion data that predicts the motion of the object corresponding to the t image frame.

[0013] In one example, (c) the object motion estimation device inputs the t predicted object motion data and the t vehicle system data into a sequential filtering network, and uses the sequential filtering network to (i) generate (t-1) corrected object motion data by correcting the (t-1) predicted object motion data in the (t-1) image frame through a state prediction model (the (t-1) corrected object motion data is object motion data obtained by correcting the (t-1) predicted object motion data with reference to the (t-1) predicted object motion data and the (t-1) vehicle system data), and t predicted corrected object motion data by applying a learning operation to the t vehicle system data, and (ii) generate a state vector based on a difference between the t predicted object motion data and the (t-1) predicted object motion data through a state vector generation module. (iii) applying a recurrent operation to the t-th corrected differential state vector through a first filtering model to generate a t_1th uncertainty probability value that estimates the uncertainty of the t-th predicted corrected object motion data, and applying a recurrent operation to the t-th predicted differential state vector and the t-th predicted-corrected differential state vector through a second filtering model to generate a t_2th uncertainty probability value that estimates the uncertainty of the t-th predicted object motion data; (iv) applying a t-th feedback gain generated with reference to the t_1st uncertainty probability value and the t_2nd uncertainty probability value to the t-th predicted-corrected differential state vector to correct the t-th predicted corrected object motion data, therebygenerating a t-th corrected object motion data in the t-th image frame;

[0014] In one example, in step (c), the object motion estimation device uses the sequential filtering network to further generate a t-th object distance state vector corresponding to the t-th predicted object relative distance included in the t-th predicted object motion data through the state vector generation module, apply recurrent operations to the t-th corrected differential state vector and the t-th object distance state vector through the first filtering model to generate the t_1 uncertainty probability value that estimates the uncertainty of the t-th predicted corrected object motion data, and apply recurrent operations to the t-th predicted differential state vector, the t-th predicted-corrected differential state vector, and the t-th object distance state vector through the second filtering model to generate the t_2 uncertainty probability value that estimates the uncertainty of the t-th predicted object motion data.

[0015] In one example, in step (c), the t vehicle system data includes a t vehicle speed, which is the speed of the vehicle in the t image frame, a t vehicle acceleration, which is the acceleration of the vehicle in the t image frame, a t vehicle yaw rate, which is the yaw rate of the vehicle in the t image frame, and a t time difference, which is the difference between the time when the (t-1) image frame was acquired and the time when the t image frame was acquired.

[0016] In one example, in the step (c), the state prediction model is configured with a motion model including at least one of a constant velocity model and a constant acceleration model.

[0017] In one example, in step (c), each of the first filtering model and the second filtering model is composed of one of a recurrent neural network (RNN), a long short-term memory (LSTM), and a gated recurrent unit (GRU).

[0018] In one example, in step (a), the object motion estimation device uses the data preprocessing module to refer to the t vision sensor data and the t vehicle system data, and generates, as the t object motion estimation vector, a t object relative distance estimation vector including t object relative distance estimation data, a t object absolute velocity estimation vector including t object absolute velocity estimation data, and a t object absolute acceleration estimation vector including t object absolute acceleration estimation data; and in step (b), the object motion estimation device uses the sequential regression network to apply recurrent operations to the t object relative distance estimation vector through an object relative distance estimation model, to generate the t object relative distance estimation vector for the t image. a t-th predicted object relative distance that predicts the relative distance of the object from the vehicle in the t-th image frame; a t-th predicted object absolute velocity that predicts the absolute velocity of the object in the t-th image frame by applying recurrent operations to the t-th object absolute velocity estimation vector through an object absolute velocity estimation model; and a t-th predicted object absolute acceleration that predicts the absolute acceleration of the object in the t-th image frame by applying recurrent operations to the t-th object absolute acceleration estimation vector through an object absolute acceleration estimation model, thereby generating the t-th predicted object motion data including the t-th predicted object relative distance, the t-th predicted object absolute velocity, and the t-th predicted object absolute acceleration.

[0019] In one example, in step (a), the object motion estimation device uses the data pre-processing module to generate the t object motion estimation vector, which includes at least a portion of the t original object relative distance, the t original object relative velocity which is the relative velocity of the object with respect to the vehicle in the t image frame, the t original object absolute velocity which is the absolute velocity of the object in the t image frame, the t vehicle speed which is the velocity of the vehicle in the t image frame, the t vehicle acceleration which is the acceleration of the vehicle in the t image frame, the t vehicle yaw rate which is the yaw rate of the vehicle in the t image frame, the t object heading angle, and the t time difference which is the difference between the time when the (t-1) image frame was acquired and the time when the t image frame was acquired, wherein the t original object relative distance, the t original object relative velocity, the t original object absolute velocity, the t vehicle speed, the t vehicle acceleration, the t vehicle yaw rate, the t object heading angle, and the t time difference are included in the t vision sensor data and the t vehicle system data, or are calculated with reference to them.

[0020] In one example, in the step (a), the object motion estimation device uses the data pre-processing module to refer to the t vision sensor data and the t vehicle system data to generate, as the t object motion estimation vectors, (i) a t-th object relative distance estimation vector including the t original object relative distance, the t original object relative velocity, the t object heading angle, and the t time difference, (ii) a t-th object absolute velocity estimation vector including the t original object relative distance, the t original object absolute velocity, the t vehicle velocity, the t vehicle yaw rate, the t object heading angle, and the t time difference, and (iii) a t-th object absolute acceleration estimation vector including the t original object relative distance, the t original object absolute velocity, the t vehicle velocity, the t vehicle acceleration, the t vehicle yaw rate, the t object heading angle, and the t time difference, and in the step (b), The motion estimation device uses the sequential regression network to apply recurrent operations to the t object relative distance estimation vector through an object relative distance estimation model to generate a t-th predicted object relative distance that predicts the relative distance of the object from the vehicle in the t image frame, apply recurrent operations to the t object absolute velocity estimation vector through an object absolute velocity estimation model to generate a t-th predicted object absolute velocity that predicts the absolute velocity of the object in the t image frame, and apply recurrent operations to the t object absolute acceleration estimation vector through an object absolute acceleration estimation model to generate a t-th predicted object absolute acceleration that predicts the absolute acceleration of the object in the t image frame, thereby generating the t-th predicted object relative distance, the t-th predicted object absolute velocity, and the t-th predicted object absolute acceleration.

[0021] In one example, in step (a), the object motion estimation device uses the data pre-processing module to generate the t object motion estimation vector so that each of the t object relative distance estimation vector, the t object absolute velocity estimation vector, and the t object absolute acceleration estimation vector further includes a t vehicle pitch, which is the pitch of the vehicle in the t image frame, and a t vehicle roll, which is the roll of the vehicle in the t image frame.

[0022] In one example, in step (b), the sequential regression network is configured with any one model of a recurrent neural network (RNN), a long short-term memory (LSTM), and a gated recurrent unit (GRU).

[0023] According to one embodiment of the present invention, an object motion estimation device for estimating the motion of an object based on a vision sensor includes: a memory storing instructions for estimating the motion of an object based on a vision sensor; and a processor that performs operations for estimating the motion of the object based on the vision sensor in accordance with the instructions stored in the memory; wherein the processor (I) sequentially acquires vision sensor data, including an original object relative distance, which is the distance from the vehicle to the object, and an object heading angle, which is the heading angle of the object, detected through a detection model based on deep learning from at least one image captured by at least one camera installed in a vehicle, and vehicle system data detected from a system of the vehicle, An object motion estimation device is provided that executes the following processes: (i) inputting t-th vision sensor data and t-th vehicle system data corresponding to a t-th image frame, which is a current image frame, into a data pre-processing module, and using the data pre-processing module to pre-process the t-th vision sensor data and the t-th vehicle system data to generate a t-th object motion estimation vector for estimating the motion of the object; and (ii) inputting the t-th object motion estimation vector into a sequential regression network based on deep learning, and using the sequential regression network to perform a recurrent learning operation on the t-th object motion estimation vector to generate t-th predicted object motion data that predicts the motion of the object corresponding to the t-th image frame.

[0024] In one example, the processor (III) inputs the t predicted object motion data and the t vehicle system data into a sequential filtering network, and uses the sequential filtering network to (i) generate (t-1) corrected object motion data obtained by correcting the (t-1) predicted object motion data in the (t-1) image frame through a state prediction model (the (t-1) corrected object motion data is object motion data obtained by correcting the (t-1) predicted object motion data with reference to the (t-1) predicted object motion data and the (t-1) vehicle system data), and t predicted corrected object motion data obtained by predicting the corrected object motion data in the t image frame by applying a learning operation to the t vehicle system data; and (ii) generate a state vector based on a difference between the t predicted object motion data and the (t-1) predicted object motion data through a state vector generation module. generating a t-th predicted differential state vector, a t-th predicted-corrected differential state vector based on the difference between the t-th predicted object motion data and the t-th predicted-corrected object motion data, and a t-th corrected differential state vector based on the difference between the t-th predicted corrected object motion data and the (t-1)-th corrected object motion data; (iii) applying a recurrent operation to the t-th corrected differential state vector through a first filtering model to generate a t_1-th uncertain probability value that estimates the uncertainty of the t-th predicted corrected object motion data; and applying a recurrent operation to the t-th predicted differential state vector and the t-th predicted-corrected differential state vector through a second filtering model to generate a t_2-th uncertain probability value that estimates the uncertainty of the t-th predicted object motion data; (iv) applying a t-th feedback gain generated with reference to the t_1-th uncertain probability value and the t_2-th uncertain probability value to the t-th predicted-corrected differential state vector to correct the t-th predicted corrected object motion data, therebyFurther performing a process to generate a t-th corrected object motion data in the t-th image frame.

[0025] In one example, in the process (III), the processor uses the sequential filtering network to further generate a t-th object distance state vector according to the t-th predicted object relative distance included in the t-th predicted object motion data through the state vector generation module, apply recurrent operations to the t-th corrected differential state vector and the t-th object distance state vector through the first filtering model to generate the t_1 uncertainty probability value that estimates the uncertainty of the t-th predicted corrected object motion data, and apply recurrent operations to the t-th predicted differential state vector, the t-th predicted-corrected differential state vector, and the t-th object distance state vector through the second filtering model to generate the t_2 uncertainty probability value that estimates the uncertainty of the t-th predicted object motion data.

[0026] In one example, in the (III) process, the t vehicle system data includes a t vehicle speed, which is the speed of the vehicle in the t image frame, a t vehicle acceleration, which is the acceleration of the vehicle in the t image frame, a t vehicle yaw rate, which is the yaw rate of the vehicle in the t image frame, and a t time difference, which is the difference between the time when the (t-1) image frame was acquired and the time when the t image frame was acquired.

[0027] In one example, in the process (III), the state prediction model is configured with a motion model including at least one of a constant velocity model and a constant acceleration model.

[0028] In one example, in the process (III), each of the first filtering model and the second filtering model is composed of any one of a recurrent neural network (RNN), a long short-term memory (LSTM), and a gated recurrent unit (GRU).

[0029] In one example, in the process (I), the processor uses the data preprocessing module to refer to the t vision sensor data and the t vehicle system data to generate, as the t object motion estimation vector, a t object relative distance estimation vector including t object relative distance estimation data, a t object absolute velocity estimation vector including t object absolute velocity estimation data, and a t object absolute acceleration estimation vector including t object absolute acceleration estimation data; and in the process (II), the processor uses the sequential regression network to apply recurrent operations to the t object relative distance estimation vector through an object relative distance estimation model to calculate a distance from the vehicle in the t image frame. a t-th predicted object relative distance that predicts the relative distance of the object from the t-th predicted object relative distance; a t-th predicted object absolute velocity that predicts the absolute velocity of the object in the t-th image frame that is generated by applying recurrent operations to the t-th object absolute velocity estimation vector through an object absolute velocity estimation model; and a t-th predicted object absolute acceleration that predicts the absolute acceleration of the object in the t-th image frame that is generated by applying recurrent operations to the t-th object absolute acceleration estimation vector through an object absolute acceleration estimation model, thereby generating the t-th predicted object motion data including the t-th predicted object relative distance, the t-th predicted object absolute velocity, and the t-th predicted object absolute acceleration.

[0030] In one example, in the process (I), the processor causes the data pre-processing module to generate the t object motion estimation vector, which includes at least a portion of the t original object relative distance, the t original object relative velocity which is the relative velocity of the object with respect to the vehicle in the t image frame, the t original object absolute velocity which is the absolute velocity of the object in the t image frame, the t vehicle velocity which is the velocity of the vehicle in the t image frame, the t vehicle acceleration which is the acceleration of the vehicle in the t image frame, the t vehicle yaw rate which is the yaw rate of the vehicle in the t image frame, the t object heading angle, and the t time difference which is the difference between the time when the (t-1) image frame was acquired and the time when the t image frame was acquired, and the t original object relative distance, the t original object relative velocity, the t original object absolute velocity, the t vehicle speed, the t vehicle acceleration, the t vehicle yaw rate, the t object heading angle, and the t time difference are included in the t vision sensor data and the t vehicle system data, or are calculated with reference to them.

[0031] In one example, in the process (I), the processor, by using the data preprocessing module, references the t vision sensor data and the t vehicle system data to generate, as the t object motion estimation vector, (i) a t object relative distance estimation vector including the t original object relative distance, the t original object relative velocity, the t object heading angle, and the t time difference; (ii) a t object absolute velocity estimation vector including the t original object relative distance, the t original object absolute velocity, the t vehicle velocity, the t vehicle yaw rate, the t object heading angle, and the t time difference; and (iii) the t original object relative distance, the t original object absolute velocity, the t vehicle velocity, the t vehicle acceleration, the t vehicle yaw rate, and the t object heading angle. and generate a t-th predicted object absolute acceleration estimation vector including the t-th predicted object relative distance, the t-th predicted object absolute velocity, and the t-th predicted object absolute acceleration; and in process (II), using the sequential regression network, apply recurrent operations to the t-th object relative distance estimation vector through an object relative distance estimation model to generate a t-th predicted object relative distance that predicts the relative distance of the object from the vehicle in the t-th image frame, apply recurrent operations to the t-th object absolute velocity estimation vector through an object absolute velocity estimation model to generate a t-th predicted object absolute velocity that predicts the absolute velocity of the object in the t-th image frame, and apply recurrent operations to the t-th object absolute acceleration estimation vector through an object absolute acceleration estimation model to generate a t-th predicted object absolute acceleration that predicts the absolute acceleration of the object in the t-th image frame, thereby generating the t-th predicted object motion data including the t-th predicted object relative distance, the t-th predicted object absolute velocity, and the t-th predicted object absolute acceleration.

[0032] In one example, in the process (I), the processor causes the data pre-processing module to generate the t object motion estimation vector such that each of the t object relative distance estimation vector, the t object absolute velocity estimation vector, and the t object absolute acceleration estimation vector further includes a t vehicle pitch, which is the pitch of the vehicle in the t image frame, and a t vehicle roll, which is the roll of the vehicle in the t image frame.

[0033] In one example, in the process (II), the sequential regression network is configured with any one model of a recurrent neural network (RNN), a long short-term memory (LSTM), and a gated recurrent unit (GRU). [Effects of the Invention]

[0034] According to the present invention, physical quantities such as velocity and acceleration can be stably estimated from distance information of an object detected by a vision sensor.

[0035] According to the present invention, stable motion data of an object required for ADAS control can be provided from distance information of the object acquired based on a vision sensor.

[0036] According to the present invention, the ADAS of a vehicle can operate stably using the stable velocity and acceleration of an object estimated from distance information of the object detected based on a vision sensor. [Brief explanation of the drawings]

[0037] The following drawings attached for use in explaining embodiments of the present invention are only a part of the embodiments of the present invention, and a person having ordinary knowledge in the technical field to which the present invention pertains (hereinafter referred to as "ordinary engineer") can derive other drawings based on these drawings without performing any inventive work. FIG. 1 is a diagram illustrating an object motion estimation apparatus for estimating the motion of an object based on a vision sensor according to an embodiment of the present invention.

[0038] FIG. 2 is a diagram illustrating a schematic configuration of a deep learning network of an object motion estimation device that estimates the motion of an object based on a vision sensor according to an embodiment of the present invention. FIG. 3 is a diagram illustrating a schematic diagram of a method for estimating object motion based on a vision sensor according to an embodiment of the present invention.

[0039] FIG. 4 is a diagram illustrating a detailed process of data pre-processing and generating t-th predicted object motion data in a method for estimating object motion based on a vision sensor according to an embodiment of the present invention.

[0040] FIG. 5 is a diagram illustrating an example of a deep learning model used for estimating the motion of an object in a method for estimating the motion of an object based on a vision sensor according to an embodiment of the present invention.

[0041] FIG. 6 is a diagram illustrating a process of generating corrected object motion data by correcting predicted object motion data in a method for estimating object motion based on a vision sensor according to an embodiment of the present invention. FIG. 7 is a diagram illustrating a process for generating a state vector used to correct predicted object motion data according to an embodiment of the present invention.

[0042] FIG. 8 is a diagram illustrating a process for generating training data for use in training sequential regression networks and sequential filtering networks according to the present invention.

[0043] FIG. 9 is a diagram illustrating an example of selecting targets in an image frame in the process of generating training data for use in training a sequential regression network and a sequential filtering network according to the present invention.

[0044] FIG. 10 is a diagram illustrating the preprocessing of sequential data in the process of generating training data for use in training the sequential regression network and sequential filtering network according to the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0045] The following detailed description of the present invention refers to the accompanying drawings, which show, by way of illustration, specific embodiments in which the invention may be practiced. These embodiments are described in sufficient detail to enable one of ordinary skill in the art to practice the invention. It should be understood that although the various embodiments of the present invention are different from one another, they are not necessarily mutually exclusive. For example, specific shapes, structures, and characteristics described herein may be varied and implemented from one embodiment to another without departing from the spirit and scope of the present invention. It should also be understood that the location or arrangement of individual components within each embodiment may be changed without departing from the spirit and scope of the present invention. Therefore, the following detailed description should not be taken in a limiting sense, and the scope of the present invention should be understood to encompass the scope of the appended claims and all equivalents thereto. In the drawings, like reference numerals indicate the same or similar components throughout the various aspects.

[0046] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS In the following, a number of preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings so that those skilled in the art can easily implement the present invention.

[0047] FIG. 1 is a diagram illustrating an object motion estimation device for estimating the motion of an object based on a vision sensor according to one embodiment of the present invention. Referring to FIG. 1, the object motion estimation device 1000 may include a memory 1100 storing instructions for estimating the motion of an object based on a vision sensor, and a processor 1200 that performs operations for estimating the motion of an object based on the vision sensor in accordance with the instructions stored in the memory 1100.

[0048] In particular, the object motion estimation device 1000 may typically, but is not limited to, use a combination of computing devices (e.g., devices that may include a computer processor, memory, storage, input and output devices, and other conventional computing device components; electronic communication devices such as routers, switches, etc.; electronic information storage systems such as network-attached storage (NAS) and storage area networks (SAN)) and computer software (i.e., instructions that cause a computing device to function in a particular manner) to achieve the desired system performance.

[0049] The processor 1200 of the object motion estimation device 1000 may include hardware components such as a micro processing unit (MPU) or a central processing unit (CPU), a cache memory, a data bus, etc. The computing device may also include software components such as an operating system and an application that executes a specific purpose.

[0050] However, this does not exclude the case where the object motion estimation device 1000 includes an integrated processor in which a medium, processor, and memory for implementing the present invention are integrated.

[0051] Meanwhile, the processor 1200 of the object motion estimation device 1000 can execute, in accordance with the instructions stored in the memory 1100, a process in which, when vision sensor data including an original object relative distance, which is the distance from the vehicle to the object, and an object heading angle, which is the heading angle of the object, detected through a deep learning-based detection model from at least one image taken by at least one camera installed in the vehicle, and vehicle system data detected from the vehicle's system, are sequentially acquired, the t-th vision sensor data and the t-th vehicle system data corresponding to the current image frame, the t-th image frame, are input into a data pre-processing module, and the data pre-processing module pre-processes the t-th vision sensor data and the t-th vehicle system data to generate a t-th object motion estimation vector for estimating the motion of the object. The processor 1200 of the object motion estimation device 1000 can execute a process in accordance with the instructions stored in the memory 1100, such as inputting the t-th object motion estimation vector into a sequential regression network based on deep learning, and using the sequential regression network to perform a recurrent learning operation on the t-th object motion estimation vector to generate t-th predicted object motion data that predicts the motion of the object corresponding to the t-th image frame.

[0052] In addition, the processor 1200 of the object motion estimation device 1000, in accordance with the instructions stored in the memory 1100, inputs the t-th predicted object motion data and the t-th vehicle system data to a sequential filtering network, and uses the sequential filtering network to (i) generate (t-1)-th corrected object motion data by correcting the (t-1)-th predicted object motion data in the (t-1)-th image frame through a state prediction model, i.e., (t-1)-th corrected object motion data which is object motion data obtained by correcting the (t-1)-th predicted object motion data with reference to the (t-1)-th predicted object motion data and the (t-1)-th vehicle system data, and (t-1) predicted corrected object motion data which predicts the corrected object motion data in the t-th image frame by applying a learning operation to the t-th vehicle system data; and (ii) generate (t-1)-th predicted object motion data and the (t-1)-th vehicle system data through a state vector generation module. ) generating a t-th predicted differential state vector based on the difference with the predicted object motion data, a t-th predicted-corrected differential state vector based on the difference between the t-th predicted object motion data and the t-th predicted-corrected object motion data, and a t-th corrected differential state vector based on the difference between the t-th predicted-corrected object motion data and the (t-1)th corrected object motion data; (iii) applying a recurrent operation to the t-th corrected differential state vector through a first filtering model to generate a t_1th uncertain probability value that estimates the uncertainty of the t-th predicted-corrected object motion data; applying a recurrent operation to the t-th predicted differential state vector and the t-th predicted-corrected differential state vector through a second filtering model to generate a t_2th uncertain probability value that estimates the uncertainty of the t-th predicted object motion data; (iv) applying the t-th feedback gain generated by referring to the t_1st uncertain probability value and the t_2nd uncertain probability value to the t-th predicted-corrected differential state vector to correct the t-th predicted-corrected object motion data, therebyA process may further be performed to generate t-th corrected object motion data in the t-th image frame.

[0053] A method for estimating the motion of an object based on a vision sensor in the object motion estimation device 1000 configured as above will be described in more detail below with reference to FIGS. 2 and 3. FIG.

[0054] First, the object motion estimation device 1000 can sequentially acquire (S110) vision sensor data including an original object relative distance, which is the distance from the vehicle to the object, and an object heading angle, which is the heading angle of the object, detected through a detection model based on deep learning from at least one image captured by at least one camera installed in the vehicle, and vehicle system data detected from the vehicle's system.

[0055] In this case, the detection model may be a deep learning model trained to detect objects from images captured by a camera installed in the vehicle and output the bounding box, class information, original object relative distance, and object heading angle of the detected object. Various learning methods can be used to train the detection model, such as a supervised learning method using training images annotated with the object's bounding box, class information, original object relative distance, and object heading angle, a learning method that fuses sensing information such as radar and lidar with training images, and a learning method that uses training data having consecutive image frames.

[0056] The detection model may also be a deep learning model trained to detect 3D objects from 2D images captured by a monocular camera installed on a vehicle and output the bounding box, class information, relative distance of the original object, and object heading angle of the detected 3D object. Various learning methods can be used to train the detection model, such as a learning method using training images annotated with 3D object information corresponding to 2D images, or a learning method using lidar sensing information, i.e., lidar images.

[0057] Meanwhile, in the above, vision sensor data was acquired by training a conventional detection model that detects an object and outputs the object's bounding box and class information to additionally output the original object relative distance and object heading angle. However, alternatively, vision sensor data can also be acquired using another deep learning model that is trained to track an object in successive image frames by referring to the results of outputting the object's bounding box and class information through the conventional detection model, and estimate the object's original object relative distance and object heading angle.

[0058] Next, the object motion estimation device 1000 can preprocess the t-th vision sensor data and t-th vehicle system data corresponding to the t-th image frame from the sequentially acquired vision sensor data and vehicle system data to generate a t-th object motion estimation vector (S120).

[0059] In other words, the object motion estimation device 1000 can input the t-th vision sensor data and t-th vehicle system data corresponding to the current image frame, the t-th image frame, from the sequentially acquired vision sensor data and vehicle system data to the data pre-processing module 1300, and use the data pre-processing module 1300 to pre-process the t-th vision sensor data and t-th vehicle system data to generate the t-th object motion estimation vector for estimating the motion of the object.

[0060] As an example, referring to FIG. 4, the object motion estimation device 1000 can use the data preprocessing module 1300 to refer to the t-th vision sensor data and the t-th vehicle system data to generate, as the t-th object motion estimation vector, a t-th object relative distance estimation vector including the t-th object relative distance estimation data, a t-th object absolute velocity estimation vector including the t-th object absolute velocity estimation data, and a t-th object absolute acceleration estimation vector including the t-th object absolute acceleration estimation data.

[0061] Meanwhile, object motion estimation device 1000 can use data preprocessing module 1300 to generate a t-th object motion estimation vector including at least some of the t-th original object relative distance, the t-th original object relative velocity (which is the velocity of the object relative to the vehicle in the t-th image frame), the t-th original object absolute velocity (which is the absolute velocity of the object in the t-th image frame), the t-th vehicle velocity (which is the velocity of the vehicle in the t-th image frame), the t-th vehicle acceleration (which is the acceleration of the vehicle in the t-th image frame), the t-th vehicle yaw rate (which is the yaw rate of the vehicle in the t-th image frame), the t-th object heading angle, and the t-th time difference (which is the difference between the time the (t-1)th image frame was captured and the time the t-th image frame was captured). In this case, the t-th original object relative distance, the t-th original object relative velocity, the t-th original object absolute velocity, the t-th vehicle velocity, the t-th vehicle acceleration, the t-th vehicle yaw rate, the t-th object heading angle, and the t-th time difference may be included in the t-th vision sensor data and the t-th vehicle system data, or may be calculated with reference to these. Furthermore, in an embodiment according to the present invention, the distance, velocity, and acceleration associated with an object or vehicle may be scalar values ​​representing distance values, velocity, and acceleration, or may be vector values ​​that further include directionality.

[0062] As an example, the object motion estimation device 1000 can use the data pre-processing module 1300 to refer to the t vision sensor data and the t vehicle system data to generate, as the t object motion estimation vectors, (i) a t object relative distance estimation vector including the t original object relative distance, the t original object relative velocity, the t object heading angle, and the t time difference, which is the difference between the time the (t-1) image frame was acquired and the time the t image frame was acquired; (ii) a t object absolute velocity estimation vector including the t original object relative distance, the t original object absolute velocity, the t vehicle speed, the t vehicle yaw rate, the t object heading angle, and the t time difference; and (iii) a t object absolute acceleration estimation vector including the t original object relative distance, the t original object absolute velocity, the t vehicle speed, the t vehicle acceleration, the t vehicle yaw rate, the t object heading angle, and the t time difference.

[0063] Furthermore, the object motion estimation apparatus 1000 may generate the t-th object motion estimation vector using the data preprocessing module 1300, such that each of the t-th object relative distance estimation vector, the t-th object absolute velocity estimation vector, and the t-th object absolute acceleration estimation vector further includes the t-th vehicle pitch, which is the pitch of the vehicle in the t-th image frame, and the t-th vehicle roll, which is the roll of the vehicle in the t-th image frame. In this case, the pitch and roll of the vehicle may be obtained using vehicle system data detected through an acceleration sensor, tilt sensor, angular velocity sensor, etc. of the vehicle, or may be obtained by analyzing consecutive image frames captured by a camera.

[0064] Next, referring again to FIGS. 2 and 3, the object motion estimation device 1000 can perform recurrent calculations on the t-th object motion estimation vector to generate the t-th predicted object motion data corresponding to the t-th image frame (S130).

[0065] In other words, the object motion estimation device 1000 can input the t-th object motion estimation vector into a sequential regression network 1400 based on deep learning, and use the sequential regression network 1400 to perform a recurrent learning operation on the t-th object motion estimation vector to generate t-th predicted object motion data that predicts the motion of the object corresponding to the t-th image frame.

[0066] As an example, referring again to FIG. 4 , the object motion estimation apparatus 1000 uses a sequential regression network 1400 to apply recurrent operations to the t-th object relative distance estimation vector via the object relative distance estimation model 1410 to generate the t-th predicted object relative distance that predicts the relative distance of the object from the vehicle in the t-th image frame, apply recurrent operations to the t-th object absolute velocity estimation vector via the object absolute velocity estimation model 1420 to generate the t-th predicted object absolute velocity that predicts the absolute velocity of the object in the t-th image frame, and apply recurrent operations to the t-th object absolute acceleration estimation vector via the object absolute acceleration estimation model 1430 to generate the t-th predicted object absolute acceleration that predicts the absolute acceleration of the object in the t-th image frame, thereby generating the t-th predicted object motion data including the t-th predicted object relative distance, the t-th predicted object absolute velocity, and the t-th predicted object absolute acceleration.

[0067] In this case, the sequential regression network 1400, for example, the object relative distance estimation model 1410, the object absolute velocity estimation model 1420, and the object absolute acceleration estimation model 1430 included in the sequential regression network 1400 can each be configured using any one of a recurrent neural network (RNN), a long short-term memory (LSTM), and a gated recurrent unit (GRU).

[0068] As an example, FIG. 5 is a diagram illustrating a sequential regression network 1400 configured with a GRU 1450, which is a model for processing sequential data.

[0069] The t-th object motion estimation vector x generated by the data preprocessing module t When this signal is input, the reset gate r t is the t-th motion estimation vector x t and the previous hidden state h t-1 By performing a FC (Fully Connected) calculation, it is possible to determine how much of the previous state should be retained in the current state, that is, the t-th motion estimation vector, and this can be expressed by the following equation. JPEG0007766220000002.jpg763

[0070] JPEG0007766220000003.jpg30170

[0071] Also, GRU1450 Update Z t is the t-th motion estimation vector x t and the previous hidden state h t-1 By performing a Fully Connected (FC) operation, the balance between the previous state information, i.e., the previous hidden state, and the new information, i.e., the t-th motion estimation vector, can be determined, which can be expressed mathematically as follows: JPEG0007766220000004.jpg763

[0072] JPEG0007766220000005.jpg25170At this time, z t determines the rate of application of the previous information, and accordingly, the rate of application of the current information is 1-z t It may be determined by: Then the GRU1450 will turn on the reset gate t and Update Z t Both can be applied to compute the hidden state.

[0073] JPEG0007766220000006.jpg53170JPEG0007766220000007.jpg883JPEG0007766220000008.jpg763

[0074] JPEG0007766220000009.jpg26170

[0075] Therefore, according to the present invention, by performing recurrent calculations on vision sensor data and vehicle system data related to an object detected based on a vision sensor and regressing object motion data, it is possible to obtain stable physical quantities through analysis of images captured by a single camera without incurring minute errors in the estimated distance to the object or amplification of noise.

[0076] On the other hand, the object motion data regressed by the above method may contain temporal jitter, such as irregular vibrations and distortions that may occur due to the transmission time and period of the vision sensor data and vehicle system data.

[0077] Therefore, in order to obtain more stable object motion data for use in ADAS and the like, additional corrections can be made to the predicted object motion data predicted by regression.

[0078] Therefore, the object motion estimation device 1000 can use the t-th predicted object motion data and the t-th vehicle system data to generate the t-th corrected object motion data (S140) by correcting the temporal instability of the t-th predicted object motion data.

[0079] In other words, object motion estimation device 1000 inputs the t-th predicted object motion data and the t-th vehicle system data output from sequential regression network 1400 into sequential filtering network 1500, and uses sequential filtering network 1500 to perform recurrent calculations on the t-th predicted object motion data and the t-th vehicle system data to correct the t-th predicted object motion data and generate the t-th corrected object motion data, which is a more stable physical quantity.

[0080] As an example, referring to Figures 6 and 7, the object motion estimation device 1000 inputs the t-th predicted object motion data and the t-th vehicle system data to a sequential filtering network 1500, and uses the sequential filtering network 1500 to apply a learning operation to the (t-1)th corrected object motion data and the t-th vehicle system data, which are obtained by correcting the (t-1)th predicted object motion data in the (t-1)th image frame, through a state prediction model 1510, to generate t-th predicted corrected object motion data that predicts the corrected object motion data in the t-th image frame.

[0081] In this case, the (t-1)th corrected object motion data may be object motion data obtained by correcting the (t-1)th predicted object motion data by referring to the (t-1)th predicted object motion data and the (t-1)th vehicle system data. The t-th predicted object motion data may include the t-th predicted object relative distance, the t-th predicted object absolute velocity, and the t-th predicted object absolute acceleration, and the t-th vehicle system data may include the t-th vehicle speed, which is the speed of the vehicle in the t-th image frame, the t-th vehicle acceleration, which is the acceleration of the vehicle in the t-th image frame, the t-th vehicle yaw rate, which is the yaw rate of the vehicle in the t-th image frame, and the t-th time difference, which is the difference between the time the (t-1)th image frame was acquired and the time the t-th image frame was acquired.

[0082] The object motion estimation device 1000 can also use the sequential filtering network 1500 to apply a learning operation to the (t-1)th corrected object motion data, the tth vehicle system data, and the tth predicted object relative distance included in the tth predicted object motion data through the state prediction model 1510 to generate the tth predicted corrected object motion data.

[0083] On the other hand, the state prediction model may be configured as a motion model including at least one of a constant velocity model and a constant acceleration model, but the present invention is not limited to this and may be configured as various motion estimation models that estimate the motion of objects in image frames based on deep learning.

[0084] The object motion estimation device 1000 can use the sequential filtering network 1500 to generate, via the state vector generation module 1520, a t-th predicted differential state vector based on the difference between the t-th predicted object motion data and the (t-1)-th predicted object motion data, a t-th predicted-corrected differential state vector based on the difference between the t-th predicted object motion data and the t-th predicted-corrected object motion data, and a t-th corrected differential state vector based on the difference between the t-th predicted-corrected object motion data and the (t-1)-th corrected object motion data.

[0085] Thereafter, the object motion estimation apparatus 1000 may use the sequential filtering network 1500 to apply a recurrent operation to the t-th corrected differential state vector through the first filtering model 1530 to generate a t_1-th uncertainty probability value that estimates the uncertainty of the t-th predicted corrected object motion data, and may apply a recurrent operation to the t-th predicted differential state vector and the t-th predicted-corrected differential state vector through the second filtering model 1540 to generate a t_2-th uncertainty probability value that estimates the uncertainty of the t-th predicted object motion data. In this case, each of the first filtering model and the second filtering model may be configured with any one of a recurrent neural network (RNN), a long short-term memory (LSTM), and a gated recurrent unit (GRU).

[0086] On the other hand, the object motion estimation device 1000 may use a sequential filtering network 1500 to further generate a t-th object distance state vector corresponding to the t-th predicted object relative distance included in the t-th predicted object motion data through a state vector generation module 1520, apply recurrent operations to the t-th corrected differential state vector and the t-th object distance state vector through a first filtering model 1530 to generate a t_1-th uncertainty probability value that estimates the uncertainty of the t-th predicted corrected object motion data, and apply recurrent operations to the t-th predicted differential state vector, the t-th predicted-corrected differential state vector, and the t-th object distance state vector through a second filtering model 1540 to generate a t_2-th uncertainty probability value that estimates the uncertainty of the t-th predicted object motion data.

[0087] The object motion estimation device 1000 can then use the sequential filtering network 1500 to generate the t-th corrected object motion data in the t-th image frame by applying the t-th feedback gain, generated by referring to the t_1-th uncertain probability value and the t_2-th uncertain probability value, to the t-th predicted-corrected differential state vector to correct the t-th predicted corrected object motion data.

[0088] Therefore, according to the present invention, by removing temporal jitter from the object motion data regressed in the sequential regression network 1400, it is possible to provide more stable object motion data for use in ADAS and the like.

[0089] On the other hand, the sequential regression network 1400 and the sequential filtering network 1500 according to the present invention may be trained individually using training data, or may be trained simultaneously using an end-to-end method, or may be trained by supervised learning using training data.

[0090] The process of generating training data for use in training the sequential regression network 1400 and the sequential filtering network 1500 according to the present invention will be described below with reference to FIG.

[0091] First, to collect learning data, images captured by a camera installed on a real vehicle while the vehicle is traveling, i.e., image frames, real vehicle system data sensed by a sensor installed on the real vehicle, and radar data related to surrounding objects detected by a radar installed on the real vehicle can be acquired (S210).

[0092] Then, a target, which is any one of the objects detected through image analysis of the image frame, can be selected (S220).

[0093] At this time, referring to FIG. 9, the target (CIPV, Closest In-path Vehicle) is the closest vehicle on the travel path of the actual vehicle, and may be a reference object for ADAS control. The camera-radar tracks for the target can then be matched (S230).

[0094] In other words, by synchronizing the image frames captured by the camera with the radar data relating to the target detected by the radar, the image data and radar data for the target can be matched on the time axis.

[0095] Then, based on the vehicle system data and radar data, original sequential data including range, velocity, and acceleration for the target can be generated (S240). Then, data preprocessing and augmentation (S250) can be performed on the original sequential data to generate training data.

[0096] In other words, the original sequential data can be subjected to preprocessing such as sampling and noise removal, and then various methods of data augmentation can be performed to improve the generalization performance of the model.

[0097] As an example, referring to Figure 10, when original sequential data is input (a), the original sequential data is grouped into sequential data whose number corresponds to a set range of minimum to maximum image frame numbers based on the data vector of the original sequential data, thereby generating multiple group sequential data GS1, GS2, and GS3 (b). At this time, the number of data included in each of the multiple sequential data may be the same or different. Note that, for convenience of explanation, Figure 10 shows three group sequential data GS1, GS2, and GS3.

[0098] Then, a sliding window technique is applied to each of the plurality of group sequential data to generate a plurality of sliding window data SW1_1 to SW3_n corresponding to each of the plurality of group sequential data (c). That is, for one group sequential data, a window having a predetermined time interval is moved through the group sequential data, and the portion of the group sequential data within the window is generated as each sliding window data. In this case, in Fig. 10, each of the sliding window data corresponding to the three group sequential data is shown as being the same, that is, n pieces of sliding window data, but this is for convenience of explanation, and the number of sliding window data generated according to the number of data included in each group sequential data may be the same or different.

[0099] Thereafter, each of the plurality of sliding window data SW1_1 to SW3_n is converted into tensor data to generate a window tensor (d), and then filtering (e) is performed to remove sliding window data that does not satisfy a predetermined minimum or maximum number of data from the plurality of sliding window data SW1_1 to SW3_n, thereby generating a training data set. Alternatively, a training data set can be generated by removing sliding window data that does not satisfy a predetermined minimum or maximum number of data from the plurality of sliding window data SW1_1 to SW3_n, and then generating the remaining sliding window data as a window tensor.

[0100] In addition, to eliminate imbalances in distance and speed, data included in multiple sequential data may be binned by distance and speed, an average distance value or an average speed value of the data included in each binned group may be calculated, and the distance or speed of the data included in each binned group may be replaced with the calculated average distance value or average speed value. Additionally, in order to simulate various driving situations, when there is no gradually approaching object, data may be augmented by inverting the signs of the speed and acceleration of a gradually receding object or correcting the heading angle of the object.

[0101] The above-described embodiments of the present invention may be embodied in the form of program instructions that can be executed by various computer components and stored on a computer-readable storage medium. The computer-readable storage medium may include, alone or in combination, program instructions, data files, data structures, and the like. The program instructions stored on the computer-readable storage medium may be specially designed and constructed for the present invention, or may be well known and available to those skilled in the computer software arts. Examples of computer-readable storage media include magnetic media such as hard disks, floppy disks, and magnetic tape; optical media such as CD-ROMs and DVDs; magneto-optical media such as floptical disks; and hardware devices specially configured to store and execute program instructions, such as ROM, RAM, and flash memory. Examples of program instructions include not only machine code, such as produced by a compiler, but also high-level language code that can be executed by a computer using an interpreter, etc. The hardware devices may be configured to operate as one or more software modules to perform processes according to the present invention, or vice versa.

[0102] Although the present invention has been described above using specific details such as concrete components and limited examples and drawings, these are merely provided to facilitate a more general understanding of the present invention, and the present invention is not limited to the above examples. Those skilled in the art will be able to make various modifications and variations from such descriptions.

[0103] Therefore, the spirit of the present invention should not be limited to the above-described embodiments, and all modifications equivalent to or equivalent to the scope of the claims, as well as the scope of the claims, are within the spirit of the present invention. [Explanation of symbols]

[0104] 1000: Object motion estimation device 1100:Memory 1200: Processor

Claims

1. 1. A method for estimating motion of an object based on a vision sensor, comprising: (a) when vision sensor data, including an original object relative distance, which is a distance from the vehicle to an object, and an object heading angle, which is a heading angle of the object, detected through a detection model based on deep learning from at least one image taken by at least one camera installed in a vehicle, and vehicle system data detected from a system of the vehicle, are sequentially acquired, the object motion estimation device inputs the t-th vision sensor data and the t-th vehicle system data corresponding to the t-th image frame, which is a current image frame, into a data pre-processing module, and causes the data pre-processing module to pre-process the t-th vision sensor data and the t-th vehicle system data to generate a t-th object motion estimation vector for estimating the motion of the object; and (b) the object motion estimation device inputs the t object motion estimation vector into a sequential regression network based on deep learning, and uses the sequential regression network to perform a recurrent learning operation on the t object motion estimation vector to generate t predicted object motion data that predicts the motion of the object corresponding to the t image frame; A method comprising:

2. (c) the object motion estimation device inputs the t predicted object motion data and the t vehicle system data into a sequential filtering network, and uses the sequential filtering network to (i) generate (t-1) corrected object motion data obtained by correcting the (t-1) predicted object motion data in the (t-1) image frame through a state prediction model (the (t-1) corrected object motion data is object motion data obtained by correcting the (t-1) predicted object motion data with reference to the (t-1) predicted object motion data and the (t-1) vehicle system data), and t predicted corrected object motion data obtained by predicting the corrected object motion data in the t image frame by applying a learning operation to the t vehicle system data; and (ii) generate a state vector based on a difference between the t predicted object motion data and the (t-1) predicted object motion data through a state vector generation module. (iii) applying a recurrent operation to the t-th corrected differential state vector through a first filtering model to generate a t_1th uncertainty probability value that estimates the uncertainty of the t-th predicted corrected object motion data; and applying a recurrent operation to the t-th predicted differential state vector and the t-th predicted-corrected differential state vector through a second filtering model to generate a t_2th uncertainty probability value that estimates the uncertainty of the t-th predicted object motion data; (iv) applying a t-th feedback gain generated with reference to the t_1st uncertainty probability value and the t_2nd uncertainty probability value to the t-th predicted-corrected differential state vector to correct the t-th predicted corrected object motion data, therebygenerating t-th corrected object motion data in the t-th image frame; The method of claim 1 further comprising:

3. In the step (c), 3. The method of claim 2, wherein the object motion estimation device further generates, through the state vector generation module, a t-th object distance state vector corresponding to the t-th predicted object relative distance included in the t-th predicted object motion data, the t_1 uncertainty probability value estimating uncertainty of the t-th predicted corrected object motion data by applying a recurrent operation to the t-th corrected differential state vector and the t-th object distance state vector by using the sequential filtering network, and the t_2 uncertainty probability value estimating uncertainty of the t-th predicted object motion data by applying a recurrent operation to the t-th predicted differential state vector, the t-th predicted-corrected differential state vector, and the t-th object distance state vector by using the second filtering model.

4. 3. The method of claim 2, wherein in step (c), the t vehicle system data includes a t vehicle speed, which is the speed of the vehicle in the t image frame, a t vehicle acceleration, which is the acceleration of the vehicle in the t image frame, a t vehicle yaw rate, which is the yaw rate of the vehicle in the t image frame, and a t time difference, which is the difference between the time when the (t-1) image frame was acquired and the time when the t image frame was acquired.

5. The method according to claim 2 , wherein in step (c), the state prediction model is configured as a motion model including at least one of a constant velocity model and a constant acceleration model.

6. 3. The method of claim 2, wherein in step (c), each of the first filtering model and the second filtering model is configured with any one of a recurrent neural network (RNN), a long short-term memory (LSTM), and a gated recurrent unit (GRU).

7. In the step (a), the object motion estimation device uses the data pre-processing module to refer to the t vision sensor data and the t vehicle system data, and generate, as the t object motion estimation vectors, a t-th object relative distance estimation vector including t-th object relative distance estimation data, a t-th object absolute velocity estimation vector including t-th object absolute velocity estimation data, and a t-th object absolute acceleration estimation vector including t-th object absolute acceleration estimation data; In the step (b), 2. The method of claim 1 , wherein the object motion estimation device uses the sequential regression network to apply recurrent operations to the t object relative distance estimation vector through an object relative distance estimation model to generate a t-th predicted object relative distance that predicts the relative distance of the object from the vehicle in the t image frame, apply recurrent operations to the t object absolute velocity estimation vector through an object absolute velocity estimation model to generate a t-th predicted object absolute velocity that predicts the absolute velocity of the object in the t image frame, and apply recurrent operations to the t object absolute acceleration estimation vector through an object absolute acceleration estimation model to generate a t-th predicted object absolute acceleration that predicts the absolute acceleration of the object in the t image frame, thereby generating the t-th predicted object motion data including the t-th predicted object relative distance, the t-th predicted object absolute velocity, and the t-th predicted object absolute acceleration.

8. In the step (a), the object motion estimation device uses the data pre-processing module to generate the t object motion estimation vector, which includes at least a portion of a t original object relative distance, a t original object relative velocity which is the relative velocity of the object with respect to the vehicle in the t image frame, a t original object absolute velocity which is the absolute velocity of the object in the t image frame, a t vehicle velocity which is the velocity of the vehicle in the t image frame, a t vehicle acceleration which is the acceleration of the vehicle in the t image frame, a t vehicle yaw rate which is the yaw rate of the vehicle in the t image frame, a t object heading angle, and a t time difference which is the difference between the time when the (t-1) image frame was captured and the time when the t image frame was captured; The method of claim 1, wherein the t original object relative distance, the t original object relative velocity, the t original object absolute velocity, the t vehicle velocity, the t vehicle acceleration, the t vehicle yaw rate, the t object heading angle, and the t time difference are included in the t vision sensor data and the t vehicle system data, or are calculated with reference to these.

9. In the step (a), the object motion estimation device uses the data pre-processing module to refer to the t vision sensor data and the t vehicle system data to generate, as the t object motion estimation vectors, (i) a t-th object relative distance estimation vector including the t original object relative distance, the t original object relative velocity, the t object heading angle, and the t time difference, (ii) a t-th object absolute velocity estimation vector including the t original object relative distance, the t original object absolute velocity, the t vehicle velocity, the t vehicle yaw rate, the t object heading angle, and the t time difference, and (iii) a t-th object absolute acceleration estimation vector including the t original object relative distance, the t original object absolute velocity, the t vehicle velocity, the t vehicle acceleration, the t vehicle yaw rate, the t object heading angle, and the t time difference; In the step (b), 9. The method of claim 8, wherein the object motion estimation device uses the sequential regression network to apply recurrent operations to the t object relative distance estimation vector through an object relative distance estimation model to generate a t-th predicted object relative distance that predicts the relative distance of the object from the vehicle in the t image frame, apply recurrent operations to the t object absolute velocity estimation vector through an object absolute velocity estimation model to generate a t-th predicted object absolute velocity that predicts the absolute velocity of the object in the t image frame, and apply recurrent operations to the t object absolute acceleration estimation vector through an object absolute acceleration estimation model to generate a t-th predicted object absolute acceleration that predicts the absolute acceleration of the object in the t image frame, thereby generating the t-th predicted object motion data including the t-th predicted object relative distance, the t-th predicted object absolute velocity, and the t-th predicted object absolute acceleration.

10. In the step (a), 10. The method of claim 9, wherein the object motion estimation device uses the data pre-processing module to generate the t object motion estimation vector such that each of the t object relative distance estimation vector, the t object absolute velocity estimation vector, and the t object absolute acceleration estimation vector further includes a t vehicle pitch, which is the pitch of the vehicle in the t image frame, and a t vehicle roll, which is the roll of the vehicle in the t image frame.

11. In the step (b), The method of claim 1, wherein the sequential regression network is configured using any one of a recurrent neural network (RNN), a long short-term memory (LSTM), and a gated recurrent unit (GRU).

12. An object motion estimation device that estimates the motion of an object based on a vision sensor, a memory storing instructions for estimating the motion of an object based on the vision sensor; and a processor that performs operations according to the instructions stored in the memory to estimate motion of the object based on the vision sensor; Including, The processor executes the following process: (I) when vision sensor data, including an original object relative distance, which is the distance from the vehicle to the object, and an object heading angle, which is the heading angle of the object, detected through a detection model based on deep learning from at least one image taken by at least one camera installed in the vehicle, and vehicle system data detected from the vehicle's system, are sequentially acquired, input the t-th vision sensor data and the t-th vehicle system data corresponding to the t-th image frame, which is the current image frame, to a data pre-processing module, and use the data pre-processing module to pre-process the t-th vision sensor data and the t-th vehicle system data to generate a t-th object motion estimation vector for estimating the motion of the object; and (II) input the t-th object motion estimation vector to a sequential regression network based on deep learning, and use the sequential regression network to perform recurrent learning calculations on the t-th object motion estimation vector to generate t-th predicted object motion data that predicts the motion of the object corresponding to the t-th image frame.

13. The processor (III) inputs the t predicted object motion data and the t vehicle system data into a sequential filtering network, and uses the sequential filtering network to (i) generate (t-1) corrected object motion data by correcting the (t-1) predicted object motion data in the (t-1) image frame through a state prediction model (the (t-1) corrected object motion data is object motion data obtained by correcting the (t-1) predicted object motion data with reference to the (t-1) predicted object motion data and the (t-1) vehicle system data), and t predicted corrected object motion data by applying a learning operation to the t vehicle system data to predict the corrected object motion data in the t image frame; and (ii) generates, through a state vector generation module, a t predicted object motion data based on a difference between the t predicted object motion data and the (t-1) predicted object motion data. (iii) applying a recurrent operation to the t-th corrected differential state vector through a first filtering model to generate a t_1th uncertainty probability value that estimates the uncertainty of the t-th predicted corrected object motion data; and applying a recurrent operation to the t-th predicted differential state vector and the t-th predicted-corrected differential state vector through a second filtering model to generate a t_2th uncertainty probability value that estimates the uncertainty of the t-th predicted object motion data; (iv) applying a t-th feedback gain generated with reference to the t_1st uncertainty probability value and the t_2nd uncertainty probability value to the t-th predicted-corrected differential state vector to correct the t-th predicted corrected object motion data, therebyThe object motion estimation apparatus of claim 12, further comprising a process for generating t-th corrected object motion data in the t-th image frame.

14. 14. The object motion estimation device of claim 13, wherein in process (III), the processor uses the sequential filtering network to further generate, through the state vector generation module, a t-th object distance state vector according to the t-th predicted object relative distance included in the t-th predicted object motion data; to apply, through the first filtering model, a recurrent operation to the t-th corrected differential state vector and the t-th object distance state vector to generate the t_1 uncertainty probability value that estimates uncertainty of the t-th predicted corrected object motion data; and to apply, through the second filtering model, a recurrent operation to the t-th predicted differential state vector, the t-th predicted-corrected differential state vector, and the t-th object distance state vector to generate the t_2 uncertainty probability value that estimates uncertainty of the t-th predicted object motion data.

15. 14. The object motion estimation device of claim 13, wherein in process (III), the t vehicle system data includes a t vehicle speed, which is the speed of the vehicle in the t image frame, a t vehicle acceleration, which is the acceleration of the vehicle in the t image frame, a t vehicle yaw rate, which is the yaw rate of the vehicle in the t image frame, and a t time difference, which is the difference between the time when the (t-1) image frame was acquired and the time when the t image frame was acquired.

16. The object motion estimation device according to claim 13 , wherein in the process (III), the state prediction model is configured as a motion model including at least one of a constant velocity model and a constant acceleration model.

17. 14. The object motion estimation apparatus of claim 13, wherein in the process (III), each of the first filtering model and the second filtering model is configured with any one model of a recurrent neural network (RNN), a long short-term memory (LSTM), and a gated recurrent unit (GRU).

18. The processor: In the process (I), the data preprocessing module references the t vision sensor data and the t vehicle system data to generate, as the t object motion estimation vector, a t object relative distance estimation vector including t object relative distance estimation data, a t object absolute velocity estimation vector including t object absolute velocity estimation data, and a t object absolute acceleration estimation vector including t object absolute acceleration estimation data; 13. The object motion estimation device of claim 12, wherein in process (II), the sequential regression network applies recurrent operations to the t object relative distance estimation vector through an object relative distance estimation model to generate a t-th predicted object relative distance that predicts the relative distance of the object from the vehicle in the t image frame, applies recurrent operations to the t object absolute velocity estimation vector through an object absolute velocity estimation model to generate a t-th predicted object absolute velocity that predicts the absolute velocity of the object in the t image frame, and applies recurrent operations to the t object absolute acceleration estimation vector through an object absolute acceleration estimation model to generate a t-th predicted object absolute acceleration that predicts the absolute acceleration of the object in the t image frame, thereby generating the t-th predicted object motion data including the t-th predicted object relative distance, the t-th predicted object absolute velocity, and the t-th predicted object absolute acceleration.

19. 13. The object motion estimation device of claim 12, wherein in process (I), the processor causes the data pre-processing module to generate the t object motion estimation vector, which includes at least a part of the t original object relative distance, the t original object relative velocity which is the relative velocity of the object with respect to the vehicle in the t image frame, the t original object absolute velocity which is the absolute velocity of the object in the t image frame, the t vehicle velocity which is the velocity of the vehicle in the t image frame, the t vehicle acceleration which is the acceleration of the vehicle in the t image frame, the t vehicle yaw rate which is the yaw rate of the vehicle in the t image frame, the t object heading angle, and the t time difference which is the difference between the time when the (t-1)th image frame was acquired and the time when the t image frame was acquired, and the t original object relative distance, the t original object relative velocity, the t original object absolute velocity, the t vehicle speed, the t vehicle acceleration, the t vehicle yaw rate, the t object heading angle, and the t time difference are included in the t vision sensor data and the t vehicle system data, or are calculated with reference to them.

20. The processor: In the process (I), the data preprocessing module references the t vision sensor data and the t vehicle system data to generate, as the t object motion estimation vector, (i) a t object relative distance estimation vector including the t original object relative distance, the t original object relative velocity, the t object heading angle, and the t time difference, (ii) a t object absolute velocity estimation vector including the t original object relative distance, the t original object absolute velocity, the t vehicle velocity, the t vehicle yaw rate, the t object heading angle, and the t time difference, and (iii) a t object absolute acceleration estimation vector including the t original object relative distance, the t original object absolute velocity, the t vehicle velocity, the t vehicle acceleration, the t vehicle yaw rate, the t object heading angle, and the t time difference, 20. The object motion estimation device of claim 19, wherein in process (II), the sequential regression network applies recurrent operations to the t object relative distance estimation vector through an object relative distance estimation model to generate a t-th predicted object relative distance that predicts the relative distance of the object from the vehicle in the t image frame, applies recurrent operations to the t object absolute velocity estimation vector through an object absolute velocity estimation model to generate a t-th predicted object absolute velocity that predicts the absolute velocity of the object in the t image frame, and applies recurrent operations to the t object absolute acceleration estimation vector through an object absolute acceleration estimation model to generate a t-th predicted object absolute acceleration that predicts the absolute acceleration of the object in the t image frame, thereby generating the t-th predicted object motion data including the t-th predicted object relative distance, the t-th predicted object absolute velocity, and the t-th predicted object absolute acceleration.

21. 21. The object motion estimation device of claim 20, wherein in process (I), the processor causes the data pre-processing module to generate the t object motion estimation vector such that each of the t object relative distance estimation vector, the t object absolute velocity estimation vector, and the t object absolute acceleration estimation vector further includes a t vehicle pitch, which is the pitch of the vehicle in the t image frame, and a t vehicle roll, which is the roll of the vehicle in the t image frame.

22. 13. The object motion estimation apparatus of claim 12, wherein in the process (II), the sequential regression network is configured with any one model of a recurrent neural network (RNN), a long short-term memory (LSTM), and a gated recurrent unit (GRU).

Images