Apparatus for tracking object and method thereof

Abstract

The invention relates to an apparatus for tracking an object and a method thereof. The object tracking apparatus includes a processor configured to estimate a state variable of an object for each point in time in a first direction in a time stream, and to estimate a state variable for each point in time in a second direction opposite the first direction, and configured to estimate a state variable of the object in the second direction for each point in time using the result value estimated in the first direction; and a memory configured to store data and an algorithm driven by the processor.

Description

[0001] Cross - reference to related applications

[0002] This application claims the priority of Korean Patent Application No. 10 - 2023 - 0180170, filed on December 12, 2023, the entire contents of which are incorporated herein by reference for all purposes. Technical field

[0003] The present disclosure relates to an object tracking device and method thereof, and more particularly, to a technique for improving object tracking performance using bidirectional tracking. Background art

[0004] With the development of artificial intelligence technology, techniques for identifying and tracking objects through image processing are being used in various fields.

[0005] There is an increasing demand for models that can identify and track objects in autonomous vehicles, security monitoring devices, etc.

[0006] Object tracking technology is being developed to identify objects in images and track them. Object tracking refers to computer vision technology that detects changes in the positions of specific objects (such as people, animals, and vehicles) in images captured by a camera.

[0007] Conventionally, an algorithm called the Kalman filter has been used to track such objects.

[0008] However, the traditional Kalman filter is a unidirectional Kalman filter, and due to its use of information from a specific time point and the next time point to estimate the state, it has limitations in terms of accuracy and reliability.

[0009] The information included in this background of the present disclosure is only for enhancing the understanding of the general background of the present disclosure and may not be regarded as an admission or in any form imply that this information constitutes the prior art known to those skilled in the art. Summary of the invention

[0010] Aspects of the present disclosure relate to providing an object tracking device and method thereof configured to improve object tracking performance by performing bidirectional tracking over time in response to identifying an object.

[0011] Exemplary embodiments of the present disclosure seek to provide an object tracking device and method thereof configured to improve object tracking performance by applying a bidirectional Kalman filter forward from time point t - 1 to time point t + 1 and backward from time point t + 1 to time point t - 1, and using the forward tracking result during backward tracking to improve the error during forward tracking.

[0012] Exemplary embodiments of the present disclosure attempt to provide an object tracking device and method thereof, which are configured to output a final object tracking result by applying a forward tracking result and a bidirectional tracking result to a deep learning algorithm, thereby improving object tracking performance.

[0013] The technical objectives of the present disclosure are not limited to the above-mentioned objectives, and those skilled in the art can clearly understand other technical objectives not mentioned from the following description.

[0014] Exemplary embodiments of the present disclosure provide an object tracking device, including: a processor configured to estimate state variables of an object at each time point in a first direction in a time stream, and estimate state variables at each time point in a second direction opposite to the first direction, and configured to estimate state variables of the object in the second direction at each time point using the result values estimated in the first direction; and a memory configured to store data and algorithms driven by the processor.

[0015] In an exemplary embodiment of the present disclosure, the processor may be configured to: in response to estimating state variables of the object in the first direction, estimate state variables of the object at a second time point using the state variables of the object at a first time point and sensor measurement values at the first time point, and in response to estimating state variables of the object in the second direction, estimate state variables of the object at the first time point in the second direction using the state variables of the object at the second time point and the state variables of the object at the first time point estimated in the first direction.

[0016] In an exemplary embodiment of the present disclosure, the first direction may include a forward direction from a previous time to a future time in a time stream, and the second direction may include a reverse direction from a future time to a previous time in the time stream.

[0017] In an exemplary embodiment of the present disclosure, the processor may be configured to estimate forward state variables at a second time point using forward state variables at a first time point and sensor measurement values at the first time point, estimate forward state variables at a third time point using forward state variables at the second time point and sensor measurement values at the second time point, estimate reverse state variables at the second time point using forward state variables at the third time point and forward state variables at the second time point, and estimate reverse state variables at the first time point using reverse state variables at the second time point and forward state variables at the first time point.

[0018] In an exemplary embodiment of the present disclosure, the processor may be configured to input a forward state variable at a first time point into a motion model to predict a forward state variable at a second time point, and update the forward state variable at the second time point using a Kalman gain and a sensor measurement at the second time point.

[0019] In an exemplary embodiment of the present disclosure, the processor may be configured to estimate an error covariance for each time point in a first direction in a time stream and estimate an error covariance for each time point in a second direction opposite to the first direction.

[0020] In an exemplary embodiment of the present disclosure, the processor may be configured to estimate an error covariance at a second time point using an error covariance and system noise at a first time point, and update the error covariance at the second time point using a Kalman gain.

[0021] In an exemplary embodiment of the present disclosure, the processor may be configured to determine weights by applying an input value obtained in a first direction and an input value obtained in a second direction to a deep learning algorithm.

[0022] In an exemplary embodiment of the present disclosure, the input value obtained in the first direction and the input value obtained in the second direction may include at least one of a state variable of an object, an error covariance, noise, a sensor prediction variable, and combinations thereof.

[0023] In an exemplary embodiment of the present disclosure, the processor may be configured to determine an output value in a first direction using a state variable of an object obtained in the first direction, and determine an output value in a second direction using a state variable of an object obtained in the second direction.

[0024] In an exemplary embodiment of the present disclosure, the processor may be configured to determine a final output value by reflecting weights in the output value in the first direction and the output value in the second direction.

[0025] In an exemplary embodiment of the present disclosure, the processor may be configured to compare the final output value with a previously known correct answer to receive feedback on the difference between the final output value and the previously known correct answer.

[0026] In an exemplary embodiment of the present disclosure, the processor may be configured to determine whether both a sensor measurement obtained in the first direction and a sensor measurement obtained in the second direction are input among the input value obtained in the first direction and the input value obtained in the second direction, and

[0027] Determine whether a sensor measurement value obtained in a first direction is input in response to at least one of the sensor measurement value obtained in the first direction and the sensor measurement value obtained in the second direction not being input.

[0028] In an exemplary embodiment of the present disclosure, the processor may be configured to set the weight to 1 in response to the sensor measurement value obtained in the first direction being input, and set the weight to 0 in response to the sensor measurement value obtained in the first direction not being input.

[0029] In an exemplary embodiment of the present disclosure, the processor may be configured to determine whether the difference between the state variable obtained in the first direction and the state variable obtained in the second direction among the input value obtained in the first direction and the input value obtained in the second direction is greater than a predetermined threshold in response to both the sensor measurement value obtained in the first direction and the sensor measurement value obtained in the second direction being input.

[0030] In an exemplary embodiment of the present disclosure, the processor may be configured to set the weight to 1 in response to the difference between the state variable obtained in the first direction and the state variable obtained in the second direction being less than or equal to the predetermined threshold.

[0031] In an exemplary embodiment of the present disclosure, the processor may be configured to determine whether the error covariance obtained in the first direction is less than the error covariance obtained in the second direction between the input value obtained in the first direction and the input value obtained in the second direction in response to the difference between the state variable obtained in the first direction and the state variable obtained in the second direction being greater than the predetermined threshold.

[0032] In an exemplary embodiment of the present disclosure, the processor may be configured to set the weight to 0.5 or greater than 0.5 in response to the error covariance obtained in the first direction being less than the error covariance obtained in the second direction, and set the weight to less than 0.5 in response to the error covariance obtained in the first direction being greater than or equal to the error covariance obtained in the second direction.

[0033] An exemplary embodiment of the present disclosure provides an object tracking method, including: estimating, by a processor, a state variable of an object at each time point in a first direction in a time stream, and estimating, by the processor, a state variable at each time point in a second direction opposite to the first direction, and using a result value estimated in the first direction to estimate a state variable of the object in the second direction at each time point.

[0034] In an exemplary embodiment of the present disclosure, it may further include: the processor determines weights by applying the input value obtained in the first direction and the input value obtained in the second direction to a deep learning algorithm, the processor determines the output value in the first direction by using the state variable of the object obtained in the first direction, the processor determines the output value in the second direction by using the state variable of the object obtained in the second direction; and the processor determines the final output value by reflecting the weights in the output value in the first direction and the output value in the second direction.

[0035] According to an exemplary embodiment of the present disclosure, object tracking performance can be improved by performing bidirectional tracking over time in response to tracking an object.

[0036] According to an exemplary embodiment of the present disclosure, object tracking performance can be improved by applying a bidirectional Kalman filter in the forward direction from time point t - 1 to time point t + 1 and in the reverse direction from time point t + 1 to time point t - 1, and using the forward tracking result during reverse tracking to improve the error during forward tracking.

[0037] According to an exemplary embodiment of the present disclosure, object tracking performance can be improved by applying the forward tracking result and the bidirectional tracking result to a deep learning algorithm to output the final object tracking result.

[0038] In addition, various effects that can be directly or indirectly recognized through this specification may be provided.

[0039] The method and device of the present disclosure have other features and advantages that will be apparent from or more specifically described in the accompanying drawings incorporated herein and the following detailed description, which together with the accompanying drawings and the following detailed description are used to explain certain principles of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 A block diagram showing the configuration of an exemplary vehicle system including an object tracking device is shown.

[0041] Figure 2 A diagram showing an exemplary basic process for describing estimating state variables and error covariance using a Kalman filter is shown.

[0042] Figure 3 A diagram showing an exemplary bidirectional state variable tracking process is shown.

[0043] Figure 4 A diagram showing an exemplary forward object tracking process in detail is shown.

[0044] Figure 5 A diagram showing an exemplary reverse object tracking process in detail is shown.

[0045] Figure 6 A diagram is shown that illustrates an example for describing performing forward tracking during object tracking and performing compensation by backward tracking in response to the absence of sensor measurement values.

[0046] Figure 7 A diagram is shown that illustrates an exemplary process for determining a final object tracking output value using bidirectional tracking results.

[0047] Figure 8 A flowchart is shown that illustrates an exemplary method for determining a final object tracking output value using bidirectional tracking results.

[0048] Figure 9 An exemplary computing system is shown.

[0049] It can be understood that the drawings are not necessarily drawn to scale and present a somewhat simplified representation of various features illustrating the basic principles of the present disclosure. Specific design features of the present disclosure as included herein (including, for example, specific dimensions, orientations, positions, and shapes) will be determined in part by the specific intended application and the usage environment.

[0050] In the drawings, throughout several views of the drawings, reference numerals refer to the same or equivalent parts of the present disclosure. Detailed Description of the Invention

[0051] Reference will now be made in detail to various embodiments of the present disclosure, examples of which are illustrated in the drawings and described below. Although the present disclosure will be described in conjunction with the exemplary embodiments of the present disclosure, it should be understood that this specification is not intended to limit the present disclosure to those exemplary embodiments. On the contrary, the present disclosure is intended to cover not only the exemplary embodiments of the present disclosure, but also various alternatives, modifications, equivalents, and other embodiments that may be included within the spirit and scope of the present disclosure as defined by the appended claims.

[0052] Hereinafter, some exemplary embodiments of the present disclosure will be described in detail with reference to the exemplary drawings. It should be noted that when adding reference numerals to the constituent elements of each drawing, even if the same constituent elements are shown in different drawings, they include as identical reference numerals as possible. When describing the exemplary embodiments of the present disclosure, when it is determined that a detailed description of a well-known configuration or function associated with the exemplary embodiments of the present disclosure may obscure the gist of the present disclosure, its description will be omitted.

[0053] When describing the components according to exemplary embodiments of the present disclosure, terms such as first, second, A, B, (a), and (b) may be used. These terms are only used to distinguish a component from other components, and the nature, sequence, or order of the components is not limited by these terms. In addition, unless differently defined, all terms including technical and scientific terms used herein include the same meanings as those commonly understood by a person skilled in the art (a person having ordinary skill in the art) in the technical field to which the exemplary embodiments of the present disclosure belong. Terms defined in a commonly used dictionary should be interpreted as having meanings that match the meanings in the context of the prior art, and should not be interpreted as having idealized or overly formal meanings, unless they are clearly defined in this specification.

[0054] Hereinafter, various exemplary embodiments of the present disclosure will be described in detail with reference to Figures 1 to 9 Detailed description of various exemplary embodiments of the present disclosure.

[0055] Figure 1 A block diagram showing the configuration of an example vehicle system including an object tracking device is shown.

[0056] With reference to Figure 1 , a vehicle system according to an exemplary embodiment of the present disclosure may include an object tracking device 100 and a sensing device 200.

[0057] The object tracking device 100 according to an exemplary embodiment of the present disclosure may be implemented inside or outside the vehicle. In this case, the object tracking device 100 may be integrally formed with the vehicle's internal control unit, or may be implemented as a separate hardware device connected to the vehicle's control unit through a connecting means. For example, the object tracking device 100 may be implemented integrally with the vehicle, may be implemented in a form separated from the vehicle for installation or attachment to the vehicle, or a part thereof may be implemented integrally with the vehicle while another part may be implemented in a form separated from the vehicle for installation or attachment to the vehicle.

[0058] In response to detecting an object based on output values from a camera, LiDAR, radio detection and ranging (RADAR), etc., the object tracking device 100 may be configured to track the object by applying a Kalman filtering algorithm, and improve object tracking performance through bidirectional tracking by tracking the object in the forward direction and then in the reverse direction. In this case, by using the results of forward tracking instead of sensor measurement values during reverse tracking, the object tracking device 100 may be configured to improve object tracking performance through reverse tracking even when information is missing during forward tracking.

[0059] The object tracking device 100 may include a communication device 110, a memory 120, an interface device 130, and a processor 140. According to an exemplary embodiment of the present disclosure, the object tracking device 100 may be implemented as a single unit by coupling the components to each other, and some components may be omitted.

[0060] The communication device 110 is a hardware device implemented with various electronic circuits to transmit and receive signals through wireless or wired connections, and may transmit and receive information based on in-vehicle device and in-vehicle network communication technologies. As an exemplary embodiment of the present disclosure, the in-vehicle network communication technologies may include Controller Area Network (CAN) communication, Local Interconnect Network (LIN) communication, FlexRay communication, etc.

[0061] In addition, the communication device 110 may communicate with a server, infrastructure, or a third vehicle outside the vehicle, etc. through wireless communication technologies, short-range communication technologies, or mobile communication technologies.

[0062] Herein, the wireless Internet communication may include Wireless Local Area Network (WLAN), Wi-Fi, Wi-Fi Direct, Digital Living Network Alliance (DLNA), Wireless Broadband (WiBro), Worldwide Interoperability for Microwave Access (WiMAX), High-Speed Downlink Packet Access (HSDPA), High-Speed Uplink Packet Access (HSUPA), Long-Term Evolution (LTE), Long-Term Evolution-Advanced (LTE-A), etc. In addition, the short-range communication technologies may include Bluetooth TM , Radio Frequency Identification (RFID), Infrared Data Association (IrDA), Ultra-Wideband (UWB), ZigBee, Near Field Communication (NFC), Wireless Universal Serial Bus (USB) technology, and at least one of any combination thereof.

[0063] The mobile communication technologies may include technical standards for mobile communication, communication methods (e.g., Global System for Mobile Communications (GSM), Code Division Multiple Access (CDMA), Code Division Multiple Access 2000 (CDMA 2000), Enhanced Voice Data Optimized or Enhanced Voice Data Only (EV-DO), Wideband CDMA (WCDMA), High-Speed Downlink Packet Access (HSDPA), High-Speed Uplink Packet Access (HSUPA), Long-Term Evolution (LTE), Long-Term Evolution-Advanced (LTE-A), Fourth Generation Mobile Telecommunications (4G), Fifth Generation Mobile Telecommunications (5G)), etc.

[0064] The memory 120 may store the detection results of the sensing device 200 and data and / or algorithms required for the operation of the processor 140, etc.

[0065] For example, the memory 120 may store algorithms for bidirectional tracking, algorithms for determining the final output value, etc. For example, the memory 120 may store a deep learning model for determining the final output value. In addition, the memory 120 may store a threshold value of the weight value for determining the final output value.

[0066] The memory 120 may include at least one type of storage medium of the following types of memories: such as flash memory, hard disk, micro memory, card (e.g., Secure Digital (SD) card or Extreme Digital (XD) card), random access memory (RAM), static RAM (SRAM), read only memory (ROM), programmable ROM (PROM), electrically erasable PROM (EEPROM), magnetic memory (MRAM), magnetic disk, and optical disk.

[0067] The interface device 130 may include an input device for receiving a control command from a user and an output device for outputting an operation state and its result of the device 100. Herein, the input device may include a button, and may include a mouse, a joystick, a shuttle, a stylus, etc. In addition, the input device may include soft keys implemented on a display.

[0068] The interface device 130 may be implemented as a head-up display (HUD), a cluster, an audio video navigation (AVN), or a human machine interface (HMI).

[0069] The output device may include a display, and may further include a voice output device such as a speaker. In this case, in response to a touch sensor formed of a touch film, a touch sheet, or a touch pad being provided on the display, the display may operate as a touch screen and may be implemented in a form integrating an input device and an output device. In an exemplary embodiment of the present disclosure, the output device may output an object tracking result.

[0070] In this case, the display may include at least one of a liquid crystal display (LCD), a thin film transistor liquid crystal display (TFT LCD), an organic light emitting diode display (OLED display), a flexible display, a field emission display (FED), and a 3D display.

[0071] The processor 140 may be electrically connected to the communication device 110, the memory 120, the interface device 130, etc., may electrically control each component, and may be a circuit that executes software commands and performs various data processing and calculations described below.

[0072] The processor 140 may be configured to process signals transmitted between each component of the object tracking device 100 and perform overall control so that each component can normally execute its function. The processor 140 may be implemented in the form of hardware, software, or a combination of hardware and software. For example, the processor 140 may be implemented as a microprocessor, but the present disclosure is not limited thereto. For example, the processor 140 may be, for example, an electronic control unit (ECU), a microcontroller unit (MCU), or other sub-controller installed in a vehicle.

[0073] The processor 140 may be configured to estimate a state variable of an object for each time point in a first direction (e.g., the forward direction) in a time stream, and estimate the state variable for each time point in a second direction (e.g., the reverse direction) opposite to the first direction, and may be configured to use the result value estimated in the first direction to estimate the state variable of the object in the second direction for each time point.

[0074] In response to estimating the state variable of the object in the first direction, the processor 140 may be configured to estimate the state variable of the object at a second time point t-1 using the state variable of the object at a first time point t-2 and the sensor measurement value at the first time point t-1.

[0075] In response to estimating the state variable of the object in the second direction, the processor 140 may be configured to estimate the state variable of the object at a first time point t-2 in the second direction using the state variable of the object at a second time point t-1 and the state variable of the object at a first time point t-2 estimated in the first direction.

[0076] The first direction may include the forward direction from a previous time to a future time in a time stream, and the second direction may include the reverse direction from a future time to a previous time in a time stream.

[0077] The processor 140 may be configured to estimate the forward state variable at a second time point using the forward state variable at a first time point and the sensor measurement value at the first time point, estimate the forward state variable at a third time point using the forward state variable at the second time point and the sensor measurement value at the second time point, estimate the reverse state variable at the second time point using the forward state variable at the third time point and the forward state variable at the second time point, and estimate the reverse state variable at the first time point using the reverse state variable at the second time point and the forward state variable at the first time point.

[0078] The processor 140 can be configured to input a forward state variable at a first time point into a motion model to predict the forward state variable at a second time point, and use a Kalman gain and a sensor measurement at the second time point to update the forward state variable at the second time point.

[0079] The processor 140 can be configured to estimate an error covariance for each time point in a first direction in the time stream of object tracking, and estimate an error covariance for each time point in a second direction opposite to the first direction.

[0080] The processor 140 can be configured to use the error covariance and system noise at a first time point to estimate the error covariance at a second time point, and use a Kalman gain to update the error covariance at the second time point.

[0081] The processor 140 can be configured to determine weights by applying an input value obtained in the first direction and an input value obtained in the second direction to a deep learning algorithm.

[0082] In the processor 140, the input value obtained in the first direction and the input value obtained in the second direction can include at least one of a state variable of an object, an error covariance, noise, a sensor prediction variable, and combinations thereof.

[0083] The processor 140 can be configured to use the state variable of the object obtained in the first direction to determine an output value in the first direction, and use the state variable of the object obtained in the second direction to determine an output value in the second direction.

[0084] The processor 140 can be configured to determine a final output value by reflecting the weights in the output value in the first direction and the output value in the second direction.

[0085] The processor 140 can be configured to compare the final output value with a previously known correct answer to receive feedback on the difference between the final output value and the previously known correct answer. Thus, the processor 140 can be configured to use the difference to determine the weights.

[0086] The processor 140 can be configured to determine whether both a sensor measurement obtained in the first direction and a sensor measurement obtained in the second direction are input among the input value obtained in the first direction and the input value obtained in the second direction, and in response to at least one of the sensor measurement obtained in the first direction and the sensor measurement obtained in the second direction not being input, determine whether the sensor measurement obtained in the first direction is input.

[0087] The processor 140 may be configured to set the weight to 1 in response to a sensor measurement obtained in the first direction for an input, and set the weight to 0 in response to no sensor measurement obtained in the first direction for the input.

[0088] In response to both a sensor measurement obtained in the first direction for an input and a sensor measurement obtained in the second direction for the input, the processor 140 may be configured to determine whether the difference between the state variable obtained in the first direction and the state variable obtained in the second direction among the input value obtained in the first direction and the input value obtained in the second direction is greater than a predetermined threshold.

[0089] The processor 140 may be configured to set the weight to 1 in response to the difference between the state variable obtained in the first direction and the state variable obtained in the second direction being less than or equal to the predetermined threshold.

[0090] In response to the difference between the state variable obtained in the first direction and the state variable obtained in the second direction being greater than the predetermined threshold, the processor 140 may be configured to determine whether the error covariance obtained in the first direction is less than the error covariance obtained in the second direction between the input value obtained in the first direction and the input value obtained in the second direction.

[0091] The processor 140 may be configured to set the weight to 0.5 or greater than 0.5 in response to the error covariance obtained in the first direction being less than the error covariance obtained in the second direction, and set the weight to less than 0.5 in response to the error covariance obtained in the first direction being greater than or equal to the error covariance obtained in the second direction.

[0092] The sensing device 200 may include one or more sensors that detect an object (e.g., a vehicle ahead) located inside or outside the vehicle and measure the distance to and / or relative speed of the obstacle, and may send the result to the object tracking device 100.

[0093] The sensing device 200 may include multiple sensors for detecting external objects of the vehicle to obtain information related to the position of the external object, the speed of the external object, the moving direction of the external object, and / or the type of the external object (e.g., vehicle, pedestrian, bicycle, or motorcycle, etc.). To this end, the sensing device 200 may include ultrasonic sensors, radio detection and ranging (RADAR), cameras, laser scanners, and / or angular radio detection and ranging (RADAR), light detection and ranging (LiDAR), acceleration sensors, yaw rate sensors, torque measurement sensors, and / or wheel speed sensors, steering angle sensors, global positioning systems, etc.

[0094] Figure 2A diagram is shown for describing an exemplary basic process of estimating a state variable and an error covariance using a Kalman filter.

[0095] The object tracking device 100 may be configured to estimate a state variable using a Kalman filter and an error covariance . The Kalman filter is a recursive filter that estimates the state of a linear dynamic system based on measurement values containing noise and is an adaptive filter. That is, the Kalman filter uses a motion model defined at the position at time point t - 1 (immediately preceding) and sensor measurement values at time point t to update the coordinates at t.

[0096] For a detailed description, the object tracking device 100 may be configured to select initial values for state variable estimation and error covariance estimation. In this case, the initial value for state variable estimation may be the initial state variable value Figure 2 and the initial error covariance value . Such an initial state variable value or the initial error covariance value may be an initial value, a variable state value determined in a previous process, or an error covariance value. In this case, the state variable may be the state information of the object to be estimated and may indicate the position information, speed information, etc. of the object. In addition, the error covariance may be used as a measure of how accurate the state variable prediction is.

[0097] Therefore, the object tracking device 100 may be configured to predict the state variable at time point t (S101) by applying the initial value of the state variable at time point t - 1

[0098] to the motion model and output the estimated state variable value . In this case, the motion model is a model that predicts movement (position, speed, etc.) based on input values. In addition, the object tracking device 100 may be configured to predict the error covariance at time point t using the initial error covariance value

[0099] and the system noise Q at time point t - 1 and output the error covariance estimate (S102). In this case, the system noise Q may be the noise that affects the state variable flowing from the system and may be a predetermined value. In this case, the superscript "-" of the state variable estimate value and the error covariance estimate value indicates the predicted value.

[0100] Subsequently, the object tracking device 100 may be configured to determine the Kalman gain K using the sensor prediction variable R (S103). The Kalman gain is used to set weights in response to updating the Kalman filtering algorithm, and the method for determining the Kalman gain may be used as the general method for determining the Kalman gain, and detailed description thereof will be omitted.

[0101] Accordingly, the object tracking device 100 may be configured to estimate the final state variable through the update process of the Kalman filtering algorithm , the Kalman gain K, and the sensor measurement value (S104). In this case, the sensor measurement value may include the detection result measured by the sensing device 200, for example, the position and velocity information of the object. That is, the object tracking device 100 may be configured to use the Kalman gain K and the sensor measurement value

[0102] to update the state variable estimate , and the updated value may become the final state variable .

[0103] In addition, the object tracking device 100 may be configured to perform the update process of the Kalman filter using the error covariance prediction value and the Kalman gain K (S105), thereby outputting the final error covariance . That is, the object tracking device 100 may be configured to update the predicted error covariance to the final error covariance using the Kalman gain K.

[0104] Figure 2 The process in shows the basic structure of estimating the state variable and the error covariance at the current time point t using the information at the previous time point t - 1, and the processes of estimating the state variable and the error covariance in the forward and reverse time directions will be described based on this basic structure with reference to Figure 3 The final state variables in may respectively refer to the state variables estimated at each time point in

[0105] Figure 2 , Figure 3 the state variables estimated at each time point in , , , , , , and However, there are the following differences: the state variables estimated in the forward direction are used in response to estimating the state variables in the reverse direction, and thus, the object estimation accuracy can be further improved.

[0106] Hereinafter, reference will be made to Figures 3 to 5 describe in detail the process by which the object tracking device 100 of the present disclosure tracks an object by bidirectionally estimating state variables.

[0107] Figure 3 A diagram for describing an exemplary bidirectional state variable tracking process is shown, and Figure 4 A diagram for describing an exemplary forward object tracking process in detail is shown. Figure 5 A diagram for describing an exemplary reverse object tracking process in detail is shown.

[0108] Referring to Figure 3 , the object tracking device 100 may be configured to first use the information at the first time point t-2 in the forward direction to estimate the forward state variable at the second time point t-1 (S201), use the forward state variable at the second time point t-1 to estimate the forward state variable at the third time point t (S202), use the forward state variable at the third time point t to estimate the forward state variable at the fourth time point t+1 (S203), and use the forward state variable at the fourth time point t+1 to estimate the forward state variable at the fifth time point t+2 (S204).

[0109] Figure 4 The process of S204 in response to estimating the state variables in the forward direction is shown in more detail. Referring to Figure 4 , the object tracking device 100 may be configured to use the forward state variable at the fourth time point t+1 to first estimate the forward state variable at the fifth time point t+2 (S211), determine the Kalman gain K (S212), use the sensor measurement value at the fifth time point t+2 to update the forward state variable at the fifth time point t+2 (which was first estimated at the fifth time point) (S213), and finally estimate the state variable at the fifth time point t+2 . Figure 4illustrates the process of using the forward state variable at the fourth time point t+1 to estimate the forward state variable at the fifth time point, but the forward estimation process of the state variable at each time point is the same as that in Figure 4 the same.

[0110] Therefore, the object tracking device 100 can be configured to re-estimate the state variable in the reverse direction.

[0111] That is, the object tracking device 100 can be configured to estimate the reverse state variable at the fourth time point t+1 by using the forward state variable at the fifth time point t+2 and the forward state variable at the fourth time point t+1 (S205). In addition, the object tracking device 100 can be configured to estimate the reverse state variable at the third time point t by using the reverse state variable at the fourth time point t+1 and the forward state variable at the third time point t (S206), estimate the reverse state variable at the second time point t-1 by using the reverse state variable at the third time point t and the forward state variable at the second time point t-1 (S207), and estimate the reverse state variable at the first time point t-2 by using the reverse state variable and the initial state variable at the first time point t-2 (S208).

[0112] In this case, in Figure 5 , the object tracking device 100 can be configured to use the forward state variable at the fifth time point t+2 to first estimate the forward state variable at the fourth time point t+1 (S214), and then determine the Kalman gain (S215) to update the first estimated forward state variable at the fourth time point t+1 by using the forward state variable at the fourth time point t+1 instead of the Kalman gain and the sensor measurement value at the fourth time point t+1 (S216), and finally estimate the forward state variable at the fourth time point t+1 (S214).

[0113] Figure 5illustrates a process of estimating a reverse state variable at a fourth time point using a forward state variable at a fifth time point t+2 in the reverse direction, but the reverse estimation process of the state variable at each time point is the same as that in Figure 5 .

[0114] In addition, in Figures 3 to 5 , the two-way estimation process of the error covariance is not shown, but in response to the situation of the two-way estimated error covariance, the error covariance can be estimated as in the basic structure of Figure 2 .

[0115] However, to further increase the accuracy of object tracking, similar to the state variable estimation in Figure 3 , the error covariance estimated in the forward direction can be used to estimate the error covariance in the reverse direction.

[0116] Figure 6 illustrates a diagram for describing an example of performing forward tracking during object tracking and performing compensation through reverse tracking in response to the absence of sensor measurement values.

[0117] Referring to Figure 6 , the object tracking device 100 can be configured to use the initial state variable information at the first time point t-2 to estimate the forward state variable at the second time point t-1 (S301), use the forward state variable at the second time point t-1 to estimate the forward state variable at the third time point t (S302), use the forward state variable at the third time point t to estimate the forward state variable at the fourth time point t+1 (S303), and use the forward state variable at the fourth time point t+1 to estimate the forward state variable at the fifth time point t+2 (S304). In this case, in step S302, the sensor measurement value is missing, so there may be a situation where the update process cannot be performed and the prediction process can be performed. For example, in Figure 2 , the prediction process of S101 is performed, but the sensor measurement value is not input, so that the update process of S104 is not performed, and the final output value becomes the state variable , which is the output value of the prediction process of S101.

[0118] In this case, in the next step S303, the state variable becomes the input value, so inaccurate information may continue to accumulate and be reflected in the next step.

[0119] However, the object tracking device 100 may be configured to minimize such error problems by reflecting the state variables estimated in the forward direction and re - estimating the state variables in the reverse direction.

[0120] That is, the object tracking device 100 may be configured to estimate the backward state variable at the fourth time point t + 1 by using the forward state variable at the fifth time point t + 2 and the forward state variable at the fourth time point t + 1 (S305). In addition, the object tracking device 100 may be configured to estimate the backward state variable at the third time point t by using the backward state variable at the fourth time point t + 1 and the forward state variable at the third time point t (S306), to estimate the backward state variable at the second time point t - 1 by using the backward state variable at the third time point t and the forward state variable at the second time point t - 1 (S307), and to estimate the backward state variable at the first time point t - 2 by using the backward state variable and the initial state variable at the first time point t - 2 (S308).

[0121] Therefore, according to an exemplary embodiment of the present disclosure, even if an error occurs in response to tracking an object in the forward direction, the influence of the error can be minimized by re - tracking it in the reverse direction, thereby improving object tracking performance.

[0122] In addition, according to an exemplary embodiment of the present disclosure, by reflecting the results obtained by estimating the object in the forward direction and re - tracking the object in each step instead of simply tracking the object forward and backward, object tracking performance can be improved.

[0123] Therefore, after estimating the state variables in both directions, the object tracking device 100 may be configured to output, as an output value Y, the variables among the estimated state variables X that can be measured by the sensor. For example, the state variables X may include speed, acceleration, angular velocity, position information, etc., and among these state variables, position information, which is one of the variables that can be measured by the sensor, may be output as the output value Y. The output value Y may include a forward output value output using the forward state variable and a backward output value output using the backward state variable . Therefore, the object tracking device 100 may be configured to use the forward output value and the reverse output value to output the final output value .

[0124] In the following, reference will be made to Figure 7 and Figure 8 to describe in detail a method for determining a final object tracking output value using bidirectional tracking results according to an exemplary embodiment of the present disclosure . Figure 7 FIG. shows a diagram for describing an exemplary process of determining a final object tracking output value using bidirectional tracking results , and Figure 8 FIG. shows a diagram for describing an exemplary process of determining a final object tracking output value using bidirectional tracking results .

[0125] In the following, it is assumed that Figure 1 the object tracking device 100 is configured to execute Figure 7 the process. In addition, in the Figure 7 description, the operations described as being performed by the device can be understood as being controlled by the processor 140 of the object tracking device 100. In the following exemplary embodiments of the present disclosure, the operations of steps S501 to S514 may be executed sequentially, but do not necessarily have to be executed sequentially. For example, the order of each operation can be changed, and at least two operations can be executed in parallel.

[0126] Referring to Figure 7 , the object tracking device 100 can be configured to learn by inputting the forward input value obtained in the Figure 3 forward tracking process and the reverse input value obtained in the reverse tracking process into a deep learning model to determine the weights to be applied to the forward output value and the reverse output value (S401). In this case, the forward input value obtained in the forward tracking process and the reverse input value obtained in the reverse tracking process may include the state variable Xt, the error covariance, the noise, and the sensor prediction variable. The object tracking device 100 can be configured to determine the final output value by reflecting the weights in the forward output value determined using the forward state variable and the reverse output value determined using the reverse state variable (S402). The variables of the Kalman filter can be used as its input to learn the deep learning model, and more accurate output values can be determined by implicitly learning the relationships between the variables.

[0127] The formula for determining the final output value is shown in Formula 1 below.

[0128] (Formula 1)

[0129] · + ·

[0130] As the forward output value can be the state variable X estimated during forward tracking t and can be the value output by the sensor among the values. As the reverse output value can be the state variable X estimated during reverse tracking t and can be the value output by the sensor among the values.

[0131] In addition can be the weights determined in the deep learning model and can be expressed by the following formula 2, and the weights can include the x-axis weights , y-axis weights and z-axis weights .

[0132] (Formula 2)

[0133] =

[0134] Therefore, the object tracking device 100 can be configured to improve the accuracy by comparing the determined output value with the previously known correct answer value and re-inputting the difference L2 Loss between them into the deep learning model to learn it (S403).

[0135] (Formula 3)

[0136]

[0137] Refer to Figure 8 , the object tracking device 100 can be configured to, in response to the result values obtained during forward tracking (S501) (forward state variable , forward error covariance , forward sensor measurement value ) and the result values obtained during reverse tracking (S502) (reverse state variable , reverse error covariance and reverse sensor measurement value ) are respectively input into the deep learning model (S503 and S504), and determine whether the sensor measurement value obtained during forward tracking and the sensor measurement value obtained during reverse tracking are valid values (S505). Herein, the object tracking device 100 can be configured to, in response to the absence of sensor measurement values , i.e., the sensor measurement value In the case where the sensor measurement value is not input due to a sensor failure or the like, it is determined that the sensor measurement value is invalid.

[0138] In response to at least one of the sensor measurement values being invalid, the object tracking device 100 may be configured to determine whether the forward sensor measurement value is valid (S506).

[0139] The object tracking device 100 may be configured to, in response to the forward sensor measurement value being valid, determine the weight as 1 (S507), and in response to the forward sensor measurement value being invalid, determine the weight as 0 (S508), so as to determine the final output value by applying the determined weight to Equation 1 .

[0140] Meanwhile, in step S505, if all of the sensor measurement values ( ) are valid, the object tracking device 100 may be configured to determine whether the difference between the forward state variable and the reverse state variable is greater than a predetermined threshold (S509).

[0141] In response to the difference between the forward state variable and the reverse state variable being equal to or less than the predetermined threshold, the object tracking device 100 may be configured to determine the weight as 1 (S511) and apply the determined weight to Equation 1 to determine the final output value (S514).

[0142] In response to the difference between the forward state variable and the reverse state variable being greater than the predetermined threshold, the object tracking device 100 may be configured to determine whether the forward error covariance is less than the reverse error covariance (S510). That is, the object tracking device 100 may be configured to recheck the error covariance in response to a large difference between the forward output value and the reverse output value.

[0143] The object tracking device 100 may be configured to, in response to the forward error covariance being less than the reverse error covariance determine the weight Greater than or equal to a predetermined value (A) (e.g., 0.5) (S512), and in response to a positive error covariance Greater than the reverse error covariance And determine the weight Less than the predetermined value (A) (e.g., 0.5) (S513), by applying the determined weight To the above formula 1 to determine the final output value (S514). Thus, the object tracking device 100 controls the vehicle based on the final output value Control the vehicle.

[0144] Thus, according to an exemplary embodiment of the present disclosure, the weight can be determined by using the forward tracking result and the reverse tracking result as its input to the deep learning model And the final output value can be output by applying the weight to the forward tracking result and the reverse tracking result Thereby further improving the accuracy of the final output value.

[0145] Figure 9 Shows an exemplary computing system.

[0146] Reference Figure 9 , The computing system 1000 includes at least one processor 1100, a memory 1300, a user interface input device 1400, a user interface output device 1500, and a memory 1600 and a network interface 1700 connected by a bus 1200.

[0147] The processor 1100 can be a central processing unit (CPU) or a semiconductor device configured to execute processing on commands stored in the memory 1300 and / or the memory 1600. The memory 1300 and the memory 1600 can include various types of volatile or non-volatile storage media. For example, the memory 1300 can include a read-only memory (ROM) 1310 and a random access memory (RAM) 1320.

[0148] Therefore, the steps of the method or algorithm described in connection with the exemplary embodiments included herein can be directly implemented by hardware, software modules executed by the processor 1100, or a combination of the above two. The software module can reside in a storage medium (i.e., the memory 1300 and / or the memory 1600), such as a RAM memory, a flash memory, a ROM memory, an EPROM memory, an EEPROM memory, a register, a hard disk, a removable disk, and a CD-ROM.

[0149] An exemplary storage medium is coupled to the processor 1100, and the processor 1100 can read information from and write information to the storage medium. Alternatively, the storage medium can be integrated with the processor 1100. The processor and the storage medium can reside within an application specific integrated circuit (ASIC). The ASIC can reside within a user terminal. Alternatively, the processor and the storage medium can reside within the user terminal as separate components.

[0150] The above description is only an illustration of the technical concept of the present disclosure, and those skilled in the art to which the present disclosure pertains can make various modifications and variations without departing from the basic features of the present disclosure.

[0151] In various exemplary embodiments of the present disclosure, each of the above operations can be performed by a control device, and the control device can be configured by a plurality of control devices or an integrated single control device.

[0152] In various exemplary embodiments of the present disclosure, the memory and the processor can be provided as one chip or as separate chips.

[0153] In various exemplary embodiments of the present disclosure, the scope of the present disclosure includes software or machine-executable commands (e.g., operating systems, applications, firmware, programs, etc.) for enabling the operations of the methods according to various embodiments to be performed on a device or computer, and non-transitory computer-readable media including such software or commands stored thereon and executable on the device or computer.

[0154] In various exemplary embodiments of the present disclosure, the control device can be implemented in the form of hardware or software, or can be implemented in a combination of hardware and software.

[0155] In addition, terms such as "unit" and "module" included in the specification refer to a unit for processing at least one function or operation, which can be implemented by hardware, software, or a combination thereof.

[0156] In an exemplary embodiment of the present disclosure, a vehicle can be referred to as being based on a concept including various transportation means. In some cases, a vehicle can be interpreted as being based on not only various land transportation means traveling on roads (such as cars, motorcycles, trucks, and buses) but also various transportation means such as airplanes, drones, ships, etc.

[0157] For the sake of illustration and to accurately define the appended claims, reference is made to the positions of such features shown in the accompanying drawings, and the terms "on", "under", "inside", "outside", "above", "below", "upward", "downward", "in front of", "behind", "at the back of", "inner", "outer", "inward", "outward", "inside of", "outside of", "internal", "external", "forward" and "reverse" are used to describe the features of the exemplary embodiments. It should be further understood that the term "connected" or its derivatives refers to both direct connection and indirect connection.

[0158] The term "and / or" may include combinations of multiple related listed items or any one of the multiple related listed items. For example, "A and / or B" includes all three cases, such as "A", "B", and "A and B".

[0159] In the exemplary embodiments of the present disclosure, "at least one of A and B" may refer to "at least one of A or B" or "at least one of a combination of at least one of A and B". In addition, "one or more of A and B" may refer to "one or more of A or B" or "one or more of a combination of one or more of A and B".

[0160] In this specification, otherwise singular expressions include plural expressions, unless the context clearly indicates otherwise.

[0161] In the exemplary embodiments of the present disclosure, it should be understood that terms such as "including" or "having" relate to the presence of the features, quantities, steps, operations, elements, components or combinations thereof described in the specification, and do not exclude the possibility of adding or the presence of one or more other features, quantities, steps, operations, elements, components or combinations thereof.

[0162] According to the exemplary embodiments of the present disclosure, components may be combined with each other to form one, or some components may be omitted.

[0163] Hereinafter, the fact that hardware is operably coupled may include the fact of establishing a direct and / or indirect connection between the hardware through wired and / or wireless means.

[0164] For purposes of illustration and description, the foregoing description of specific exemplary embodiments of the present disclosure has been presented. They are not intended to be exhaustive or to limit the present disclosure to the precise forms disclosed, and obviously many modifications and variations are possible in light of the above teachings. For purposes of illustrating certain principles of the invention and their practical applications, exemplary embodiments have been selected and described so that others skilled in the art can make and utilize the various exemplary embodiments of the present disclosure and their various alternatives and modifications. The scope of the present disclosure is intended to be defined by the appended claims and their equivalents.

Claims

1. An object tracking device, comprising: a processor configured to estimate a state variable of an object for each time point in a first direction in a time stream, and to estimate the state variable for each time point in a second direction opposite to the first direction in the time stream, and to estimate the state variable of the object in the second direction for each time point using a result value estimated in the first direction; as well as A memory is configured to store data and algorithms driven by the processor.

2. The object tracking device according to claim 1, wherein: The processor is further configured to: In response to estimating the state variable of the object in the first direction, estimating the state variable of the object at a second time point using the state variable of the object at a first time point and the sensor measurement at the first time point, and In response to estimating the state variable of the object in the second direction, the state variable of the object at the first time point in the second direction is estimated using the state variable of the object at the second time point and the state variable of the object at the first time point estimated in the first direction.

3. The object tracking device according to claim 1, in, The first direction comprises a forward direction from a previous time to a future time in the time stream, and Wherein, the second direction comprises a reverse direction from the future time to the previous time in the time flow.

4. The object tracking device according to claim 3, in, The state variables include forward state variables and reverse state variables. Wherein, the processor is further configured to: estimating the forward state variable at a second time point using the forward state variable at a first time point and a sensor measurement value at the first time point, estimating the forward state variable at a third time point using the forward state variable at the second time point and the sensor measurement value at the second time point, using the forward state variable at the third time point and the forward state variable at the second time point to estimate the reverse state variable at the second time point, and The reverse state variable at the first time point is estimated using the reverse state variable at the second time point and the forward state variable at the first time point.

5. The object tracking device according to claim 3, in, The state variables include forward state variables and reverse state variables. Wherein, the processor is further configured to: inputting the forward state variable at a first time point into a motion model to predict the forward state variable at a second time point, and The forward state variable at the second time point is updated using the Kalman gain and the sensor measurement value at the second time point.

6. The object tracking device according to claim 1, wherein: The processor is configured to estimate an error covariance for each time point in the first direction in the time stream, and to estimate the error covariance for each time point in the second direction opposite to the first direction in the time stream.

7. The object tracking device according to claim 6, wherein: The processor is further configured to: using the error covariance at the first time point and system noise to estimate the error covariance at the second time point, and The error covariance at the second time point is updated using a Kalman gain.

8. The object tracking device according to claim 1, wherein: The processor is further configured to determine a weight by applying an input value obtained in the first direction and an input value obtained in the second direction to a deep learning algorithm.

9. The object tracking device according to claim 8, wherein: The input value obtained in the first direction and the input value obtained in the second direction include at least one of the state variable of the object, error covariance, noise, sensor prediction variables, and a combination thereof.

10. The object tracking device according to claim 8, wherein: The processor is further configured to: determining an output value in the first direction using the state variable of the object obtained in the first direction, and The output value in the second direction is determined using the state variable of the object obtained in the second direction.

11. The object tracking device according to claim 10, wherein: The processor is further configured to determine a final output value by reflecting a weight on the output value in the first direction and the output value in the second direction.

12. The object tracking device according to claim 11, wherein: The processor is further configured to compare the final output value to a previously known correct answer to receive feedback regarding a difference between the final output value and the previously known correct answer.

13. The object tracking device according to claim 8, wherein: The processor is further configured to: determining whether a sensor measurement value obtained in the first direction and a sensor measurement value obtained in the second direction are inputted among the input value obtained in the first direction and the input value obtained in the second direction, and In response to at least one of the sensor measurement value obtained in the first direction and the sensor measurement value obtained in the second direction not being input, it is determined whether the sensor measurement value obtained in the first direction is input.

14. The object tracking device according to claim 13, wherein: The processor is further configured to: In response to inputting the sensor measurement value obtained in the first direction, setting the weight to 1, and In response to the sensor measurement value obtained in the first direction not being input, the weight is set to zero.

15. The object tracking device according to claim 13, wherein: The processor is configured to, In response to inputting the sensor measurement value obtained in the first direction and the sensor measurement value obtained in the second direction, It is determined whether a difference between the state variable obtained in the first direction and the state variable obtained in the second direction among the input value obtained in the first direction and the input value obtained in the second direction is greater than a predetermined threshold.

16. The object tracking device according to claim 15, wherein: The processor is configured to set the weight to 1 in response to the difference between the state variable obtained in the first direction and the state variable obtained in the second direction being less than or equal to the predetermined threshold.

17. The object tracking device according to claim 16, wherein: The processor is configured to: In response to the difference between the state variable obtained in the first direction and the state variable obtained in the second direction being greater than the predetermined threshold, it is determined whether the error covariance obtained in the first direction between the input value obtained in the first direction and the input value obtained in the second direction is smaller than the error covariance obtained in the second direction.

18. The object tracking device according to claim 17, wherein: The processor is further configured to: In response to the error covariance obtained in the first direction being smaller than the error covariance obtained in the second direction, setting the weight to 0.5 or greater than 0.5, and In response to the error covariance obtained in the first direction being greater than or equal to the error covariance obtained in the second direction, the weight is set to be less than 0.

5.

19. An object tracking method, comprising: estimating, by the processor, a state variable of the object for each time point in a first direction in the time stream; as well as The processor estimates the state variable for each time point in a second direction opposite to the first direction in the time stream, and estimates the state variable of the object in the second direction for each time point using the result value estimated in the first direction.

20. The object tracking method according to claim 19, further comprising: determining, by the processor, a weight by applying an input value obtained in the first direction and an input value obtained in the second direction to a deep learning algorithm; determining, by the processor, an output value in the first direction using the state variable of the object obtained in the first direction; determining, by the processor, an output value in the second direction using the state variable of the object obtained in the second direction; as well as A final output value is determined by the processor by reflecting a weight on the output value in the first direction and the output value in the second direction.

Images