Apparatus for tracking object and method of operating same

Abstract

An apparatus for object tracking is provided. Such an apparatus may include a processor and a memory storing instructions that, when executed by the processor, cause the apparatus to receive a first resolution image associated with a first time and including an object of interest, generate a second resolution image, determine a second resolution feature of the object of interest, and generate a second resolution feature of the object of interest. Determining the object-of-interest information may include position information of the object-of-interest, determining a first resolution feature associated with a region of the first resolution image, outputting temporal profile information indicative of a profile of the object-of-interest associated with a first time, tracking an action state of the object-of-interest, and outputting a second resolution feature associated with the first resolution image. Wherein the past time profile information indicates a profile of the object of interest associated with a second time before the first time, and outputting a signal for controlling an operation of the second device.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims the benefit of priority from Korean Patent Application No. 10-2023-0104285 filed on August 9, 2023, in the Korean Intellectual Property Office, the entire contents of which are incorporated herein by reference. Technical Field

[0003] The present invention relates to an object tracking device and an operating method thereof. Background Art

[0004] As part of autonomous driving technology, various sensors may be installed on the vehicle. The sensors installed in the vehicle may include cameras, light detection and ranging (LiDAR), ultrasonic sensors, etc. Autonomous vehicles may utilize various sensors to avoid collisions with surrounding objects and may stably perform autonomous driving.

[0005] For safe driving of autonomous vehicles, object tracking technology capable of detecting objects around the vehicle and tracking the movement of these objects may be used with cameras. Detectors capable of detecting objects of interest may be used as devices utilized in object tracking technology. Such detectors may utilize deep learning-based neural networks to identify objects of interest in images captured by the camera.

[0006] As the performance of cameras installed in vehicles improves, higher-resolution images are available. However, if higher-resolution images are fed into a detector, a large amount of resources are used, making it impractical to obtain the full output value through a neural network.

[0007] On the other hand, if a lower resolution image is used to detect an object, information about objects that are farther away relative to the camera's field of view may be lost due to the low resolution. In this case, the features of distant objects may not be accurately extracted.

[0008] In addition, based on the position change of (one or more) objects detected in each image frame, only using intersection over union (IoU) and Mahalanobis distance may not be able to effectively track the movement of the object. IoU and Mahalanobis distance refer to techniques for tracking objects based on the position change of the object in each image frame. Because if multiple objects overlap each other in each image frame, identifiers respectively assigned to multiple objects may be switched or confused. Summary of the invention

[0009] According to the present invention, a device may include: a processor and a memory storing instructions, which, when executed by the processor, may cause the device to: receive a first resolution image associated with a first time and including an object of interest, generate a second resolution image based on the first resolution image, determine a second resolution feature of the object of interest based on the second resolution image, determine that information about the object of interest may include location information of the object of interest based on the second resolution feature, determine a first resolution feature associated with an area of ​​the first resolution image based on the location information, output time shape information indicating an appearance of the object of interest associated with the first time based on the first resolution feature and the second resolution feature, and track an action state of the object of interest based on the time shape information and past time shape information, wherein the past time shape information indicates an appearance of the object of interest associated with a second time before the first time, and output a signal to control the operation of the second device based on the tracked action state of the object of interest.

[0010] In the device, when the instructions are executed by the processor, the device can output temporal shape information by applying a pooling operation to the first resolution feature and the second resolution feature.

[0011] In the device, when the instructions are executed by the processor, the device can: determine whether the object of interest is the same as the object previously identified at a second time based on the similarity between the time profile information and the past time profile information, and track the action status of the object of interest based on determining that the object of interest is the same.

[0012] In the device, when the instructions are executed by the processor, the device can be caused to: determine whether the object of interest is the same as a previously identified object based on the cosine similarity between the temporal shape information and the past temporal shape information.

[0013] In the device, when the instructions are executed by the processor, the device can be caused to: determine whether the object of interest is the same as a previously identified object based on at least one of intersection over union (IoU), Mahalanobis distance, and cosine similarity.

[0014] In the device, when the instructions are executed by the processor, the device can: assign a new identifier (ID) to the object of interest based on the object of interest being different from a previously identified object or based on the object of interest not being previously identified, and track the action status of the object of interest based on the new identifier.

[0015] In the device, when the instructions are executed by the processor, the device can be enabled to: obtain a second resolution image by adjusting the resolution of the first resolution image.

[0016] In the device, when the instructions are executed by the processor, the instructions may cause the device to: determine a region of the first resolution image based on the position information and based on a ratio of a size of the second resolution image to a size of the first resolution image.

[0017] According to the present invention, a method may include: receiving a first resolution image associated with a first time and including an object of interest, generating a second resolution image based on the first resolution image, determining a second resolution feature of the object of interest based on the second resolution image, determining that information about the object of interest may include position information of the object of interest based on the second resolution feature, determining a first resolution feature associated with an area of ​​the first resolution image based on the position information, outputting time shape information indicating an appearance of the object of interest associated with the first time based on the first resolution feature and the second resolution feature, and tracking an action state of the object of interest based on the time shape information and past time shape information, wherein the past time shape information indicates an appearance of the object of interest associated with a second time before the first time, and outputting a signal for the operation of a control device based on tracking the action state of the object of interest.

[0018] In the method, outputting the temporal shape information may include: applying a pooling operation to the first resolution feature and the second resolution feature. The method may also include: determining whether the object of interest is the same as a previously identified object based on similarity between the temporal shape information and past temporal shape information, wherein tracking the action state of the object of interest may include: tracking the action state of the object of interest based on the determined result.

[0019] In the method, determining whether the object of interest is the same as a previously recognized object may include determining whether the object of interest is the same as the previously recognized object based on a cosine similarity between the temporal profile information and the past temporal profile information.

[0020] In the method, determining whether the object of interest is the same as the previously recognized object may include determining whether the object of interest is the same as the previously recognized object based on at least one of intersection over union (IoU), Mahalanobis distance, and cosine similarity.

[0021] The method may further include assigning a new identifier (ID) to the object of interest based on at least one of the following: the object of interest is different from a previously identified object, or the object of interest was not previously identified. In the method, receiving the second resolution image may include adjusting the resolution of the first resolution image.

[0022] The method may further include determining a region of the first resolution image based on the position information and based on a ratio of a size of the second resolution image to a size of the first resolution image. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] The above and other objects, features and advantages of the present invention will become more apparent through the following detailed description in conjunction with the accompanying drawings:

[0024] Figure 1 An example of an object tracking apparatus according to an example of the present invention is shown;

[0025] Figure 2 An example is shown in which an object tracking device acquires a first resolution image and a second resolution image according to a method of an example of the present invention;

[0026] Figure 3 An example is shown in which an object tracking device generates a first resolution feature and a second resolution feature according to a method of an example of the present invention;

[0027] Figure 4 An example is shown in which an object tracking device outputs specific time shape information using an average pooling operation according to an example method of the present invention;

[0028] Figure 5 shows an example of a cost matrix according to an example of the present invention; and

[0029] Figure 6 An example of an operating method of an object tracking apparatus according to an example of the present invention is shown. DETAILED DESCRIPTION

[0030] Hereinafter, examples of the present invention may be described with reference to the accompanying drawings. However, those skilled in the art will recognize that various modifications, equivalents and / or substitutions may be made to the various examples described herein without departing from the scope and spirit of the present invention.

[0031] The examples of the present invention and the terms used herein are not intended to limit the technical features described in the present invention to specific examples, and it should be understood that these examples and terms include modifications, equivalents or alternatives to the corresponding examples described herein. Regarding the description of the drawings, similar or related parts may be marked with similar reference signs / numbers. Unless otherwise explained in the context, the singular form of a noun corresponding to an item may include one or more items.

[0032] In the present invention, the expressions "A or B", "at least one of A and B", "at least one of A or B", "A, B or C", "at least one of A, B and C" and "at least one of A, B or C" may include any and all combinations of one or more of the relevant listed items. Unless specifically stated to the contrary, terms such as "first", "second", "A", "B", "(a)" or "(b)" may be used to simply distinguish corresponding components from other components but do not limit corresponding components in other examples (e.g., importance or order).

[0033] In this specification, if a component (e.g., a first component) is referred to as being “coupled to” or “connected to” another component (e.g., a second component) with or without the term “operably” or “communicatively”, it may mean that the component may be directly (e.g., by wire or wirelessly) or indirectly (e.g., through a third component) connected to the other component.

[0034] The methods according to various examples disclosed in this specification may be provided as included in a computer program product. The computer program product may be traded as a product between a seller and a buyer. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., a compact disc read-only memory (CD-ROM)), or may be distributed (e.g., downloaded or uploaded) through an application store, directly between two user devices, or online. In the case of online distribution, at least a portion of the computer program product may be at least temporarily stored in a machine-readable storage medium, such as a memory of a manufacturer's server, an application store's server, or a relay server, or may be temporarily generated.

[0035] According to the examples disclosed in this specification, each of the above-mentioned components (such as a module or a program) may include a single entity or multiple entities, and a portion of the multiple objects may be arranged separately on other components. According to the examples disclosed in this specification, one or more components or operations in the above-mentioned components may be omitted, or one or more other components or operations may be added. Alternatively or additionally, multiple components (such as modules or programs) may be integrated into one component. In this case, the integrated components may perform one or more functions of each of the multiple components in the same or similar manner as the corresponding components of the multiple components before integration. According to the examples disclosed in this specification, the operations performed by modules, programs or other components may be performed sequentially, in parallel, repeatedly or irrationally. Alternatively or additionally, at least one or more operations may be performed in another order or may be omitted, or one or more operations may be added.

[0036] Figure 1 An example of an object tracking apparatus 10 according to an example of the present invention is shown.

[0037] refer to Figure 1 , the object tracking device 10 may be connected to the electronic device 12 and the sensor 14 in a wireless and / or wired manner.

[0038] In an example, the connection between the object tracking device 10 and the electronic device 12 may include a communication connection through a wired and / or wireless network. In an example, the wired network may be based on a local area network (LAN) communication or a wire communication. In an example, the wireless network may be based on a short-range communication network (e.g., Bluetooth, Wireless Fidelity (Wi-Fi), or Infrared Data Association (IrDA)) or a long-range communication network (e.g., a cellular network, a 4G network, or a 5G network).

[0039] In another example, the connection between the object tracking device 10 and the electronic device 12 may include a connection via a device-to-device communication method such as a bus, a general purpose input and output (GPIO), a serial peripheral interface (SPI), or a mobile industry processor interface (MIPI).

[0040] In an example, electronic device 12 may include a mobile device such as a mobile phone, laptop, smartphone, or smart tablet, or an electric vehicle such as an electric vehicle (EV), hybrid EV (HEV), plug-in HEV (PHEV), or fuel cell EV (FCEV).

[0041] In an example, the connection between object tracking device 10 and sensor 14 may include a communication connection through a wired and / or wireless network.

[0042] In an example, the sensor 14 may be implemented by a camera, a radio detection and ranging (radar), a light detection and ranging (LiDAR), an ultrasonic sensor, or a combination thereof. According to one example, the sensor 14 may be included in the electronic device 12 or the object tracking device 10. Below, a description will be given assuming that the sensor 14 is implemented by a camera.

[0043] The object tracking device 10 may include a communication circuit 100, a memory 120, a processor 140, or a combination thereof. According to an example, in addition to Figure 1 Parts other than those shown or as Figure 1 Replacement of components described in Figure 1 The object tracking device 10 illustrated in FIG. 1 may also include at least one component (eg, a display, an input device, an output device, or a sensor 14 ).

[0044] In an example, the communication circuit 100 may establish a wired communication channel and / or a wireless communication channel between the object tracking device 10 and the electronic device 12 and / or between the object tracking device 10 and the sensor 14, and may send data to and receive data from the electronic device 12 and / or the sensor 14 through the established communication channel.

[0045] The memory 120 may store data used by at least one component of the object tracking device 10 (e.g., the processor 140). For example, the data may include software (or instructions associated with the software), input data, or output data. In an example, when the instructions are executed by the processor 140, the instructions may cause the object tracking device 10 to perform the operation defined by the instructions.

[0046] In an example, memory 120 may include volatile memory and / or non-volatile memory.

[0047] In an example, the memory 120 may include one or more software (e.g., a first resolution image acquisition unit 122, a second resolution image acquisition unit 124, an object of interest information generation unit 126, a first resolution feature generation unit 128, an appearance information output unit 130, an object tracking unit 132, or a combination thereof).

[0048] The processor 140 can execute software (e.g., the first resolution image acquisition unit 122, the second resolution image acquisition unit 124, the object of interest information generation unit 126, the first resolution feature generation unit 128, the appearance information output unit 130, the object tracking unit 132, or a combination thereof) to control at least another component of the object tracking device 10 connected to the processor 140, and perform various data processing or calculations (or operations).

[0049] In an example, the processor 140 may include a central processing unit, an application processor, a graphics processing unit, a neural processing unit (NPU), an image signal processor, a sensor hub processor, or a communication processor.

[0050] Each of the plurality of software stored in the memory 120 (e.g., the first resolution image acquisition unit 122, the second resolution image acquisition unit 124, the attention object information generation unit 126, the first resolution feature generation unit 128, the shape information output unit 130, the object tracking unit 132, or a combination thereof) may be software that performs a portion of the operation to be performed by one neural network model. Below, the description will be made assuming that the neural network is a convolutional neural network (CNN). This is for ease of description, and the neural network is not limited to the convolutional neural network.

[0051] Below, a method will be described in which the object tracking device 10 tracks one or more objects of interest through the first resolution image acquisition unit 122, the second resolution image acquisition unit 124, the object of interest information generation unit 126, the first resolution feature generation unit 128, the appearance information output unit 130 and the object tracking unit 132.

[0052] The first resolution image acquisition unit 122 may acquire a first resolution image. The first resolution image acquisition unit 122 may acquire a first resolution image including an object of interest captured at a specific time point. The first resolution image acquisition unit 122 may acquire a first resolution image captured around the vehicle using a camera (or include information about the vehicle surroundings captured using a camera). Here, the first resolution image may refer to an image having a higher resolution than a second resolution image to be described later. The specific time point may refer to a specific time or the current time.

[0053] The second resolution image acquisition unit 124 may acquire a second resolution image. The second resolution image acquisition unit 124 may acquire a second resolution image corresponding to the first resolution image. The second resolution image acquisition unit 124 may acquire a second resolution image including an object of interest captured at a specific time point. Here, the "second resolution image corresponding to the first resolution image" may be an image having only a resolution difference from the first resolution image. The second resolution of the second resolution image may be lower than the first resolution of the first resolution image.

[0054] In an example, the second resolution image acquisition unit 124 may acquire the second resolution image by adjusting the resolution of the first resolution image. The second resolution image acquisition unit 124 may acquire the second resolution image by adjusting the size of the first resolution image.

[0055] In an example, the second resolution image acquisition unit 124 may acquire a second resolution image captured around the vehicle using a camera (or include information about the surrounding environment of the vehicle captured using a camera).

[0056] The attention object information generating unit 126 may generate a second resolution feature associated with the attention object based on the second resolution image. Here, the second resolution feature may refer to a feature map including features of the attention object extracted from the second resolution image.

[0057] The object of interest information generating unit 126 may generate the object of interest information including the position information of the object of interest based on the second resolution feature. Here, the object of interest information may include the position information and classification information of the object of interest. The position information may indicate the coordinate value for extracting a specific area in the image frame. The position information may indicate the coordinate value of the bounding box including the object of interest in the image frame at a specific time point. The classification information may indicate information for specifying the type of the object of interest according to the pre-classified items.

[0058] The first resolution feature generation unit 128 may generate a first resolution feature associated with a specific area in the first resolution image, which corresponds to the position information of the second resolution image. The first resolution feature generation unit 128 may generate a first resolution feature associated with a specific area in the first resolution image, which corresponds to the coordinate value of the bounding box including the object of interest in the second resolution image. Here, the first resolution feature may refer to a feature map including the features of the object of interest extracted from the first resolution image. The specific area may be an area where the result of converting the position information based on the ratio of the size of the second resolution image to the size of the first resolution image is applied to the first resolution image.

[0059] In an example, the first resolution feature generating section 128 may be included in the attention object information generating section 126 .

[0060] The appearance information output unit 130 may output the appearance information of the object of interest based on the first resolution feature and the second resolution feature. The appearance information output unit 130 may output the specific time appearance information of the object of interest based on the first resolution feature and the second resolution feature. Here, the specific time appearance information may refer to a feature vector associated with the appearance feature of the object of interest in the image frame acquired at a specific time point.

[0061] In an example, the shape information output unit 130 may output the specific time shape information by applying a pooling operation to the first resolution feature and the second resolution feature.

[0062] In an example, the shape information output unit 130 may output the specific time shape information using an average pooling operation performed based on the first resolution feature and the second resolution feature. Here, the average pooling operation may refer to an operation performed in a pooling layer of a CNN neural network structure and is used to extract partial data from a feature map based on an average value in a window.

[0063] In an example, the shape information output unit 130 may output the specific time shape information using a maximum pooling operation performed based on the first resolution feature and the second resolution feature. Here, the maximum pooling operation may refer to an operation performed in a pooling layer of a CNN neural network structure and is used to extract partial data from a feature map based on a maximum value in a window.

[0064] The object tracking unit 132 may track the action state of the object of interest based on the specific time shape information and the past time shape information corresponding to the past time point before the specific time point. Here, the past time point may be a time point before the specific time point. The past time shape information may refer to a feature map associated with the shape features of one or more objects of interest identified in the image frame acquired before the specific time point. The past time shape information may be information generated from the past image frame according to the above-mentioned process of outputting the specific time shape information.

[0065] The object tracking unit 132 may determine whether the object of interest is the same as a previously identified object. The object tracking unit 132 may determine whether the object of interest is the same as a previously identified object based on the similarity between the specific time shape information and the past time shape information. The object tracking unit 132 may determine whether the object of interest is the same as a previously identified object corresponding to the past time shape information based on the similarity between the specific time shape information and the past time shape information. Here, the previously identified object may include one or more objects of interest identified at a past time point.

[0066] In the example, the object tracking unit 132 may determine whether the object of interest is the same as the previously identified object based on the inner product of the feature vector of the specific time shape information and the feature vector of the past time shape information. The object tracking unit 132 may determine whether the object of interest is the same as the previously identified object based on the cosine similarity between the feature vector of the specific time shape information and the feature vector of the past time shape information. Here, the cosine similarity may be represented by the following formula 1.

[0067] [Formula 1]

[0068]

[0069] In the above formula 1, f t The feature vector representing the shape information at a specific time, f t-1 A feature vector representing the shape information of the past time.

[0070] In the example, the object tracking unit 132 may determine whether the object of interest is the same as the previously identified object based on a cost matrix. The object tracking unit 132 may determine whether the object of interest is the same as the previously identified object based on a cost matrix including a cost value based on at least one of intersection-over-union (IoU), Mahalanobis distance, and cosine similarity. Here, the cost matrix may be a matrix including multiple cost values ​​based on one or more variables. IoU may be an indicator for checking how much an area at a specific time point and an area at a past time point overlap each other. The Mahalanobis distance may indicate the distance between two objects selected in consideration of the correlation between multiple variables. Since the object tracking unit 132 also considers cosine similarity in addition to or as an alternative to IoU and Mahalanobis distance, the object tracking unit 132 may accurately distinguish multiple objects even when multiple objects overlap each other in an image frame at a past time point or an image frame at a specific time point. The cost value may be expressed by the following formula 2.

[0071] [Formula 2]

[0072] C ij =λ·d 1(i, j)+(1-λ)·d 2 (i, j)

[0073] In the above formula 2, "i" represents the identifier of an object detected at a specific time point, and "j" represents the identifier of an object detected at a past time point. ij is the cost value between the i-th object at a specific time point and the i-th object at a past time point, d 1 (i, j) is the Mahalanobis distance between the i-th object at a specific time point and the i-th object at a past time point, d 2 (i, j) is the cosine similarity between the i-th object at a specific time point and the i-th object at a past time point.

[0074] The object tracking unit 132 may track the action state of the object of interest. The object tracking unit 132 may track the action state of the object of interest based on the result of determining whether the object of interest is the same as a previously recognized object.

[0075] In the example, if the object of interest is the same as the previously identified object, the object tracking unit 132 may update the action state of the previously identified object. For example, if it is determined that the object of interest is the same as the previously identified object, the object tracking unit 132 may update the position of the object of interest at the past time point to change to the position of the object of interest at the specific time point.

[0076] In an example, if the object of interest is different from the previously recognized object, or if there is no recognized object at a past time point, the object tracking unit 132 may assign a new identifier (ID) to the object of interest. The object tracking unit 132 may track the action state of the object of interest assigned with the new identifier.

[0077] Figure 2 An example of a method according to an example of the present invention is shown, in which the object tracking apparatus 10 acquires a first resolution image 200 and a second resolution image 220 .

[0078] refer to Figure 2 , the object tracking device 10 may acquire the first resolution image 200 captured by the sensor 14. The object tracking device 10 may acquire the first resolution image 200 captured by the camera. The object tracking device 10 may acquire the first resolution image 200 through the first resolution image acquisition unit 122.

[0079] The second resolution image acquisition unit 124 may acquire the second resolution image 220. In an example, the second resolution image acquisition unit 124 may acquire the second resolution image 220 captured by the sensor 14. The second resolution image acquisition unit 124 may acquire the second resolution image 220 captured by the camera. In an example, the second resolution image acquisition unit 124 may acquire the second resolution image 220 by adjusting the resolution of the first resolution image 200. The second resolution image acquisition unit 124 may acquire the second resolution image 220 by adjusting the size of the first resolution image 200.

[0080] The attention object information generating unit 126 may generate attention object information including position information of the attention object based on the second resolution feature.

[0081] The first resolution feature generation unit 128 may extract the specific area 210 from the first resolution image 200 based on the position information of the second resolution image 220. The first resolution feature generation unit 128 may crop the specific area 210 from the first resolution image 200 based on the position information of the second resolution image 220. The first resolution feature generation unit 128 may crop the specific area 210 from the first resolution image 200, which corresponds to the coordinate value of the bounding box including the object of interest in the second resolution image.

[0082] Figure 3 An example of a method according to an example of the present invention is shown, in which the object tracking device 10 obtains a first resolution feature 312 and a second resolution feature 322 .

[0083] refer to Figure 3 , the first resolution feature generation unit 128 may generate the first resolution feature 312 based on the specific area 210 of the first resolution image 200. The first resolution feature generation unit 128 may generate the first resolution feature 312 by applying a convolution operation to the specific area 210. The first resolution feature generation unit 128 may generate the first resolution feature 312 using the first convolution layer 310. Here, the first resolution feature generation unit 128 may include a backbone (e.g., an encoder), a neck (neck) for extracting a specific scale head, and a head (head) for collecting a plurality of extracted features and outputting a final output value.

[0084] The attention object information generating unit 126 may generate the second resolution feature 322 based on the second resolution image 220. The attention object information generating unit 126 may generate the second resolution feature 322 by applying a convolution operation to the second resolution image 220. The attention object information generating unit 126 may generate the second resolution feature 322 using the second convolution layer 320.

[0085] Figure 4An example of a method according to an example of the present invention is shown, in which the object tracking device 10 outputs specific time shape information using an average pooling operation.

[0086] refer to Figure 4 , the object tracking device 10 may use a pooling layer to generate a feature vector 410 of the first resolution feature 312 and a feature vector 420 of the second resolution feature 322. The object tracking device 10 may use an average pooling technique to generate a feature vector 410 of the first resolution feature 312 and a feature vector 420 of the second resolution feature 322.

[0087] The object tracking device 10 may generate an integrated feature vector 430 by integrating the feature vector 410 of the first resolution feature 312 and the feature vector 420 of the second resolution feature 322 .

[0088] The object tracking device 10 may input the integrated feature vector 430 into a multilayer perceptron (MLP), and may output specific time shape information 440. Here, the multilayer perceptron may refer to a partial structure of a neural network including a perceptron-based layer. Each layer may process input information and may pass the processed information as an input value to the next layer.

[0089] Figure 5 An example of a cost matrix according to an example of the present invention is shown.

[0090] refer to Figure 5 , the object tracking device 10 may determine whether the object of interest detected at a specific time point is the same as a previously identified object corresponding to past time appearance information at a past time point.

[0091] The object tracking device 10 may generate a cost matrix using the above formula 2, which expresses a cost value for each detected object including the object of interest.

[0092] The object tracking device 10 may determine whether the i-th object at a specific time point is the same as the j-th object at a past time point. The object tracking device 10 may determine whether the i-th object at a specific time point is the same as the j-th object at a past time point based on the cost value. If the cost value is maximized, the object tracking device 10 may match the i-th object at the specific time point and the j-th object at the past time point. The object tracking device 10 may select the cost value so that a plurality of rows "j" (=1, 2, and 3) correspond one-to-one to a plurality of columns "i" (=1, 2, and 3). For example, if the object tracking device 10 selects 0.4 as the cost value 500 for the first row (j=1) and the second column (i=2) of a matrix including a plurality of rows "j" (=1, 2, and 3) and a plurality of columns "i" (=1, 2, and 3), it is impossible to select 0.3 and 0.5 as the remaining cost values ​​for the first row (j=1). Furthermore, if the object tracking apparatus 10 selects 0.4 as the cost value 500 corresponding to the second column (i=2), it is impossible to select 0.1 and 0.2 as the remaining cost values ​​of the second column (i=2).

[0093] The object tracking device 10 may select the cost values ​​500, 502, and 504 so as to maximize the sum of the cost values ​​of the i-th object at a specific time point and the j-th object at a past time point. For example, the object tracking device 10 may determine that the sum of 0.4 of the cost value 500 as the first row and the second column, 0.8 of the cost value 502 as the second row and the first column, and 0.5 of the cost value 504 as the third row and the third column is the maximum. Accordingly, the object tracking device 10 may determine that the first object at a past time point is the second object at a specific time point, may determine that the second object at a past time point is the first object at a specific time point, and may determine that the third object at a past time point is the third object at a specific time point.

[0094] Figure 6 An example of an operation method of the object tracking device 10 according to an example of the present invention is shown.

[0095] refer to Figure 6 In operation 600, the first resolution image acquisition unit 122 may acquire a first resolution image. The first resolution image acquisition unit 122 may acquire a first resolution image at a specific time point. The first resolution image acquisition unit 122 may acquire a first resolution image captured around the vehicle using a camera.

[0096] The second resolution image acquisition unit 124 may acquire a second resolution image. The second resolution image acquisition unit 124 may acquire a second resolution image corresponding to the first resolution image. The second resolution image acquisition unit 124 may acquire a second resolution image at a specific time point. The second resolution image acquisition unit 124 may acquire a second resolution image having a lower quality than the first resolution.

[0097] In an example, the second resolution image acquisition unit 124 may acquire the second resolution image by adjusting the resolution of the first resolution image. The second resolution image acquisition unit 124 may acquire the second resolution image by adjusting the size of the first resolution image.

[0098] In an example, the second resolution image acquisition unit 124 may acquire a second resolution image captured around the vehicle using a camera.

[0099] In operation 602 , the attention object information generation unit 126 may generate a second resolution feature associated with the attention object based on the second resolution image.

[0100] In operation 604 , the attention object information generation unit 126 may generate attention object information including position information of the attention object based on the second resolution feature.

[0101] In operation 606, the first resolution feature generation unit 128 may generate a first resolution feature associated with a specific area in the first resolution image, which corresponds to the position information of the second resolution image. The first resolution feature generation unit 128 may generate a first resolution feature associated with a specific area in the first resolution image, which corresponds to the coordinate value of the bounding box including the object of interest in the second resolution image.

[0102] In operation 608, the appearance information output unit 130 may output the appearance information of the object of interest based on the first resolution feature and the second resolution feature. The appearance information output unit 130 may output the time-specific appearance information of the object of interest based on the first resolution feature and the second resolution feature.

[0103] In an example, the shape information output unit 130 may output the specific time shape information by applying a pooling operation to the first resolution feature and the second resolution feature. In an example, the shape information output unit 130 may output the specific time shape information using an average pooling operation performed based on the first resolution feature and the second resolution feature. In an example, the shape information output unit 130 may output the specific time shape information using a maximum pooling operation performed based on the first resolution feature and the second resolution feature.

[0104] In operation 610, the object tracking section 132 may determine whether the object of interest is the same as the previously recognized object. The object tracking section 132 may determine whether the object of interest is the same as the previously recognized object based on similarity between the specific time shape information and the past time shape information.

[0105] In an example, the object tracking unit 132 may determine whether the object of interest is the same as a previously identified object based on the inner product of the feature vector of the specific time shape information and the feature vector of the past time shape information. The object tracking unit 132 may determine whether the object of interest is the same as a previously identified object based on the cosine similarity between the feature vector of the specific time shape information and the feature vector of the past time shape information.

[0106] In an example, the object tracking unit 132 may determine whether the object of interest is the same as the previously identified object based on the cost matrix. The object tracking unit 132 may determine whether the object of interest is the same as the previously identified object based on the cost matrix including a cost value based on at least one of intersection over union (IoU), Mahalanobis distance, and cosine similarity.

[0107] If the object of interest is the same as the previously recognized object, the object tracking section 132 may update the action state of the previously recognized object in operation 612 .

[0108] If the object of interest is different from a previously recognized object, or if there is no recognized object at a past point in time, the object tracking section 132 may assign a new identifier (ID) to the object of interest in operation 614 .

[0109] In operation 616 , the object tracking unit 132 may track the action state of the object of interest to which the new identifier is assigned.

[0110] The present invention is to solve the above-mentioned problems in the prior art while maintaining the advantages achieved by the prior art unchanged.

[0111] An example of the present invention provides an object tracking apparatus and an operating method thereof, which can prevent identifiers assigned to a plurality of objects from being switched even if the plurality of objects overlap each other by further considering an appearance feature of an object of interest acquired for each image frame.

[0112] The technical problems to be solved by the present invention are not limited to the above-mentioned problems, and any other technical problems not mentioned here will be clearly understood by those skilled in the art to which the present invention belongs from the following description.

[0113] According to an example of the present invention, an object tracking device may include: a first resolution image acquisition unit, which acquires a first resolution image, in which the first resolution image includes an object of interest at a specific time point; a second resolution image acquisition unit, which acquires a second resolution image corresponding to the first resolution image; an object of interest information generation unit, which generates a second resolution feature of the object of interest based on the second resolution image, and generates object of interest information including position information of the object of interest based on the second resolution feature; a first resolution feature generation unit, which generates a first resolution feature associated with a specific area of ​​the first resolution image, which corresponds to the position information; an appearance information output unit, which outputs specific time appearance information of the object of interest at a specific time point based on the first resolution feature and the second resolution feature; and an object tracking unit, which tracks the action state of the object of interest based on the specific time appearance information and the past time appearance information corresponding to a past time point before the specific time point.

[0114] In an example, the shape information output unit may output the specific time shape information by applying a pooling operation to the first resolution feature and the second resolution feature.

[0115] In an example, the object tracking unit may determine whether the object of interest is the same as a previously identified object corresponding to the past time appearance information based on the similarity between the specific time appearance information and the past time appearance information, and may track the action state of the object of interest based on the determination result.

[0116] In an example, the object tracking section may determine whether the object of interest is the same as a previously recognized object based on cosine similarity between the specific time shape information and the past time shape information.

[0117] In an example, the object tracking section may determine whether the object of interest is the same as a previously recognized object based on at least one of intersection over union (IoU), Mahalanobis distance, and cosine similarity.

[0118] In an example, if the object of interest is different from a previously recognized object or if there is no previously recognized object, the object tracking section may assign a new identifier (ID) to the object of interest and may track the action state of the object of interest.

[0119] In an example, the second resolution image acquisition unit may acquire the second resolution image by adjusting the resolution of the first resolution image.

[0120] In an example, the specific area is an area where a result of converting the position information based on a ratio of a size of the second resolution image to a size of the first resolution image is applied to the first resolution image.

[0121] According to an example of the present invention, an operating method of an object tracking device may include: acquiring a first resolution image including an object of interest at a specific time point; acquiring a second resolution image corresponding to the first resolution image; generating a second resolution feature associated with the object of interest based on the second resolution image; generating object of interest information including location information of the object of interest based on the second resolution feature; generating a first resolution feature associated with a specific area of ​​the first resolution image corresponding to the location information; outputting specific time shape information of the object of interest at a specific time point based on the first resolution feature and the second resolution feature; and tracking an action state of the object of interest based on the specific time shape information and past time shape information corresponding to a past time point before the specific time point.

[0122] In an example, outputting the specific time shape information may include outputting the specific time shape information by applying a pooling operation to the first resolution feature and the second resolution feature.

[0123] In an example, the method may further include determining whether the object of interest is the same as a previously identified object corresponding to the past time profile information based on similarity between the specific time profile information and the past time profile information. Tracking the action state of the object of interest may include tracking the action state of the object of interest based on the determination result.

[0124] In an example, determining whether the object of interest is the same as a previously recognized object may include determining whether the object of interest is the same as a previously recognized object based on a cosine similarity between the specific temporal profile information and the past temporal profile information.

[0125] In an example, determining whether the object of interest is the same as a previously identified object may include determining whether the object of interest is the same as a previously identified object based on at least one of intersection over union (IoU), Mahalanobis distance, and cosine similarity.

[0126] In an example, the method may further include assigning a new identifier (ID) to the object of interest if the object of interest is different from a previously identified object or if it is determined that there is no previously identified object.

[0127] In an example, the acquiring of the second resolution image may include acquiring the second resolution image by adjusting a resolution of the first resolution image.

[0128] In an example, the specific area is an area where a result of converting the position information based on a ratio of a size of the second resolution image to a size of the first resolution image is applied to the first resolution image.

[0129] The above terms such as "comprise / include" and "having", unless clearly stated to the contrary, mean that the corresponding components may be included, and should be interpreted as further including another component without excluding another component. Unless otherwise defined herein, all terms used herein, including technical or scientific terms, may have the same meaning as commonly understood by a person skilled in the art to which the examples disclosed in this specification belong. Commonly used terms, such as those defined in dictionaries, should be interpreted as having a meaning consistent with the meaning in the context of the relevant technology, and should not be interpreted as having an idealized or overly formal meaning unless clearly defined in this specification.

[0130] According to various examples of the present invention, an object tracking apparatus and an operating method thereof may utilize lower resolution images to reduce resources.

[0131] According to various examples of the present invention, an object tracking apparatus and an operating method thereof can utilize position information of a bounding box of an object of interest including a lower resolution image to extract features of the object of interest from a higher resolution image, thereby reducing information loss when only the lower resolution image is used.

[0132] According to various examples of the present invention, in addition to or instead of IoU and Mahalanobis distance, the object tracking device and the operating method thereof may also consider the appearance features of the object of interest acquired for each image frame, so that even if multiple objects overlap with each other, the identifiers respectively assigned to the multiple objects will not be switched.

[0133] Effects of the object tracking device and the operation method thereof according to the disclosure of the present specification are not limited to the above-mentioned effects, and other effects not mentioned can be clearly understood by those skilled in the art based on the disclosure of the present specification.

[0134] In the above, specific examples are illustrated and described. However, the present invention is not limited to these examples. Those skilled in the art to which the present invention belongs will be able to make various changes without departing from the gist of the technical idea of ​​the present invention as set forth in the following claims.

Claims

1. A device comprising: processor, and a memory storing instructions that, when executed by the processor, cause the apparatus to: receiving a first resolution image associated with a first time and including an object of interest; generating a second resolution image based on the first resolution image; determining a second resolution feature of the object of interest based on the second resolution image; Determine the object of interest information including the location information of the object of interest based on the second resolution feature; determining a first resolution feature associated with a region of the first resolution image based on the position information; outputting temporal shape information indicating a shape of the object of interest associated with the first time based on the first resolution feature and the second resolution feature; tracking the action state of the object of interest based on the time appearance information and past time appearance information, wherein the past time appearance information indicates an appearance of the object of interest associated with a second time before the first time; as well as Based on the tracked action state of the object of interest, a signal for controlling the operation of the second device is output.

2. The device according to claim 1, wherein: When the instructions are executed by the processor, the device is caused to: output the temporal shape information by applying a pooling operation to the first resolution features and the second resolution features.

3. The device according to claim 1, wherein: When the instructions are executed by the processor, the device: determining whether the object of interest is the same as an object previously identified at the second time based on a similarity between the temporal profile information and the past temporal profile information; and Based on determining that the objects of interest are the same, an action state of the objects of interest is tracked.

4. The device according to claim 3, wherein: When the instructions are executed by the processor, the device determines whether the object of interest is the same as the previously identified object based on the cosine similarity between the temporal shape information and the past temporal shape information.

5. The device according to claim 3, wherein: When the instructions are executed by the processor, the apparatus is caused to: determine whether the object of interest is the same as the previously identified object based on at least one of intersection over union (IoU), Mahalanobis distance, and cosine similarity.

6. The device according to claim 3, wherein: When the instructions are executed by the processor, the device: assigning a new identifier to the object of interest based on the object of interest being different from the previously identified object or based on the object of interest not being previously identified; and Based on the new identifier, the action status of the object of interest is tracked.

7. The device according to claim 1, wherein: When the instructions are executed by the processor, the device is enabled to: obtain the second resolution image by adjusting the resolution of the first resolution image.

8. The device according to claim 1, wherein: When the instructions are executed by the processor, the device: Based on the position information and based on a ratio of a size of the second resolution image to a size of the first resolution image, a region of the first resolution image is determined.

9. A method comprising the following steps: receiving a first resolution image associated with a first time and including an object of interest; generating a second resolution image based on the first resolution image; determining a second resolution feature associated with the object of interest based on the second resolution image; Determine the object of interest information including the location information of the object of interest based on the second resolution feature; determining a first resolution feature associated with a region of the first resolution image based on the position information; outputting temporal shape information indicating a shape of the object of interest associated with the first time based on the first resolution feature and the second resolution feature; tracking the action state of the object of interest based on the time appearance information and past time appearance information, wherein the past time appearance information indicates the appearance of the object of interest associated with a second time before the first time; and Based on tracking the action state of the object of interest, a signal for controlling the operation of the device is output.

10. The method according to claim 9, wherein: Outputting the time profile information includes: A pooling operation is applied to the first resolution features and the second resolution features.

11. The method according to claim 9, further comprising: determining whether the object of interest is the same as a previously identified object based on a similarity between the temporal profile information and the past temporal profile information, The tracking of the action status of the object of interest includes: Based on the result of the determination, the action status of the object of interest is tracked.

12. The method according to claim 11, wherein: Determining whether the object of interest is the same as a previously identified object includes determining whether the object of interest is the same as the previously identified object based on a cosine similarity between the temporal profile information and the past temporal profile information.

13. The method according to claim 11, wherein: Determining whether the object of interest is the same as a previously identified object includes determining whether the object of interest is the same as the previously identified object based on at least one of intersection over union (IoU), Mahalanobis distance, and cosine similarity.

14. The method according to claim 11, further comprising: Assigning a new identifier to the object of interest based on at least one of: The object of interest is different from the previously identified object, or The object of interest has not been previously identified.

15. The method according to claim 9, wherein: Receiving the second resolution image includes: adjusting the resolution of the first resolution image.

16. The method according to claim 9, further comprising: Based on the position information and based on a ratio of a size of the second resolution image to a size of the first resolution image, a region of the first resolution image is determined.

Images