Apparatus and Method for Tracking Moving Object based on Sparse Optical Flow

Abstract

The disclosed embodiment provides a moving object tracking device and method capable of detecting and tracking moving objects accurately and robustly against noise, by detecting multiple corner points in a sparse optical flow obtained from multiple frames of an image, identifying corner points of a moving object based on the displacement distance of the detected corner points to set a region of interest, dividing the region of interest based on the aspect ratio of the set region of interest and a reference ratio according to the object of interest to be identified, and identifying an object in each region of interest and determining the object region through neural network operation, thereby enabling real-time object tracking with a small amount of computation.

Description

Technology Field

[0001] The disclosed embodiments relate to a moving object tracking device and method, and more specifically, to a sparse optical flow-based moving object tracking device and method. Background Technology

[0002] Moving object detection and tracking technology is being applied across a wide range of research fields, including traffic monitoring and the recognition of workers in surrounding heavy equipment environments. Algorithms utilized for this purpose include frame difference algorithms, background subtraction algorithms, optical flow-based algorithms, and static learning algorithms. However, these existing algorithms have limitations in that they cannot effectively detect and track objects when input images are distorted by various factors, such as objects being obscured by obstacles or camera vibrations caused by the surrounding environment. This implies that object detection performance is poor for images acquired in harsh environments, such as CCTVs at construction sites or along roadsides.

[0003] Recently, deep learning-based algorithms utilizing artificial neural networks have been widely used. Deep learning-based algorithms are utilized in various fields due to their advantages, such as robustness against noise and excellent object detection and tracking performance even in complex situations. However, deep learning-based algorithms require not only sufficient prior training but also the execution of massive computations, which necessitates high-cost, high-performance computing equipment. In other words, when deep learning-based algorithms are executed using low-cost, low-performance computing equipment, there is a limitation in that real-time object tracking is difficult due to low computational power.

[0004] Meanwhile, most existing optical flow-based algorithms have been studied based on high-density dense optical flow. However, even when using dense optical flow, a large amount of computation is required, which not only makes real-time object tracking difficult but also presents a problem of vulnerability to noise. Prior art literature

[0005] Korean Registered Patent No. 10-1885839 (Registered on July 31, 2018) The problem to be solved

[0006] The disclosed embodiments aim to provide a moving object tracking device and method that can detect and track moving objects in real time using sparse optical flow requiring low computational power, while being robust against noise.

[0007] The disclosed embodiments aim to provide a moving object tracking device and method capable of accurately detecting a moving object by applying a moving window memory and dividing the region of interest based on the aspect ratio of the detected region of interest and a reference ratio according to the basic shape of the object of interest to be detected. means of solving the problem

[0008] A moving object tracking device according to an embodiment comprises one or more processors; and a memory for storing one or more programs executed by the one or more processors, wherein the processor detects a plurality of corner points in a sparse optical flow obtained from a plurality of frames of an image, identifies a corner point for a moving object according to the movement distance of the detected corner point to set a region of interest, divides the region of interest according to the aspect ratio of the set region of interest and a reference ratio according to the object of interest to be identified, identifies an object in each region of interest through neural network operation, and determines the object region.

[0009] The processor compares the aspect ratio of the region of interest with a plurality of predetermined reference ratios according to the type and number of objects of interest, determines the interval of the number of objects of interest that includes the aspect ratio, and can divide the region of interest into multiple parts according to the determined number of objects of interest.

[0010] The processor can divide the region of interest in a direction determined according to the basic shape of the object of interest.

[0011] The processor detects multiple corner points in a sparse optical flow obtained from multiple frames of an image, allocates a moving window that stores the position coordinate changes of each of the detected corner points in the multiple frames, and can identify corner points of a moving object and set a region of interest based on the distance of each corner point calculated based on the position coordinates stored in the moving window allocated to each corner point.

[0012] The processor may assign an identifier to distinguish a detected corner point from other corner points if it is a new corner point, and store the position coordinates by allocating the moving window having a memory space of a specified size to store the position coordinates of each corner point in multiple frames distinguished according to the identifier.

[0013] If the detected corner point is a corner point at a position corresponding to the existing corner point according to the sparse optical flow, the processor may assign an identifier of the existing corner point and store the position coordinates of the detected corner point in a moving window assigned to the existing corner point.

[0014] If the location coordinates of the detected corner point are within a reference distance from the location coordinates of the existing corner point detected within a specified time interval, the processor may assign an identifier of the existing corner point and store the location coordinates of the detected corner point in a moving window assigned to the existing corner point.

[0015] The processor can acquire input data by preprocessing the size and shape of the region of interest so that they become a predetermined size and shape according to the shape of the object of interest, and can identify the object and determine the object region by performing neural network operations on the input data.

[0016] The above processor can track the movement path of an object by checking the object area of ​​an object identified identically in multiple frames among the identified objects.

[0017] A moving object tracking method according to an embodiment is a method performed by a computing device having one or more processors and a memory for storing one or more programs executed by said one or more processors, comprising the steps of: detecting a plurality of corner points in a sparse optical flow obtained from a plurality of frames of an image, and identifying a corner point for a moving object according to the movement distance of the detected corner point to set a region of interest; dividing the region of interest according to the aspect ratio of the set region of interest and a reference ratio according to the object of interest to be identified; and identifying an object in each region of interest and determining the object region through neural network operation. Effects of the invention

[0018] Accordingly, the moving object tracking device and method according to the embodiment can enable real-time object tracking by requiring a small amount of computation using sparse optical flow, and can accurately detect and track moving objects robust to noise by applying a moving window memory to identify objects by dividing the region of interest based on the aspect ratio of the detected region of interest and a reference ratio based on the basic shape of the object of interest to be detected. Brief explanation of the drawing

[0019] FIG. 1 shows a configuration of a moving object tracking device according to one embodiment, classified according to operation. Figure 2 is a diagram illustrating the operation of the moving window storage module of Figure 1. Figure 3 is a diagram illustrating the operation of the region of interest extraction module of Figure 1. Figure 4 shows an example of the detailed configuration of the object identification module of Figure 1. FIG. 5 illustrates a method for tracking a moving object according to one embodiment. FIG. 6 is a diagram illustrating a computing environment including a computing device according to one embodiment. Specific details for implementing the invention

[0020] Hereinafter, a specific embodiment of one embodiment will be described with reference to the drawings. The following detailed description is provided to facilitate a comprehensive understanding of the methods, devices, and / or systems described herein. However, this is merely illustrative and the invention is not limited thereto.

[0021] In describing the embodiments, if it is determined that a detailed description of known technology related to the present invention might unnecessarily obscure the essence of the embodiment, such detailed description will be omitted. Furthermore, the terms described below are defined with consideration of their functions in the present invention, and these may vary depending on the intentions or practices of the user or operator. Therefore, such definitions should be based on the content throughout this specification. Terms used in the detailed description are intended merely to describe the embodiments and should not be limiting. Unless explicitly stated otherwise, expressions in the singular form include the meaning of the plural form. In this description, expressions such as "include" or "comprise" are intended to refer to certain characteristics, numbers, steps, actions, elements, parts thereof, or combinations thereof, and should not be interpreted to exclude the existence or possibility of one or more other characteristics, numbers, steps, actions, elements, parts thereof, or combinations thereof other than those described. Additionally, terms such as "...part," "...unit," "module," and "block" described in the specification refer to a unit that processes at least one function or operation, and this may be implemented in hardware, software, or a combination of hardware and software.

[0022] FIG. 1 shows a configuration of a moving object tracking device according to one embodiment classified according to operation, FIG. 2 is a diagram for explaining the operation of the region of interest acquisition module of FIG. 1, and FIG. 3 is a diagram for explaining the operation of the region of interest extraction module of FIG. 1.

[0023] Referring to FIG. 1, a moving object tracking device according to an embodiment may include an image acquisition module (10), a sparse optical flow acquisition module (20), a region of interest acquisition module (30), a preprocessing module (40), and an object identification module (50).

[0024] The image acquisition module (10) acquires an image for detecting a moving object. At this time, the image acquisition module (10) acquires an image composed of a number of consecutive frames so as to detect a moving object. The image acquisition module (10) can acquire an image by being implemented as an image acquisition device such as a CCTV or a camera, or it can acquire an image by being implemented as a database (not shown) in which a number of images are stored, or by being implemented as a communication module that receives images through a network with an external device.

[0025] The sparse optical flow acquisition module (20) acquires sparse optical flow based on pixel changes between two consecutive frames among a number of frames included in the image. As mentioned above, most existing optical flow-based algorithms have been studied based on high-density dense optical flow, but dense optical flow requires high computational performance because it requires analysis of a large number of optical flows and thus requires a large amount of computation. Therefore, it is not suitable for identifying objects in real time.

[0026] Accordingly, in the moving object tracking device of the embodiment, the image acquisition module (10) acquires a sparse optical flow in the acquired image, which has fewer optical flows compared to a dense optical flow, thereby reducing the amount of computation. In consecutive frames of the image, the sparse optical flow can be acquired based on, for example, the Lucas-Kanade pyramid method, and can also be acquired using other methods. Since the method for acquiring the sparse optical flow is a known technology, it is not described in detail here.

[0027] When a sparse optical flow is acquired among multiple frames included in the image, the region of interest acquisition module (30) analyzes the acquired sparse optical flow and sets the area presumed to contain a moving object as the region of interest (ROI). In this case, the region of interest acquisition module (30) of the embodiment estimates the region of interest for the moving object based on the change in the sparse optical flow acquired over multiple frames, rather than just the sparse optical flow acquired in two consecutive frames. Furthermore, the region of interest acquisition module (30) of the embodiment may divide the region of interest into multiple parts based on the aspect ratio of the estimated region of interest and the ratio of the object of interest to be detected.

[0028] The region of interest acquisition module (30) may include a corner point detection module (31), a moving window storage module (32), a region of interest setting module (33), and a region of interest segmentation module (34).

[0029] The corner point detection module (31) detects multiple corner points in the acquired sparse optical flow. Corner points are feature points frequently used in sparse optical flow techniques, and when optical flow occurs due to the movement of an object, the corner points also move along with the moving object. Corner points may be detected together during sparse optical flow detection, and for example, they may be detected according to the Shi-tomasi corner detection technique, but they may also be acquired by other techniques. Since the technique for detecting corner points is also a known technique, it is not described in detail here.

[0030] The moving window storage module (32) assigns an identifier to each of the multiple corner points detected by the corner point detection module (31) to distinguish each corner point, and stores the moving position of the corner point over multiple frames for a specified period according to the optical flow of each distinguished corner point. The moving window storage module (32) is a frame (f) of the current time point (t). t) and the previous time point (t-1) frame (f t-1 When a new corner point that has not previously been detected is detected in the sparse optical flow obtained between, an identifier is assigned to distinguish the detected corner point from the previously detected corner point, and a moving window is established by allocating a memory space of a specified size to store the position in each subsequent frame for each of the multiple corner points assigned the identifier.

[0031] And if the detected corner point is a corner point that has already been detected in previous frames, assigned an identifier, allocated memory space, and set up a moving window, the location of the detected corner point is stored in the set moving window. Due to the characteristics of the optical flow, the moving window storage module (32) can easily determine whether it is a previously detected corner point or a newly detected corner point. Since the optical flow indicates a change in the position of a specific pixel between consecutive frames, the position of a corner point detected in a specific frame in the next frame can be confirmed from the optical flow, and accordingly, the moving window storage module (32) can easily distinguish between a new corner point and a corner point that has already been detected previously and has the same identifier among the multiple corner points detected in each of the multiple frames, and store them in the same moving window.

[0032] That is, the moving window (MW) for each corner point assigned identifier (i). i,t ) is multiple frames (f t-n ~ f t-1 As a memory space allocated to accumulate and store changes in the position of the corresponding corner point during ), as shown in Equation 1, a specified number (e.g., n) of frames (f t-n ~ f t-1 The position coordinates (P) of the corner point according to each identifier (i) during ) i,t-n ~ P i,t-1 ) can be stored.

[0033]

[0034] At this time, the moving window storage module (32) can store the position of the corner point of the same identifier in the moving window, even if there is some error between each previously detected corner point and the position of the optical flow, by taking into account a certain level of error in the acquired sparse optical flow. In addition, in some cases, even if the corner point assigned to the previous identifier is not detected from the optical flow acquired between some frames among a number of frames, if a corner point within the error range is detected from the optical flow acquired between subsequent frames, it can be determined as the corner point of the same identifier and stored in the moving window.

[0035] Sparse optical flow has the advantage of faster computation compared to dense optical flow, as the number of acquired optical flows is smaller, thereby reducing the computational load. However, due to the limited number of optical flows, it is susceptible to large object movements or noise within the image, and optical flow calculations are limited by obstacles and backgrounds. Consequently, sparse optical flows acquired from multiple frames may contain a certain level of error, and in particular, corner points may not be detected between some of the frames.

[0036] Conventional object detection techniques based on dense optical flow utilize a large volume of high-density optical flows. Since object movement can be identified from other flows even if errors exist in some, all previously detected corner points are reset for each frame to detect new ones. However, in sparse optical flow, the smaller number of flows can lead to relatively larger errors or the omission of corner points in certain frame segments compared to dense optical flow. For instance, errors in the location of corner points can occur due to various factors, such as when an object suddenly stops while moving in a specific direction, reverses its direction of movement, or disappears behind an obstacle; consequently, corner points may be lost in certain frame segments. Furthermore, when tracking object movement based on sparse optical flow, errors in corner point locations or loss of corner points due to the characteristics of sparse optical flow make it difficult to identify the moving object. In other words, assigning a new identifier to the corner point of the same object in a subsequent frame may result in the loss of information regarding the object's previous movement.

[0037] Accordingly, the moving window storage module (32) assigns a moving window to each newly detected corner point so as to detect changes in the position of a corner point over multiple frames, and by comparing the position information stored in the moving window of an existing corner point that has an error within a certain range or a corner point that was not detected for a certain time interval (τ) and is detected again, the moving window storage module (32) ensures that the corner point has the same identifier as the existing detected corner point and stores the position coordinates of the corner point in the moving window. That is, the moving window storage module (32) ensures that the continuity of each corner point is maintained even if an error within a certain distance or an error occurs in a certain time interval, thereby enabling robust verification of changes in the position of the corner point against noise. In addition, when using a moving window, the movement of the corner point is detected during a frame interval corresponding to the size of the moving window rather than between adjacent frames, thus making it easier to detect moving objects.

[0038] The interest region setting module (33) sets an interest region, which is an area in the image where an object is presumed to have moved, based on a plurality of moving windows in which the position coordinates of each of the detected plurality of corner points have moved over a plurality of frames are accumulated and stored.

[0039] The interest area setting module (33) is the first stored location coordinate (P) among a plurality of location coordinates stored in a moving window distinguished by each identifier (i). i,t-n ) and the current frame (f) of the corresponding corner point t Position coordinates (P) at ) i,t Distance between (L) i,t Calculate ) and the calculated distance (L i,t If ) is greater than or equal to the threshold (α), it is determined that object movement has occurred, and the movement state information (D) of the corresponding corner point i,t ) activates the value for the moving object (e.g., 1). However, the calculated distance (L i,tIf ) is less than the threshold (α), it is determined that no object movement has occurred, and the movement state information (D) of the corresponding corner point i,t Disable ) (e.g., 0).

[0040] This is intended to prevent false detection of moving objects when detecting moving objects from images acquired in harsh environments where vibrations or dust are generated due to the movement of various construction equipment and vehicles, such as CCTVs installed at construction sites or roads along with moving windows.

[0041] Movement state information (D i,t When a moving object is detected according to the activation state of the ), the area of ​​interest setting module (33) sets an area of ​​interest based on the position coordinates of the corner points. The area of ​​interest setting module (33) can set an area of ​​interest based on the position coordinates of the corner points in the current frame. However, since the corner points only represent feature points of the object and do not represent the entire area of ​​the object, the area of ​​interest setting module (33) can set an area of ​​interest by adding margin areas centered on each corner point, and among the areas of interest set, areas of interest that overlap each other can be integrated into a single area of ​​interest as shown in FIG. 2. That is, the areas of interest set by each corner point are integrated so that the entire shape of the object can be included in the area of ​​interest. At this time, the area of ​​interest can be created to have a specified shape (e.g., a rectangular shape).

[0042] When an area of ​​interest is set as in FIG. 2, the area of ​​interest division module (34) can divide the area of ​​interest into multiple parts by comparing the area of ​​interest set by the area of ​​interest setting module (33) with a reference ratio based on the basic shape of the object of interest to be tracked.

[0043] When a region of interest is set to include the entire area of ​​an object, and the regions of interest are integrated because corner points are located adjacent to each other to form overlapping areas, cases may occur where multiple objects are included in a single region of interest.

[0044] In Figure 2, the blue square boxes represent the respective regions of interest. It can be seen that the region of interest on the left contains one person, whereas the blue square box on the right contains three people. This implies that the region of interest was set by simply considering the three people moving together as a single integrated moving object. In other words, the region of interest is set to specify only the area presumed to contain moving objects, and the actual types or number of objects within the region of interest cannot be verified.

[0045] The moving object tracking device of the embodiment aims to identify specific types of objects of interest separately from each other, rather than all objects, and to track the movement of each identified object of interest. For example, in the case of traffic volume surveys, it must be possible to detect only vehicles excluding pedestrians or bicycles, and to track the detected vehicles separately. Furthermore, in the case of construction sites, there may be instances where it is necessary to track the movement of people, specifically individual workers, rather than construction vehicles or equipment.

[0046] However, as mentioned above, the configured region of interest does not distinguish between the types or number of moving objects included, and therefore, if each object is simply identified within the region of interest, identification performance may be degraded.

[0047] Accordingly, the interest area segmentation module (34) divides the interest area into multiple areas by comparing the aspect ratio of the set interest area with the reference ratio based on the basic shape of the interest object being tracked, thereby improving the identification performance of the interest object. For example, if the interest object is a vehicle, the basic shape of the vehicle has a horizontal length longer than the vertical length; therefore, if the length in the vertical direction of the interest area is longer than the horizontal length, the identification performance of the interest object can be improved by dividing the interest area in the horizontal direction. As another example, if the interest object is a person, the basic shape of the person has a vertical length longer than the horizontal length; therefore, if the horizontal width is longer than the vertical width of the interest area, the identification performance of the interest object can be improved by dividing the interest area in the vertical direction. That is, the interest area segmentation module (34) can divide a single interest area into multiple interest areas according to a designated direction based on the reference ratio according to the interest object.

[0048] When multiple people move together, as in the right region of interest of Figure 2, the set horizontal and vertical widths may be nearly similar; however, as mentioned above, since the vertical width of a person must be longer than the horizontal width, the region of interest must be divided into multiple parts. At this time, the number of regions of interest to be divided can also be determined according to the aspect ratio of the region of interest.

[0049] When people move, compared to when each individual moves individually, when two or three or more people move together, they move closer to the people moving with them. In other words, this means that the ratio of the width to the aspect ratio of the region of interest detected for a group moving together does not increase in multiples of the ratio of the width to the aspect ratio of the region of interest detected for a moving individual.

[0050] Accordingly, the interest area division module (34) can divide the interest area into multiple parts by comparing it with a plurality of preset reference ratios based on the aspect ratio of the interest area and the type and number of objects of interest. For example, if the object of interest is a person and the aspect ratio calculated as the ratio (W / H) of the vertical length (H) to the horizontal length (W) of the interest area is less than or equal to the first reference ratio, the interest area is considered to contain one person and is not divided. However, if the aspect ratio exceeds the first reference ratio and is less than or equal to the second reference ratio which is greater than the first reference ratio, the interest area is considered to contain two people and is divided into two parts in the vertical direction so that the aspect ratio of the divided interest area is reduced. Additionally, if the aspect ratio exceeds the second reference ratio and is less than or equal to the third reference ratio which is greater than the second reference ratio, the interest area can be divided into three parts in the vertical direction. At this time, the interest area division module (34) can perform equal division so that the sizes of the divided interest areas are equal to each other, but is not limited thereto.

[0051] In FIG. 3, (a) represents an undivided region of interest, and (b) represents a divided region of interest. In FIG. 3 (a) and (b), the green square box represents an undivided region of interest with an aspect ratio less than or equal to a first reference ratio, whereas the blue square box represents a region of interest with an aspect ratio greater than or equal to a second reference ratio that is greater than or equal to the first reference ratio. Accordingly, the region of interest division module (34) can divide the region of interest indicated by the blue square box in (a) into two as shown in (b).

[0052] In this way, if the interest region segmentation module (34) divides the interest region according to the aspect ratio so that only individual objects are included in each interest region, the object identification module (50) can have high object identification performance compared to the case where multiple objects are included in the interest region.

[0053] The preprocessing module (40) is the current frame (f tA plurality of interest regions set by the interest region acquisition module (30) can be extracted, and the extracted interest regions can be converted to a size suitable for the object identification module (50) to obtain input data.

[0054] In the embodiment, the object identification module (50) is implemented as an artificial neural network, and the input data input to the object identification module (50) implemented as an artificial neural network needs to be normalized as much as possible. However, the region of interest set in the region of interest acquisition module (30) may have various sizes and shapes (aspect ratios). Although the region of interest segmentation module (34) segments the region of interest into shapes according to the object of interest, the size and aspect ratio of the region of interest are not the same. Accordingly, the preprocessing module (40) can acquire input data to be input to the object identification module (50) by performing a size conversion operation so that the shapes of multiple regions of interest acquired by the region of interest acquisition module (30), that is, the size and aspect ratio, become the same. Here, the preprocessing module (40) can also convert the shape of the region of interest to follow a shape pre-specified according to the shape of the object of interest. For example, if the object of interest is a person, the input data can be converted so that the number of horizontal × vertical pixels becomes 64 × 128, and if the object of interest is a vehicle, conversely, it can be converted so that it becomes 128 × 64.

[0055] The preprocessing module (40) can convert the size of the region of interest to become smaller by performing average pooling, etc., when the size of the region of interest is larger than the size of the specified input data. Conversely, when the size of the region of interest is smaller than the size of the specified input data, it can convert the size of the region of interest to become larger by applying interpolation, etc. However, since various techniques for converting the size of an image are known, the preprocessing module (40) may also obtain the input data by converting the size of the region of interest according to other known techniques.

[0056] The object identification module (50) receives the region of interest, i.e., input data, which has been converted to a normalized size by the preprocessing module (40), and performs neural network operations to identify objects included in the input data and determines the object region containing the identified object within the region of interest.

[0057] Figure 4 shows an example of the detailed configuration of the object identification module of Figure 1.

[0058] As illustrated in FIG. 4, the object identification module (50) is implemented as a pre-trained artificial neural network to identify objects and determine object regions. Since the artificial neural network for object identification and object region determination can be implemented as a Convolutional Neural Network (CNN) or various other neural networks and is already known, the moving object tracking device of the embodiment may utilize an artificial neural network already trained according to known technology as the object identification module (50). However, since the moving object tracking device of the embodiment requires real-time high-speed processing, it is preferable that the object identification module (50) be implemented as an artificial neural network with a simple structure containing only three or fewer layers, as illustrated in FIG. 4, in order to reduce the amount of computation.

[0059] At this time, since input data in which the area of ​​interest is divided and resized according to the aspect ratio of the object of interest is input, the object identification module (50) may not be able to perform object identification on some of the input data. However, this is a factor that improves the performance of the moving object tracking device of the embodiment that intends to track only the movement path of the object of interest. For example, when a person is presumed to be the object of interest, if the area of ​​interest acquired for a vehicle is divided and resized and input, the object identification module (50) is less able to identify the object than when the area of ​​interest is input without being divided, and this allows moving objects that are not the object of interest to be easily excluded, thereby making the identification performance for the person even better.

[0060] Meanwhile, the moving object tracking device of the embodiment may further include an object tracking module (60). The object tracking module (60) identifies the same object among the objects identified by the object identification module (50) in a plurality of frames included in the image acquired by the image acquisition module (10), and tracks the movement path of a specific object by identifying the object area in each frame determined for the same object.

[0061] As described above, the moving object tracking device of the embodiment utilizes a sparse optical flow that requires a small amount of computation, and by verifying the positional change of corner points extracted from the sparse optical flow over multiple frames using a movie window, it is possible to detect a region of interest containing a moving object that is robust against noise. In addition, by comparing the aspect ratio of the region of interest with multiple reference ratios based on the shape of the object of interest, the region of interest can be divided into multiple parts, thereby significantly improving the identification and tracking performance of the object of interest.

[0062] In the illustrated embodiments, each component may have different functions and capabilities other than those described below, and may include additional components other than those not described below. Additionally, in one embodiment, each component may be implemented using one or more physically separated devices, or by one or more processors or a combination of one or more processors and software, and may not be clearly distinguished in specific operation as in the illustrated examples.

[0063] And the moving object tracking device illustrated in FIG. 1 may be implemented in a logic circuit by hardware, firmware, software, or a combination thereof, or may be implemented using a general-purpose or specific-purpose computer. The device may be implemented using a hardwired device, a field programmable gate array (FPGA), an application-specific integrated circuit (ASIC), etc. Additionally, the device may be implemented as a system-on-chip (SoC) including one or more processors and controllers.

[0064] In addition, the moving object tracking device may be installed in the form of software, hardware, or a combination thereof on a computing device or server equipped with hardware elements. A computing device or server may refer to various devices that include, in whole or in part, communication devices such as communication modems for communicating with various devices or wired / wireless communication networks, memory for storing data for executing programs, and microprocessors for executing programs to perform calculations and commands.

[0065] FIG. 5 illustrates a method for tracking a moving object according to one embodiment.

[0066] Referring to FIGS. 1 to 4, the moving object tracking method of FIG. 5 first acquires an image in which a moving object is to be detected and tracked (71). Here, the image includes a plurality of frames acquired in succession.

[0067] When an image is acquired, a sparse optical flow is acquired between adjacent frames among multiple frames of the acquired image (72). Then, multiple corner points are detected according to the acquired sparse optical flow (73). Among the detected corner points, a new corner point is assigned an identifier, and a moving window with a specified size memory space is allocated to each corner point assigned an identifier, so that the frame-by-frame position coordinates of the corner point with the same identifier are stored (74). At this time, even if a corner point is not detected in some frame intervals, a corner point detected within a reference range from a previously detected corner point within a specified time interval (τ) can have the same identifier as the previous corner point and its position coordinates can be stored in the same moving window.

[0068] Meanwhile, the movement distance of the corner point stored in each moving window is calculated, and it is determined whether the calculated movement distance is greater than or equal to the reference distance (75). If it is less than the reference distance, the corner point is determined not to be a corner point of the moving object, and another corner point is detected and the movement distance is calculated again. However, if the movement distance is greater than or equal to the reference distance, it is determined to be a corner point of the moving object and a region of interest is set (76). At this time, the region of interest can be set as a range specified around the corner point location in the current frame, and if it overlaps with a region of interest set by another corner point, it is integrated and set. In addition, the region of interest is set as a specified shape (here, as an example, a rectangle).

[0069] When a region of interest is set, the aspect ratio of the set region of interest is compared with a number of reference ratios set according to the type and number of objects of interest, and a reference ratio interval corresponding to the aspect ratio of the region of interest is identified, and the region of interest is divided into multiple parts in a specified direction according to the identified reference ratio interval (77). That is, a number interval of objects of interest that includes the aspect ratio of the region of interest is determined, and the region of interest is divided so that a number of regions of interest according to the determined number interval is obtained. At this time, the region of interest may not be divided according to the aspect ratio of the region of interest.

[0070] When the division of each region of interest is determined and individual regions of interest are obtained, preprocessing is performed on the obtained regions of interest to convert the size of each obtained region of interest into a standard size specified according to the object of interest, thereby obtaining input data (78).

[0071] Then, the acquired input data is input into a pre-trained artificial neural network to perform neural network operations, thereby identifying an object included in the region of interest and detecting the region of the identified object (79). Once the object is identified and the region of each object is detected, the object region identified as the same object on multiple frames of the image is checked to track the movement path of the object (80).

[0072] Although FIG. 5 describes each process as being executed sequentially, this is merely an illustrative description, and a person skilled in the art can apply various modifications and variations by changing the order described in FIG. 5, executing one or more processes in parallel, or adding other processes, within the scope of not departing from the essential characteristics of the embodiment of the present invention.

[0073] FIG. 6 is a diagram illustrating a computing environment including a computing device according to one embodiment.

[0074] In the illustrated embodiments, each component may have different functions and capabilities in addition to those described below, and may include additional components in addition to those not described below. The illustrated computing environment (90) may include a computing device (91) to perform the moving object tracking method illustrated in FIG. 5. In one embodiment, the computing device (91) may be one or more components included in the moving object tracking device illustrated in FIG. 1.

[0075] A computing device (91) includes at least one processor (92), a computer-readable storage medium (93), and a communication bus (95). The processor (92) may enable the computing device (91) to operate according to the exemplary embodiment described above. For example, the processor (92) may execute one or more programs (94) stored in the computer-readable storage medium (93). The one or more programs (94) may include one or more computer-executable instructions, and the computer-executable instructions may be configured to enable the computing device (91) to perform operations according to the exemplary embodiment when executed by the processor (92).

[0076] The communication bus (95) interconnects various other components of the computing device (91), including the processor (92) and the computer-readable storage medium (93).

[0077] The computing device (91) may also include one or more input / output interfaces (96) and one or more communication interfaces (97) that provide an interface for one or more input / output devices (98). The input / output interfaces (96) and communication interfaces (97) are connected to a communication bus (95). The input / output devices (98) may be connected to other components of the computing device (91) through the input / output interfaces (96). An exemplary input / output device (98) may include an input device such as a pointing device (such as a mouse or trackpad), a keyboard, a touch input device (such as a touchpad or touchscreen), a voice or sound input device, various types of sensor devices and / or imaging devices, and / or an output device such as a display device, a printer, a speaker and / or a network card. An exemplary input / output device (98) may be included inside the computing device (91) as a component constituting the computing device (91), or it may be connected to the computing device (91) as a separate device distinct from the computing device (91).

[0078] Although the present invention has been described in detail above through representative embodiments, those skilled in the art will understand that various modifications and equivalent alternative embodiments are possible therefrom. Accordingly, the true technical scope of protection of the present invention should be determined by the technical spirit of the appended claims. Explanation of the symbols

[0079] 10: Image acquisition module 20: Rare optical flow acquisition module 30: Region of Interest Acquisition Module 31: Corner Point Detection Module 32: Moving Window Save Module 33: Region of Interest Setting Module 34: Region of Interest Partitioning Module 40: Preprocessing Module 50: Object Identification Module 60: Object Tracking Module

Claims

Claim 1 A moving object tracking device comprising: one or more processors; and a memory for storing one or more programs executed by said one or more processors, wherein the processor detects a plurality of corner points in a sparse optical flow obtained from a plurality of frames of an image, identifies a corner point for a moving object according to the movement distance of each of the detected plurality of corner points to set a region of interest, divides the region of interest according to the aspect ratio of the set region of interest and a reference ratio according to the object of interest to be identified, identifies an object in each region of interest divided by neural network operation, and determines the object region. Claim 2 A moving object tracking device according to claim 1, wherein the processor compares the aspect ratio of the region of interest with a plurality of reference ratios predetermined according to the type and number of objects of interest in order to divide the region of interest, determines the interval of the number of objects of interest that includes the aspect ratio, and divides the region of interest into a plurality according to the determined number of objects of interest. Claim 3 In claim 1, the processor is a moving object tracking device that divides the region of interest in a direction determined according to the basic shape of the object of interest in order to divide the region of interest. Claim 4 A moving object tracking device according to claim 1, wherein the processor detects a plurality of corner points in a sparse optical flow obtained from a plurality of frames of an image to set the region of interest, allocates a moving window that stores the position coordinate change of each of the detected corner points in the plurality of frames, and identifies a corner point for a moving object according to the moving distance of each corner point calculated based on the position coordinates stored in the moving window allocated to each corner point to set the region of interest. Claim 5 A moving object tracking device according to claim 4, wherein the processor assigns an identifier to distinguish a detected corner point from another corner point if it is a new corner point, and allocates a moving window having a memory space of a specified size to store position coordinates in multiple frames of each corner point distinguished according to the identifier. Claim 6 A moving object tracking device according to claim 4, wherein the processor, in order to identify the corner point, assigns an identifier of the existing corner point if the detected corner point is a corner point at a position corresponding to the existing corner point according to the sparse optical flow, and stores the position coordinates of the detected corner point in a moving window assigned to the existing corner point. Claim 7 A moving object tracking device according to claim 4, wherein the processor, in order to identify the corner point, assigns an identifier of the existing corner point if the position coordinates of the detected corner point are within a reference distance from the position coordinates of the existing corner point detected within a specified time interval, and stores the position coordinates of the detected corner point in a moving window assigned to the existing corner point. Claim 8 A moving object tracking device according to claim 1, wherein the processor obtains input data by preprocessing the size and shape of a segmented region of interest to determine the object region so that the size and shape are predetermined according to the shape of the object of interest, and performs neural network operations on the input data to identify an object and determine the object region. Claim 9 In claim 1, the processor is a moving object tracking device that tracks the movement path of an object by verifying the object area of ​​an object identified identically in multiple frames among the identified objects to determine the object area. Claim 10 A method for tracking a moving object, performed by a computing device having one or more processors and a memory for storing one or more programs executed by said one or more processors, comprising the steps of: detecting a plurality of corner points in a sparse optical flow obtained from a plurality of frames of an image, and identifying a corner point for a moving object according to the displacement distance of each of the detected plurality of corner points to set a region of interest; dividing the region of interest according to the aspect ratio of the set region of interest and a reference ratio according to the object of interest to be identified; and identifying an object in each region of interest divided by neural network operation and determining the object region. Claim 11 In claim 10, the step of dividing the area of ​​interest comprises comparing the aspect ratio of the area of ​​interest with a plurality of reference ratios predetermined according to the type and number of objects of interest, determining the interval of the number of objects of interest that includes the aspect ratio, and dividing the area of ​​interest into multiple parts according to the determined number of objects of interest. Claim 12 In claim 10, the step of dividing the region of interest is a moving object tracking method that divides the region of interest in a direction determined according to the basic shape of the object of interest. Claim 13 A method for tracking a moving object according to claim 10, wherein the step of setting the region of interest comprises: detecting a plurality of corner points in a sparse optical flow obtained from a plurality of frames of an image; assigning a moving window that stores the position coordinate change of each of the detected corner points in the plurality of frames; and identifying a corner point for a moving object and setting the region of interest according to the distance of movement of each corner point calculated based on the position coordinates stored in the moving window assigned to each corner point. Claim 14 In claim 13, the step of identifying corner points for the moving object and setting a region of interest comprises assigning an identifier to distinguish the detected corner point from other corner points if it is a new corner point, and allocating the moving window having a memory space of a specified size to store the position coordinates of each corner point distinguished according to the identifier in multiple frames, thereby storing the position coordinates. Claim 15 In claim 13, the step of identifying a corner point for the moving object and setting a region of interest comprises, if the detected corner point is a corner point at a position corresponding to the existing corner point according to the sparse optical flow, assigning an identifier to the existing corner point and storing the position coordinates of the detected corner point in a moving window assigned to the existing corner point. Claim 16 In claim 13, the step of identifying a corner point for the moving object and setting a region of interest is a moving object tracking method in which, if the location coordinates of the detected corner point are within a reference distance from the location coordinates of an existing corner point detected within a specified time interval, an identifier of the existing corner point is assigned, and the location coordinates of the detected corner point are stored in a moving window assigned to the existing corner point. Claim 17 In claim 10, the step of determining the object region involves obtaining input data by preprocessing the size and shape of the divided interest region so that it becomes a size and shape predetermined according to the shape of the interest object, and performing neural network operations on the input data to identify the object and determine the object region, thereby forming a moving object tracking method. Claim 18 In claim 10, the step of determining the object area is a moving object tracking method that tracks the movement path of an object by confirming the object area of ​​an object identified identically in multiple frames among the identified objects.

Images