Apparatus and method for tracking object for robot

Abstract

The embodiments disclosed herein relate to an object tracking device and method for a robot manipulating a moving object. An object tracking device according to one embodiment includes a memory for storing data and an object tracking model and one or more processors, and includes a control unit that determines whether a target object has deviated from a frame image using the object tracking model.

Description

Technology Field

[0001] The embodiments disclosed in this specification relate to an object tracking device and method for a robot manipulating a moving object, and more specifically, to an object tracking device and method that enables the robot to recognize the presence or absence of an object in order to safely manipulate a moving object. Background Technology

[0002] Various real-life applications of robots are emerging, such as manufacturing goods in factories, preparing ordered beverages, or kneading pizza dough. It is essential for a robot to determine the location of a target object to manipulate it; currently, robots calculate the target object's position using a ceiling camera capable of observing both the robot arm and the object, or a hand camera mounted on the robot arm to obtain first-person view footage. After calculating the position, the robot arm is moved to that location to manipulate the object. In the case of a ceiling camera, it can be installed in a fixed position to reliably determine the robot's location.

[0003] However, it is difficult to install ceiling cameras capable of generating global coordinates in all workspaces, and since the position of the ceiling camera may change due to unexpected situations, there are cases where a hand camera is mounted only on the robot arm, and the robot must be operated by determining the position of the target object through the video captured by the hand camera. In the case of a hand camera, since the camera is constantly moving, it is difficult to stably recognize objects due to external factors such as light. To overcome this, objects in the hand camera can be recognized using an object tracking technology utilizing an artificial neural network, such as Korean Registered Patent 10-1912569.

[0004] In addition, to safely manipulate objects, it is necessary to recognize whether a target object is within the hand camera's view and to allow manipulation only when the target object is present. Therefore, there is an increasing need for object tracking technology capable of explicitly determining the absence of a target object.

[0005] Meanwhile, the aforementioned background technology is technical information that the inventor possessed for the derivation of the present invention or acquired during the process of deriving the present invention, and it cannot be considered as prior art disclosed to the general public prior to the filing of the present invention. The problem to be solved

[0006] One embodiment disclosed in this specification aims to provide an object tracking device and method that enables explicit determination of whether a target object is absent within a camera image mounted on a robot.

[0007] One embodiment disclosed in this specification aims to provide an object tracking device and method that transmits a control signal to stop the operation of a robot when the absence of a target object is detected. means of solving the problem

[0008] According to one embodiment, as a technical means for achieving the aforementioned technical problem, an object tracking device is disclosed that includes a memory storing data and an object tracking model and one or more processors, and a control unit that determines whether a target object deviates from a frame image using the object tracking model, wherein the frame image is a frame image of a video captured by a camera attached to a robot, and the object tracking model includes a transformer encoder that receives original features of the frame image extracted from the frame image, template features extracted from an initial template of the target object and a dynamic template of the target object, and outputs features of the frame image; a transformer encoder and a transformer decoder that receive features of the frame image and a target query and output features of the target object query; a bounding box prediction head that predicts the position coordinates of the target object within the frame image; a template update prediction head that predicts whether the dynamic template of the target object needs updating; and an object deviation prediction head that predicts whether the target object deviates from the frame image.

[0009] According to another embodiment, an object tracking method is disclosed, comprising the steps of: acquiring features of a frame image based on original features and template features of the frame image; acquiring features of a target object query based on features of the frame image and a target query; predicting position coordinates of a bounding box of the target object within an image based on features of the frame image and a target object query; predicting whether a dynamic template of the target object needs to be updated based on features of the target object query; and predicting whether the target object deviates from the frame image of the image based on original features.

[0010] According to another embodiment, a computer program stored on a recording medium for performing an object tracking method is disclosed, wherein the object tracking method comprises the steps of: acquiring features of the frame image based on original features and template features of the frame image; acquiring features of a target object query based on features of the frame image and a target query; predicting position coordinates of a bounding box of the target object within an image based on features of the frame image and a target object query; predicting whether a dynamic template of the target object needs to be updated based on features of the target object query; and predicting whether the target object deviates from the frame image of the image based on original features.

[0011] According to another embodiment, a computer-readable recording medium is disclosed having a program for performing an object tracking method, wherein the object tracking method comprises the steps of: acquiring features of the frame image based on original features and template features of the frame image; acquiring features of a target object query based on features of the frame image and a target query; predicting position coordinates of a bounding box of the target object within an image based on features of the frame image and a target object query; predicting whether a dynamic template of the target object needs to be updated based on features of the target object query; and predicting whether the target object deviates from the frame image of the image based on original features. Effects of the invention

[0012] According to any one of the aforementioned means for solving the problem, it is possible to explicitly determine whether a target object has moved out of a frame image of a video captured through a camera mounted on a robot.

[0013] According to any one of the aforementioned means for solving the problem, by transmitting a signal to stop the robot's operation when the departure of a target object is detected, the robot's operation can be stopped, thereby preventing the robot's malfunction and preventing accidents that may occur due to the robot's malfunction, thus maintaining a safe operating environment.

[0014] The effects obtainable from the disclosed embodiments are not limited to those mentioned above, and other unmentioned effects will be clearly understood by those skilled in the art to which the disclosed embodiments belong from the description below. Brief explanation of the drawing

[0015] FIG. 1 is a reference diagram for explaining an object tracking device according to one embodiment. FIG. 2 is a block diagram illustrating the configuration of an object tracking device according to one embodiment. FIG. 3 is a configuration diagram illustrating the configuration of an object tracking model according to one embodiment. FIG. 4 is a configuration diagram illustrating the configuration of an object escape prediction head according to one embodiment. FIGS. 5, FIGS. 6a to 6c are reference diagrams for illustrating the performance of an object tracking model according to one embodiment. FIG. 7 is a flowchart illustrating an object tracking method according to one embodiment. Specific details for implementing the invention

[0016] Various embodiments are described in detail below with reference to the attached drawings. The embodiments described below may be implemented in various different forms. In order to explain the features of the embodiments more clearly, detailed descriptions of matters widely known to those skilled in the art to which the following embodiments belong have been omitted. Additionally, parts of the drawings unrelated to the description of the embodiments have been omitted, and similar parts throughout the specification have been given similar reference numerals.

[0017] Throughout the specification, when a configuration is described as being "connected" to another configuration, this includes not only cases where they are "directly connected," but also cases where they are "connected with another configuration in between." Furthermore, when a configuration is described as "including" another configuration, this means that, unless specifically stated otherwise, it does not exclude other configurations but may include additional configurations.

[0018] The embodiments will be described in detail below with reference to the attached drawings.

[0019] FIG. 1 is a reference diagram for explaining a robot control system using an object tracking device according to one embodiment. Referring to FIG. 1, a robot control system according to one embodiment includes a robot (10), a camera (20), and an object tracking device (not shown).

[0020] An object tracking device (not shown) according to one embodiment can determine whether a target object is within the current frame of an image based on an image captured through a camera, and if the target object is not within the current frame of an image, transmit a control signal to the drive unit of the robot to stop the operation of the robot (10). The object tracking device (not shown) can track a target object using a transformer-based object tracking model, and details related to this will be explained in detail in FIG. 2.

[0021] The camera (20) can be attached to the robot as shown in FIG. 1 to capture images and transmit the captured images to an object tracking device. The images captured by the camera (20) may be first-person view images, and when the camera (20) is attached to the arm of the robot (10), the field of view may change as the robot's arm moves.

[0022] FIG. 1 is the case of a robot (10) that places sushi on a plate in a conveyor belt sushi restaurant, and an object tracking device (not shown) receives an image captured by a camera (20) attached to the arm of the robot (10), and based on the received image, can determine whether a target object, a plate (30), is within the current frame of the image.

[0023] Plates, including the target object plate (30), can move on a conveyor belt. Initially, if the plate (30) is not within the current frame of the image (Out-of-Distribution), the object tracking device (not shown) may determine that the object has been removed and may transmit a control signal to stop the robot's operation to a robot control device (not shown) that controls the robot's operation. The robot control device (not shown) may be included in the robot or may exist externally, separate from the robot.

[0024] Over time, the plate (30) is present within the current frame of the video, and the object tracking device (not shown) identifies the position of the plate (30) within the current frame of the video and transmits it to the robot control device, and the robot (10) can place sushi on the plate.

[0025] An object tracking device according to one embodiment may be included in the robot (10) or may exist separately from the robot, and if it exists separately from the robot, it may use a network to transmit and receive control signals or object location information necessary for driving the robot.

[0026] FIG. 2 is a block diagram illustrating the configuration of an object tracking device according to one embodiment.

[0027] Referring to FIG. 2, an object tracking device (100) according to one embodiment may include a memory (110), a control unit (120), and a communication unit (130).

[0028] The memory (110) may have data and programs necessary for performing object tracking installed and stored. The memory (110) may be configured with various types of memory, and may have an object tracking model stored as a program that enables the control unit (120), described below, to perform an object tracking method that can explicitly identify object deviations within a search area according to the process presented below, and may have threshold values ​​used in the object tracking model and data necessary for learning the object tracking model stored.

[0029] The control unit (120) is configured to include at least one processor, such as a CPU or GPU, and can perform the object tracking method described below by executing a program stored in memory (110). Specifically, the control unit (120) can determine whether an object has moved out of the frame image of the video based on the camera image received through the communication unit (130) described later, and can control other components included in the object tracking device (100) to perform a corresponding operation. If it is determined that an object has moved out of the frame image of the video, the control unit (120) can transmit a control signal to the driving unit of the robot through the communication unit (130) to stop the robot. If an object exists within the frame image of the video, the control unit (120) can transmit the position of the object to the robot control device through the communication unit (130). The method of determining whether an object has moved out of the video and tracking the object based on the camera image of the control unit (120) will be explained in detail below with reference to other drawings. Additionally, in this specification, the frame image of the video, the frame image of the camera video, and the frame image all refer to the frame image constituting the video received from the camera.

[0030] The communication unit (130) can perform wired or wireless communication with other devices or networks. For example, the communication unit (130) can receive images captured through a camera and can operate to transmit control signals, etc. to a robot control device. To this end, the communication unit (130) may include a communication module that supports at least one of various wired or wireless communication methods, and the communication module may be implemented in the form of a chipset. The wireless communication supported by the communication unit (220) may be, for example, WiFi (Wireless Fidelity), Wi-Fi Direct, Bluetooth, UWB (Ultra Wide Band), or NFC (Near Field Communication).

[0031] According to an embodiment, the object tracking device (100) may further include an input / output unit (not shown) for receiving input from an administrator or for displaying information to the administrator, such as whether an object has moved out of the current frame of the image. The input / output unit (not shown) may include various types of input devices (e.g., keyboard, touchscreen, camera, etc.) for receiving input from a user, and may include output devices such as a display panel or a speaker.

[0032] In the following, an object tracking process according to one embodiment is described in detail by the control unit (120) executing a program stored in memory (110). Unless otherwise specifically stated, the processes described below are performed by the control unit (120) executing a program stored in memory (110).

[0033] The control unit (120) can implement the object tracking model described below, which is used for object tracking, by executing a program stored in the memory (110), and can input an image received from a camera attached to the robot arm, specifically a frame image of the camera image, into the object tracking model to output results such as whether the object has deviated and the position of the object. If it is predicted that the object has deviated, it can transmit a control signal to stop the robot to a robot control device that controls the robot's operation, or not transmit the position coordinates of the boundary box of the target object, and if it is predicted that the object has not deviated, it can transmit the position coordinates of the boundary box of the target object.

[0034] Hereinafter, the image received from the camera attached to the robot arm is referred to as the image, and the frame image of the image is referred to as the frame image.

[0035] FIG. 3 is a diagram illustrating an object tracking model used to determine the position of an object and whether the object has deviated from the object in an object tracking device according to one embodiment. Referring to FIG. 3, the object tracking model (300) may include a backbone (310), a transformer encoder (320), a transformer decoder (330), an object out-of-distribution score head (OOD Score Head; 340), a bounding box prediction head (350), and a template update prediction head (Score Head; 360). The object tracking model (300) according to one embodiment may operate as a long-term tracker that fuses and updates target object information. In addition, in one embodiment, the object tracking model may be a single object tracking model (or single object tracker) that tracks a single target object.

[0036] Specifically, the object tracking model (300) receives camera images and extracts features of frame images and features of target object queries from the received images through a backbone (310), a transformer encoder (320), and a transformer decoder (330), and then predicts the position coordinates of the bounding box of the target object, whether the dynamic template needs updating, and whether the target object is out of (absent) the object out of (absence) through an object out of (absence) prediction head (340), a bounding box prediction head (350), and a template update prediction head (360).

[0037] The backbone (310) is composed of a convolutional network and outputs the features of the input frame image in the form of a feature map. In other words, the backbone (310) is the original features of the frame image of the video. and template features including the initial template and dynamic template features of the target object It can output.

[0038] A backbone (310) according to one embodiment receives a frame image (Search region), an initial template of a target object, and a dynamic template of a target object from a camera attached to a robot. Before inputting the frame image to the backbone (310), the control unit (120) may preprocess the frame image to be input by adding a small disturbance (perturbation), which will be explained in relation to the object deviation prediction head (340) described later.

[0039] A dynamic template is used to provide additional time information by capturing the appearance of a target object over time. The dynamic template can be updated by capturing an image of the target object within a frame image, and the update of the dynamic template can be performed every 10 to 200 frames and can be done by merging it into an existing template list.

[0040] The original features and template features of the output frame image can be preprocessed so that they can be input into the transformer encoder (320). The transformer encoder (320) is composed of N encoder layers, for example, it may be composed of 6 encoder layers, and each encoder layer may include a multi-head self-attention module accompanied by a feedforward network. The template features and original features are input in the form of a feature sequence, and the features of the frame image modeled overall in both time and space dimensions. It can output.

[0041] The transformer decoder (330) is a single target query and features of the frame image output from the transformer encoder (320). Characteristics of a target object query for specifying the location of the target object's bounding box, receiving as input It can output. The transformer decoder (330) is composed of M decoder layers, and for example, can be composed of 6 decoder layers, and each decoder layer can be composed of a self-attention module, an encoder-decoder attention module, and a feedforward network, and since the object tracking model (300) is a single object tracking model, the transformer decoder (330) uses one target query.

[0042] The bounding box prediction head (350) is a feature of the frame image output from the transformer encoder (320). and the characteristics of the target object query output from the transformer decoder (330) Based on this, the location coordinates of the bounding box of the target object can be predicted. Specifically, to indicate which parts of the input frame image have similarity to the template of the target object, the features of the frame image Characteristics of target object queries The similarity score can be calculated and input into a Fully Convolution Network (FCN) to predict the top-left coordinate and a Fully Convolution Network to predict the bottom-right coordinate. Then, the probability values, which are the outputs of the two Fully Convolution Networks, are multiplied by the x and y coordinates of the search area to obtain the top-left x and y coordinates of the bounding box of the target object within the search area and the bottom-right x and y coordinates of the bounding box of the target object.

[0043] The template update prediction head (360) is a feature of the target query output from the transformer decoder (330). A dynamic template update score can be predicted by receiving input to determine whether a dynamic template update is necessary. The template update prediction head (360) predicts the template update prediction score by a Multi-Layer Perceptron (MLP), and if the predicted template update prediction score is greater than a threshold, it predicts that a dynamic template update is necessary, and the image of the target object within the corresponding frame image can be updated with the dynamic template. The dynamic template update score can be a value between 0 and 1, and the threshold can be, for example, 0.5.

[0044] The object out-of-body prediction head (340) is the original feature of the frame image output from the backbone (310). It receives input and predicts whether the target object is within the frame image. Specifically, it calculates an object departure prediction score, and if the calculated score is smaller than a threshold, it can be predicted that the target object has departed (is not present) from the frame image. If it is predicted that the target object has departed from the frame image, the control unit (120) can transmit a control signal to stop the robot's operation to a robot control device that controls the robot's operation.

[0045] Specifically, the object out-of-distribution prediction head (340) is implemented based on Equation 1, which classifies out-of-distribution samples.

[0046]

[0047] In Equation 1, Class posterior probability is joint-class domain probability Domain Probability It can be calculated based on. To predict whether an object deviates more accurately, the class trailing probability Rather than learning only the domain probability of the input data Since it must be learned together, the object departure prediction head according to one embodiment is similar to FIG. 4 class It can have a structure that predicts each.

[0048] FIG. 4 is a block diagram illustrating the configuration of an object deviation prediction head (340). FIG. 4 is implemented based on Equation 2 corresponding to Equation 1, and the object deviation prediction head (340) has a logit score for class i as illustrated in Equation 2 It may include a modified MLP network (Multi-Layer Perceptron network) that outputs.

[0049]

[0050] Specifically, the object deviation prediction head (340) comprises a linear layer (410) that receives original features of a frame image output from the backbone (310), and Equation 2 Corresponding to, an h layer (420) that calculates a probability for each classification class, and of Equation 2 It includes a g layer (430) that corresponds to and calculates the domain probability distribution of the overall training data. The object out-of-bounds prediction head (340) receives features of the input frame image, calculates a probability for each classification class in the h layer (420), calculates a domain probability in the g layer (430), and then calculates a logit score based on the calculated values. is calculated. The calculated logit score The object can operate as an object out-of-bounds prediction score, and the control unit (120) can compare the object out-of-bounds prediction score with a threshold value and predict that the target object is not present in the frame image if the object out-of-bounds prediction score is smaller than the threshold value.

[0051] Meanwhile, the control unit (120) may perform a perturbation process that introduces a small disturbance to the frame image of the video input to the backbone (310) in order to increase the accuracy of the object departure prediction head (340) as described above. The perturbation process may be carried out using Equation 3 below, and the control unit (120) [uses] a perturbation intensity during the testing process after training the object tracking model and can determine a threshold for judging whether the object has deviated. During the perturbation process, is usually or It can be the maximum value of.

[0052]

[0053] Referring to Equation 3, the perturbation process Calculate and use it to obtain the frame image of the camera video, which is the input image of the backbone. manipulated It can output. In this case, the strength of the perturbation The intensity of perturbation during the test process so that the object deviation prediction score of the frame image from which the target object is detached and the object deviation score of the frame image containing the target object can have distinctly different values ​​and be bifurcated. You can input various values ​​and apply them, and then select an appropriate value.

[0054] Meanwhile, in order to select a threshold for determining whether an object has deviated, the scoring function must be consistent and stable, but there is a problem in that the output value of the scoring function is not restricted to a specific range of values ​​within a certain range. In addition, since the original features of the frame image input to the object deviation prediction head (340) are time-series data, the object deviation prediction score must be consistent. Therefore, the control unit (120) can determine the moving average value of the object deviation prediction scores over a specific period as the final object deviation prediction score used to determine whether an object has deviated, and can determine whether an object has deviated by comparing the final object deviation prediction score with the threshold. Furthermore, the threshold used for object deviation prediction can be determined by reflecting the change in the final object deviation prediction score.

[0055] The control unit (120) can train an object tracking model (300) based on data collected in the environment where the robot is to be used. Specifically, the control unit (120) inputs the collected image data into the backbone (310), inputs the original features of the frame image output from the backbone (310) and the template features of the target object into the transformer encoder (320), and trains a bounding box prediction head (350) and a template update prediction head (360) using the output values ​​of the transformer encoder (320) and the transformer decoder (330), and at the same time trains an object deviation prediction head (340) using the original features of the frame image. This is because the performance of the prediction heads is better when the object out-of-bounds prediction head (340), bounding box prediction head (350), and template update prediction head (360) are trained simultaneously, compared to a two-stage training method in which the bounding box prediction head (350) is trained first, and then the template update prediction head (360) and the object out-of-bounds prediction head (340) are trained. This will be described below.

[0056] Table 1 presents the experimental results for identifying the most efficient features for predicting object deviation in an input image. In Table 1, EXOT (EXit-aware Object Tracker) refers to an object tracking model (300) according to an embodiment illustrated in FIG. 3, and EXOTm, EXOTm-s, EXOT-s, EXOT-e, and EXOT-tq are object tracking models having the same configuration as EXOT. However, EXOT and EXOTm are the original features of the frame image While was used as the input value for the object out-of-bounds prediction head (340), EXOT-s and EXOTm-s used similarity scores as the input value for the object out-of-bounds prediction head (340), similar to the boundary prediction head (350), and EXOT-e was the output of the transformer encoder This is input into the object departure prediction head (340), and EXOT-tq is a characteristic of the target query. used as input.

[0057] In addition, EXOTm and EXOT differ in the training method of the object tracking model. EXOTm trains the object out-of-bounds prediction head (340), bounding box prediction head (350), and template update prediction head (360) simultaneously during the training of the object tracking model, whereas EXOT uses a two-stage training method, that is, after training the bounding box prediction head (350), the object out-of-bounds prediction head (340) and the template update prediction head (360) are trained. Likewise, EXOTm-s and EXOT-s are similar in that EXOTm-s trains the prediction heads simultaneously during training, while EXOT-s follows a two-stage training method.

[0058]

[0059] FIG. 5 is a graph comparing the performance of an object tracking model according to one embodiment and an object tracking model that does not include an object departure prediction head. In FIG. 5, the solid line represents the prediction of whether the object has departed by the object tracking model, and the dotted line represents whether the object has actually departed (is absent) from the frame image. The more similar the shapes of the two graphs are, the higher the accuracy of the object departure prediction. Referring to FIG. 5, it can be seen that the accuracy of the object departure prediction of the object tracking model including the object departure prediction head according to one embodiment is higher.

[0060] Likewise, FIGS. 6a to 6c are drawings illustrating the performance of an object tracking model according to one embodiment. FIG. 6a is a drawing illustrating the case where a block, which is a target object, is within a frame image. Referring to FIG. 6a, when it is predicted that the target object is within the frame image, a bounding box is displayed around the outer edge of the target object.

[0061] FIG. 6b is a diagram illustrating the result of predicting an object's departure from an object tracking model according to one embodiment. Referring to FIG. 6b, it can be seen that the beige block, which is the target object, is accurately predicted not to be within the frame image, and thus the bounding box is not displayed.

[0062] FIG. 6c is a diagram illustrating the object out-of-bounds prediction results of an object tracking model that does not include an object out-of-bounds prediction head. Referring to FIG. 6c, it can be seen that even though the block, which is the target object, is not within the frame image, it is determined that the target object exists within the frame image and a bounding box is displayed.

[0063] According to the above description, an object tracking device according to one embodiment can explicitly determine whether a target object has moved out of a frame image of a video captured through a camera mounted on a robot, and by transmitting a signal to stop the robot's operation when the robot's movement is stopped when the robot's movement is detected, it can prevent the robot from malfunctioning and maintain a safe operating environment by preventing accidents that may occur due to the robot's malfunction.

[0064] In the embodiments above, the term 'part' refers to a software or hardware component such as an FPGA (field programmable gate array) or an ASIC, and the 'part' performs certain roles. However, the meaning of 'part' is not limited to software or hardware. The 'part' may be configured to reside in an addressable storage medium or may be configured to run one or more processors. Accordingly, as an example, the 'part' includes components such as software components, object-oriented software components, class components, and task components, as well as processes, functions, attributes, procedures, subroutines, segments of program code, drivers, firmware, microcode, circuits, data, databases, data structures, tables, arrays, and variables.

[0065] The functions provided within the components and 'parts' can be combined into fewer components and 'parts' or separated from additional components and 'parts'.

[0066] In addition, the components and '~parts' may be implemented to play one or more CPUs within the device or secure multimedia card.

[0067] Meanwhile, FIG. 7 is a flowchart for explaining an object tracking method according to one embodiment. The object tracking method of FIG. 7 includes steps processed chronologically in the object tracking device (100) shown in FIG. 1 to 6. Therefore, even if the content is omitted below, the content described above regarding the object tracking device (100) shown in FIG. 1 to 6 can also be used in the object tracking method according to the embodiment shown in FIG. 7.

[0068] Referring to FIG. 7, the object tracking device (100) receives an image captured from a camera, obtains original features of the frame image based on the frame image of the received image, and obtains template features of the target object based on the initial template and dynamic template of the target object (S710). At this time, the object tracking device (100) may preprocess the frame image through a perturbation process using Equation 3 and then obtain original features of the preprocessed frame image.

[0069] Next, the object tracking device (100) can acquire features of the frame image based on the original features of the frame image and the template features of the target object, and can acquire features of the target object query based on the acquired features of the frame image and the target object query (S720).

[0070] Then, the object tracking device (100) can predict the location coordinates of the bounding box of the target object based on the features of the frame image and the features of the target object query, and can predict whether the dynamic template needs to be updated based on the features of the target object query (S730). The object tracking device (100) can calculate a template update prediction score using a Multi-Layer Perceptron (MLP) based on the features of the target object query, and if the calculated template update prediction score is greater than a threshold, it can predict that the dynamic template needs to be updated. If it is predicted that the dynamic template needs to be updated, the object tracking device (100) updates the target object image of the corresponding frame image as a dynamic template, and the updated dynamic template can be added to the existing template list.

[0071] Meanwhile, the object tracking device (100) predicts whether the target object is out of (absent) the frame image based on the original features of the frame image (S740), and if it is predicted that the target object is out of the frame image, the object tracking device (100) can transmit a control signal to the robot control device to stop (stop) the robot's operation (S750). The object tracking device (100) obtains an object out of the frame prediction score based on Equations 1 and 2, and compares the predicted object out of the frame prediction score with a threshold value; if the object out of the frame prediction score is smaller than the threshold value, it can predict that the target object is not present in the frame image. Although step S740 is indicated as a separate step from step S730, the two steps can be performed simultaneously.

[0072] The object tracking method according to the embodiment described through FIG. 7 may also be implemented in the form of a computer-readable medium that stores instructions and data executable by a computer. In this case, the instructions and data may be stored in the form of program code, and when executed by a processor, may generate a specific program module to perform a specific operation. Furthermore, the computer-readable medium may be any available medium accessible by a computer and includes both volatile and non-volatile media, as well as removable and non-removable media. Additionally, the computer-readable medium may be a computer recording medium, which may include both volatile and non-volatile, removable and non-removable media implemented by any method or technique for storing information such as computer-readable instructions, data structures, program modules, or other data. For example, the computer recording medium may be a magnetic storage medium such as an HDD and an SSD, an optical recording medium such as a CD, DVD, and Blu-ray disc, or a memory included in a server accessible via a network.

[0073] Additionally, the object tracking method according to the embodiment described through FIG. 7 may be implemented as a computer program (or computer program product) comprising instructions executable by a computer. The computer program includes programmable machine instructions processed by a processor and may be implemented in a high-level programming language, an object-oriented programming language, assembly language, or machine language, etc. Additionally, the computer program may be recorded on a tangible computer-readable recording medium (e.g., memory, hard disk, magnetic / optical medium, or SSD (Solid-State Drive), etc.).

[0074] Accordingly, the object tracking method according to the embodiment described through FIG. 7 can be implemented by executing a computer program as described above by a computing device. The computing device may include at least some of a processor, memory, a storage device, a high-speed interface connected to the memory and a high-speed expansion port, and a low-speed interface connected to a low-speed bus and a storage device. Each of these components is connected to one another using various buses and may be mounted on a common motherboard or mounted in other suitable ways.

[0075] Here, the processor can process instructions within the computing device, such as instructions stored in memory or storage devices to display graph information for providing a Graphic User Interface (GUI) on external input and output devices, such as a display connected to a high-speed interface. In another embodiment, a plurality of processors and / or a plurality of buses may be used together with a plurality of memories and memory types as appropriate. Additionally, the processor may be implemented as a chipset comprising a plurality of independent analog and / or digital processors.

[0076] In addition, memory stores information within a computing device. For example, memory may consist of volatile memory units or a set thereof. As another example, memory may consist of non-volatile memory units or a set thereof. Furthermore, memory may be other forms of computer-readable media, such as magnetic or optical discs.

[0077] And memory can provide a large amount of storage space to a computing device. Memory may be a computer-readable medium or a configuration containing such a medium, and may include, for example, devices or other configurations within a Storage Area Network (SAN), and may be a floppy disk device, a hard disk device, an optical disk device, or a tape device, flash memory, or other similar semiconductor memory device or device array.

[0078] The embodiments described above are for illustrative purposes only, and those skilled in the art will understand that the embodiments described above can be easily modified into other specific forms without altering the technical concept or essential features of the embodiments described above. Therefore, the embodiments described above should be understood as illustrative in all respects and not restrictive. For example, each component described as a single unit may be implemented in a distributed manner, and components described as distributed may likewise be implemented in a combined form.

[0079] The scope of protection sought through this specification is defined by the claims set forth below rather than by the detailed description above, and should be interpreted to include all modifications or variations derived from the meaning and scope of the claims and the concept of equivalents. Explanation of the symbols

[0080] 100: Object tracking device 110: Memory 120: Control unit 130: Communication unit

Claims

Claim 1 An object tracking device for a robot manipulating a moving object comprises: a memory for storing data and an object tracking model; and one or more processors, and a control unit for determining whether a target object deviates from a frame image using the object tracking model, wherein the frame image is a frame image of a video captured by a camera attached to the robot, and the object tracking model comprises: a transformer encoder that receives inputs of original features of the frame image extracted from the frame image, an initial template of the target object, and template features extracted from a dynamic template of the target object, and outputs features of the frame image; a transformer decoder that receives inputs of features of the frame image and a target query, and outputs features of the target object query; a bounding box prediction head that predicts the position coordinates of a bounding box of the target object within the frame image; and a template update prediction head that predicts whether an update of the dynamic template of the target object is necessary. An object tracking device comprising an object deviation prediction head that predicts whether the target object deviates from the frame image, wherein the object deviation prediction head receives the original feature as input, calculates the domain probability and the probability by classification class of the entire data, calculates the probability by classification class for the domain probability as a logit score that operates as an object deviation score, and compares the logit score with a threshold value, and if the logit score is smaller than the threshold value, predicts that the target object does not exist within the frame image. Claim 2 An object tracking device according to claim 1, wherein the object tracking model is a single object tracking model that tracks a single target object. Claim 3 In claim 1, the control unit simultaneously trains the object deviation prediction head, the bounding box prediction head, and the template update prediction head, an object tracking device. Claim 4 delete Claim 5 An object tracking device according to claim 1, wherein the control unit transmits a control signal to a robot control device to stop the operation of the robot when it is predicted from the object deviation prediction head that a target object has deviated from the frame image. Claim 6 An object tracking method performed in an object tracking device comprises: a step of extracting template features from an original feature of a frame image extracted from a frame image of a video captured by a camera attached to a robot, an initial template of a target object, and a dynamic template of said target object; a step of acquiring features of said frame image based on the original feature of said frame image and template features; a step of acquiring features of a target object query based on the features of said frame image and a target query; a step of predicting the position coordinates of a bounding box of said target object within a video image based on the features of said frame image and the features of the target object query; a step of predicting whether the dynamic template of said target object needs to be updated based on the features of said target object query; and a step of predicting whether said target object deviates from the frame image of said video based on the original feature, wherein the step of predicting whether said target object deviates comprises: a step of receiving said original feature and calculating a domain probability and a probability by classification class of the entire data; and a step of calculating the probability by classification class for said domain probability into a logit score that operates as an object deviation score. An object tracking method comprising the step of comparing the logit score with a threshold value and predicting that the target object does not exist within the frame image if the logit score is smaller than the threshold value. Claim 7 A method for tracking an object according to claim 6, wherein the step of predicting whether the target object deviates, the step of predicting whether the dynamic template of the target object needs to be updated, and the step of predicting the position coordinates of the bounding box of the target object are performed simultaneously. Claim 8 delete Claim 9 In claim 6, the object tracking method further comprises the step of transmitting a control signal to a robot control device to stop the operation of the robot when, in the object deviation prediction step, it is predicted that the target object has deviated from the frame image. Claim 10 A computer program stored on a computer-readable recording medium that is performed by a computing device and performs the method described in paragraph 6. Claim 11 A computer-readable recording medium on which a computer program for performing the method described in paragraph 6 is recorded.

Images