Object tracking device and method for robot manipulating moving object

The object tracking device uses a transformer-based model to predict the presence or absence of a target object within a robot's camera frame, addressing the challenge of object detection under varying conditions and preventing robot malfunctions by transmitting control signals to stop operations when the object is out of view, ensuring a safe operating environment.

WO2024242233A1PCT designated stage expired Publication Date: 2024-11-28SEOUL NATIONAL UNIVERSITY R&DB FOUNDATION

Patent Information

Application Number
PCT/KR2023/011511
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Priority Date
2023-05-25
Filing Date
2023-08-04
Publication Date
2024-11-28

AI Technical Summary

Technical Problem

Robots manipulating moving objects face challenges in accurately determining the presence or absence of target objects within their field of view, especially under varying lighting conditions and dynamic camera perspectives, leading to potential malfunctions and safety hazards.

Method used

An object tracking device and method utilizing a transformer-based object tracking model with a memory and processor system that processes camera images to predict the presence or absence of a target object within a frame image, including features extraction, template updates, and bounding box predictions to transmit control signals for robot operation.

Benefits of technology

The solution enables explicit determination of a target object's absence, preventing robot malfunctions by stopping operations when the object is out of view, thus ensuring a safe operating environment by accurately tracking objects using a camera mounted on the robot.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure KR2023011511_28112024_PF_FP_ABST
    Figure KR2023011511_28112024_PF_FP_ABST
Patent Text Reader

Abstract

Embodiments disclosed in the present specification relate to an object tracking device and method for a robot manipulating a moving object. The object tracking device according to an embodiment comprises: a memory storing data and an object tracking model; and a control unit including one or more processors and determining whether a target object deviates from a frame image by using the object tracking model.
Need to check novelty before this filing date? Find Prior Art

Description

Object tracking device and method for a robot manipulating a moving object

[0001] The embodiments disclosed herein relate to an object tracking device and method for a robot that manipulates a moving object, and more specifically, to an object tracking device and method that enable a robot to recognize the presence or absence of an object in order to safely manipulate a moving object.

[0002] This study was conducted as a result of research on the following tasks by the Ministry of Science and ICT and the National IT Industry Promotion Agency.

[0003] 1) (IITP-2022-0-00951-002) Project on "Development of agent technology that grows while recognizing uncertainty and asking questions" of the Human-Centered Artificial Intelligence Core Source Technology Development Project,

[0004] 2) (IITP-2022-0-00953-002) Project on “Development of self-directed artificial intelligence technology to solve new problems” of the Human-Centered Artificial Intelligence Core Source Technology Development Project,

[0005] 3) (IITP-2021-0-01343-003) "Artificial Intelligence Graduate School Support (Seoul National University)" project of the Information and Communication Broadcasting Innovation Talent Development Project,

[0006] 4) (IITP-2021-0-02068-003) "Artificial Intelligence Innovation Hub Research and Development" project of the Information and Communications Broadcasting Innovation Talent Development Project.

[0007] Robots are increasingly being used in real-world applications, such as manufacturing goods in factories, preparing ordered drinks, or kneading pizza dough. For robots to manipulate objects, it's essential to determine the location of the target object. Currently, ceiling cameras, which can observe both the robot arm and the object, or hand cameras mounted on the robot arm to capture first-person perspective images, are used to calculate the target object's location. The robot then moves the robot arm to that location to manipulate the object. Ceiling cameras can be installed in fixed locations to reliably determine the robot's location.

[0008] However, it is difficult to install a ceiling camera that can generate global coordinates in all workspaces, and since the location of the ceiling camera may change due to an unexpected situation, there are cases where a hand camera is mounted only on the robot's arm, and the location of the target object is identified through the image captured by the hand camera and the robot is operated. In the case of a hand camera, since the camera is constantly moving, it is difficult to reliably recognize an object due to external factors such as light. To overcome this, an object tracking technology using an artificial neural network, such as Korean Patent No. 10-1912569, can be used to recognize the object of the hand camera.

[0009] Furthermore, to safely manipulate objects, it is necessary to recognize whether the target object is within the handheld camera's field of view and to manipulate the object only if it is within the handheld camera's field of view. Therefore, the need for object tracking technology capable of explicitly determining the absence of a target object is increasing.

[0010] Meanwhile, the background technology described above is technical information that the inventor possessed for the purpose of deriving the present invention or acquired during the process of deriving the present invention, and cannot necessarily be said to be publicly known technology disclosed to the general public prior to the application for the present invention.

[0011] One embodiment disclosed herein aims to provide an object tracking device and method that can explicitly determine whether a target object is absent within an image of a camera mounted on a robot.

[0012] One embodiment disclosed herein aims to provide an object tracking device and method that transmits a control signal to stop the operation of a robot when the absence of a target object is detected.

[0013] As a technical means for achieving the above-described technical problem, according to one embodiment, an object tracking device is disclosed, including a memory for storing data and an object tracking model and one or more processors, and a control unit for determining whether a target object has deviated within a frame image using the object tracking model, wherein the frame image is a frame image of an image captured by a camera attached to a robot, and the object tracking model includes a transformer encoder for receiving original features of the frame image extracted from the frame image, an initial template of the target object, and template features extracted from a dynamic template of the target object and outputting features of the frame image, a transformer encoder and a transformer decoder for receiving features of the frame image and a target query and outputting features of a target object query, a bounding box prediction head for predicting position coordinates of the target object within the frame image, a template update prediction head for predicting whether a dynamic template of the target object needs to be updated, and an object departure prediction head for predicting whether the target object has deviated within the frame image.

[0014] According to another embodiment, a method for tracking an object is disclosed, including the steps of: obtaining features of the frame image based on original features and template features of the frame image; obtaining features of a target object query based on the features of the frame image and a target query; predicting position coordinates of a bounding box of the target object within a video image based on the features of the frame image and the features of the target object query; predicting whether a dynamic template of the target object needs to be updated based on the features of the target object query; and predicting whether the target object deviates from the frame image of the video based on the original features.

[0015] According to another embodiment, a computer program stored in a recording medium is disclosed for performing an object tracking method, which is performed by an object tracking device, and which comprises the steps of: acquiring features of the frame image based on original features and template features of the frame image; acquiring features of a target object query based on the features of the frame image and a target query; predicting position coordinates of a bounding box of the target object within a video image based on the features of the frame image and the features of the target object query; predicting whether a dynamic template of the target object needs to be updated based on the features of the target object query; and predicting whether the target object deviates from the frame image of the video based on the original features.

[0016] According to another embodiment, a computer-readable recording medium having recorded thereon a program for performing an object tracking method is disclosed, the object tracking method comprising: a step of obtaining features of the frame image based on original features and template features of the frame image; a step of obtaining features of a target object query based on the features of the frame image and a target query; a step of predicting position coordinates of a bounding box of the target object within a video image based on the features of the frame image and the features of the target object query; a step of predicting whether a dynamic template of the target object needs to be updated based on the features of the target object query; and a step of predicting whether the target object deviates from the frame image of the video based on the original features.

[0017] According to any one of the aforementioned problem solving means, it is possible to explicitly determine whether a target object has escaped from a frame image of a video captured by a camera mounted on a robot.

[0018] According to any one of the aforementioned problem solving means, when the departure of a target object is detected, a signal for stopping the operation of the robot is transmitted to stop the operation of the robot, thereby preventing malfunction of the robot and maintaining a safe operating environment by preventing accidents that may occur due to malfunction of the robot.

[0019] The effects that can be obtained from the disclosed embodiments are not limited to the effects mentioned above, and other effects that are not mentioned can be clearly understood by a person having ordinary skill in the art to which the disclosed embodiments belong from the description below.

[0020] FIG. 1 is a reference diagram for explaining an object tracking device according to one embodiment.

[0021] FIG. 2 is a block diagram illustrating a configuration of an object tracking device according to one embodiment.

[0022] FIG. 3 is a schematic diagram illustrating the configuration of an object tracking model according to one embodiment.

[0023] FIG. 4 is a schematic diagram illustrating the configuration of an object departure prediction head according to one embodiment.

[0024] FIG. 5 is a graph comparing the performance of an object tracking model according to one embodiment and an object tracking model that does not include an object departure prediction head.

[0025] Figures 6 to 8 are diagrams illustrating the performance of an object tracking model according to an embodiment.

[0026] FIG. 9 is a flowchart illustrating an object tracking method according to one embodiment.

[0027] Below, various embodiments are described in detail with reference to the attached drawings. The embodiments described below may be modified and implemented in various different forms. To more clearly explain the features of the embodiments, detailed descriptions of matters widely known to those skilled in the art to which the embodiments pertain below have been omitted. In addition, parts of the drawings that are not related to the description of the embodiments have been omitted, and similar parts have been designated with similar drawing reference numerals throughout the specification.

[0028] Throughout the specification, when a component is said to be "connected" to another component, this includes not only the "direct connection" but also the "connection with other components in between." Furthermore, when a component is said to "include" another component, this does not exclude other components, but rather implies that other components may be included, unless otherwise specifically stated.

[0029] The embodiments will be described in detail with reference to the attached drawings below.

[0030] FIG. 1 is a reference diagram for explaining a robot control system using an object tracking device according to one embodiment. Referring to FIG. 1, the robot control system according to one embodiment includes a robot (10), a camera (20), and an object tracking device (not shown).

[0031] An object tracking device (not shown) according to one embodiment can determine whether a target object is within a current frame of an image captured by a camera, and if the target object is not within the current frame of the image, can transmit a control signal to the driving unit of the robot to stop the operation of the robot (10). The object tracking device (not shown) can track a target object using a transformer-based object tracking model, and details related to this will be described in detail with reference to FIG. 2.

[0032] A camera (20) can be attached to a robot as shown in Fig. 1 to capture images and transmit the captured images to an object tracking device. Images captured by the camera (20) may be images from a first-person perspective, and if the camera (20) is attached to the arm of the robot (10), the field of view may change depending on the movement of the arm of the robot.

[0033] FIG. 1 is a case of a robot (10) placing sushi on a plate in a conveyor belt sushi restaurant. An object tracking device (not shown) receives an image captured by a camera (20) attached to the arm of the robot (10), and based on the received image, can determine whether a plate (30), which is a target object, is within the current frame of the image.

[0034] Plates, including a plate (30) that is a target object, can move on a conveyor belt. Initially, if the plate (30) is not within the current frame of the image, an object tracking device (not shown) can determine that the object has escaped and transmit a control signal to stop the robot's operation to a robot control device (not shown) that controls the robot's operation. The robot control device (not shown) may be included in the robot or may exist externally, separately from the robot.

[0035] As time passes, the plate (30) exists within the current frame of the image, and the object tracking device (not shown) determines the position of the plate (30) within the current frame of the image and transmits it to the robot control device, and the robot (10) can place sushi on the plate.

[0036] An object tracking device according to one embodiment may be included in a robot (10) or may exist separately from the robot. If it exists separately from the robot, it may transmit and receive control signals necessary for driving the robot or object location information using a network.

[0037] FIG. 2 is a block diagram illustrating a configuration of an object tracking device according to one embodiment.

[0038] Referring to FIG. 2, an object tracking device (100) according to one embodiment may include a memory (110), a control unit (120), and a communication unit (130).

[0039] The memory (110) can be installed and stored with data and programs required to perform object tracking. The memory (110) can be configured with various types of memories, and an object tracking model that enables the control unit (120) to be described later to perform an object tracking method that can explicitly detect object departure from a search area according to the process presented below can be stored as a program, and threshold values ​​used in the object tracking model and data required for learning the object tracking model can be stored.

[0040] The control unit (120) is a configuration including at least one processor, such as a CPU, a GPU, etc., and can perform the object tracking method presented below by executing a program stored in the memory (110). Specifically, the control unit (120) can determine whether an object has escaped from a frame image of an image based on a camera image received through a communication unit (130) to be described later, and control other components included in the object tracking device (100) to perform a corresponding operation. If it is determined that an object has escaped from a frame image of an image, the control unit (120) can transmit a control signal to the robot's driving unit through the communication unit (130) to stop (suspend) the robot. If an object exists in the frame image of the image, the control unit (120) can transmit the location of the object to the robot control device through the communication unit (130). The method of determining whether an object has escaped and tracking the object based on the camera image by the control unit (120) will be described in detail with reference to other drawings below. Additionally, in this specification, a frame image of a video, a frame image of a camera video, and a frame image all refer to a frame image constituting an video received from a camera.

[0041] The communication unit (130) can perform wired or wireless communication with other devices or networks. For example, the communication unit (130) can receive images captured by a camera and can operate to transmit control signals, etc. to a robot control device. To this end, the communication unit (130) can include a communication module that supports at least one of various wired or wireless communication methods, and the communication module can be implemented in the form of a chipset. The wireless communication supported by the communication unit (220) can be, for example, WiFi (Wireless Fidelity), Wi-Fi Direct, Bluetooth, UWB (Ultra Wide Band), or NFC (Near Field Communication).

[0042] According to an embodiment, the object tracking device (100) may further include an input / output unit (not shown) for receiving input from an administrator or displaying information to the administrator, such as whether an object has escaped from the current frame of the image. The input / output unit (not shown) may include various types of input devices (e.g., keyboard, touch screen, camera, etc.) for receiving input from a user, and may include output devices such as a display panel or speaker.

[0043] Below, an object tracking process according to one embodiment is described in detail by having the control unit (120) execute a program stored in the memory (110). The processes described below are performed by having the control unit (120) execute a program stored in the memory (110), unless otherwise specifically stated.

[0044] The control unit (120) can implement an object tracking model to be described later used for object tracking by executing a program stored in the memory (110), and input an image received from a camera attached to the arm of the robot, specifically a frame image of the camera image, into the object tracking model to output results such as whether the object has escaped and the position of the object, and if it is predicted that the object has escaped, a control signal to stop the robot and a robot control device that controls the operation of the robot can be transmitted, or the position coordinates of the bounding box of the target object can be not transmitted, and if it is predicted that the object has not escaped, the position coordinates of the bounding box of the target object can be transmitted.

[0045] Hereinafter, the image received from the camera attached to the robot's arm is called an image, and the frame image of the image is called a frame image.

[0046] FIG. 3 is a diagram illustrating an object tracking model used to determine an object location and whether an object has departed in an object tracking device according to one embodiment. Referring to FIG. 3, the object tracking model (300) may include a backbone (310), a transformer encoder (320), a transformer decoder (330), an object departure prediction head (340), a bounding box prediction head (350), and a template update prediction head (360). The object tracking model (300) according to one embodiment may operate as a long-term tracker that fuses and updates target object information. In addition, in one embodiment, the object tracking model may be a single object tracking model (or single object tracker) that tracks one target object.

[0047] Specifically, the object tracking model (300) receives a camera image as input, extracts features of a frame image and features of a target object query from the input image through a backbone (310), a transformer encoder (320), and a transformer decoder (330), and then predicts the position coordinates of the bounding box of the target object, whether a dynamic template needs to be updated, and whether the target object has departed (absent) through an object departure prediction head (340), a bounding box prediction head (350), and a template update prediction head (360).

[0048] The backbone (310) is composed of a convolutional network and outputs the features of the input frame image in the form of a feature map. In other words, the backbone (310) outputs the original features of the frame image of the video. and template features including the initial template and dynamic template features of the target object. can be printed.

[0049] In one embodiment, the backbone (310) receives a frame image of an image received from a camera attached to the robot, an initial template of a target object, and a dynamic template of the target object. Before inputting the frame image to the backbone (310), the control unit (120) may preprocess the frame image to be input by adding a small disturbance (perturbation). This will be described in detail in the object departure prediction head (340) described below.

[0050] Dynamic templates are used to capture the appearance of a target object over time to provide additional temporal information. Dynamic templates can be updated by capturing images of the target object within a frame image (template cropping). Dynamic templates can be updated every 10 to 200 frames, and can be merged into the existing template list.

[0051] The original features and template features of the output frame image can be input to the transformer encoder (320) by preprocessing them so that they can be input to the transformer encoder (320). The transformer encoder (320) is composed of N encoder layers, for example, six encoder layers, and each encoder layer can include a multi-head self-attention module accompanied by a feedforward network. The template features and original features are input in the form of a feature sequence, and the features of the frame image are modeled overall in both the time and space dimensions. can be printed.

[0052] The transformer decoder (330) is a feature of a frame image output from a single target query and a transformer encoder (320). By inputting the target object query, the characteristics of the target object query for specifying the location of the bounding box of the target object can output. The transformer decoder (330) is composed of M decoder layers, and for example, can be composed of 6 decoder layers, and each decoder layer can be composed of a self-attention module, an encoder-decoder attention module, and a feedforward network, and since the object tracking model (300) is a single object tracking model, the transformer decoder (330) uses one target query.

[0053] The bounding box prediction head (350) is a feature of the frame image output from the transformer encoder (320). and the characteristics of the target object query output from the transformer decoder (330). The position coordinates of the bounding box of the target object can be predicted based on the features of the frame image. Specifically, to indicate which part of the input frame image is similar to the template of the target object, and the characteristics of target object queries The similarity score can be calculated and the calculated similarity score can be input to a Fully Convolutional Network (FCN) for predicting the top-left coordinate and a Fully Convolutional Network for predicting the bottom-right coordinate. Then, by multiplying the probability values, which are the output values ​​of the two Fully Convolutional Networks, by the x, y coordinates of the search region, the top-left x, y coordinates of the bounding box of the target object in the search region and the bottom-right x, y coordinates of the bounding box of the target object can be obtained.

[0054] The template update prediction head (360) is a feature of the target query output from the transformer decoder (330). It is possible to predict a dynamic template update score that determines whether a dynamic template needs to be updated by receiving input. The template update prediction head (360) predicts a template update prediction score by a multi-layer perceptron (MLP), and if the predicted template update prediction score is greater than a threshold, it is predicted that the dynamic template needs to be updated, and the image of the target object in the corresponding frame image can be updated with the dynamic template. The dynamic template update score can be a value between 0 and 1, and the threshold can be, for example, 0.5.

[0055] The object departure prediction head (340) is the original feature of the frame image output from the backbone (310). By inputting, it predicts whether a target object is within a frame image. Specifically, an object deviation prediction score is calculated, and if the calculated score is less than a threshold value, it can be predicted that the target object has (is not) within the frame image. If it is predicted that the target object has departed from the frame image, the control unit (120) can transmit a control signal to a robot control device that controls the operation of the robot to stop the operation of the robot.

[0056] Specifically, the object departure prediction head (340) is implemented based on [Mathematical Formula 1], which classifies out-of-distribution samples.

[0057] [Mathematical Formula 1]

[0058]

[0059] In [Equation 1], class posterior probability is the joint-class domain probability and domain probability can be calculated based on the class posterior probability. In order to more accurately predict whether an object has escaped, Rather than learning only the domain probability of the input data Since they must be learned together, the object departure prediction head according to one embodiment is as shown in Fig. 4. class can have a structure that predicts each.

[0060] Fig. 4 is a block diagram illustrating the configuration of an object departure prediction head (340). Fig. 4 is implemented based on [Mathematical Formula 2] corresponding to [Mathematical Formula 1], and the object departure prediction head (340) calculates the logit score for class i as shown in [Mathematical Formula 2]. It may include a modified MLP network (Multi-Layer Perceptron network) that outputs .

[0061] [Equation 2]

[0062]

[0063] Specifically, the object departure prediction head (340) includes a linear layer (410) that receives the original features of the frame image output from the backbone (310), and [Mathematical Formula 2] corresponds to, and the h layer (420) that calculates the probability for each classification class, and [Mathematical Formula 2] corresponds to, and includes a g layer (430) that calculates the domain probability distribution of the entire learning data. The object departure prediction head (340) receives the features of the input frame image, calculates the probability for each classification class in the h layer (420), calculates the domain probability in the g layer (430), and then calculates the logit score based on the calculated values. is produced. The produced logit score can operate as an object departure prediction score, and the control unit (120) can compare the object departure prediction score with a threshold value, and if the object departure prediction score is less than the threshold value, it can predict that the target object does not exist within the frame image.

[0064] Meanwhile, the control unit (120) may perform a perturbation process that provides a small disturbance to the frame image of the video to be input to the backbone (310) in order to increase the accuracy of the object departure prediction head (340) as described above. The perturbation process may be performed using the following [Mathematical Formula 3], and the control unit (120) may perform a perturbation intensity in the test process after learning the object tracking model. And we can determine the threshold to determine whether the object has escaped. During the perturbation process, is usually or can be the maximum value.

[0065] [Equation 3]

[0066]

[0067] Referring to [Mathematical Formula 3], the perturbation process is Obtain and use this to obtain the frame image of the camera image, which is the input image of the backbone. manipulated At this time, the intensity of the perturbation is such that the object deviation prediction score of the frame image from which the target object has been deviated and the object deviation score of the frame image containing the target object have significantly different values ​​so that they can be dichotomized during the testing process. After entering various values ​​and applying them, an appropriate value can be selected.

[0068] Meanwhile, in order to select a threshold for determining whether an object has deviated, the score function must be consistent and stable, but there is a problem in that the output value of the score function is not restricted to be a value within a certain range. In addition, since the original features of the frame image input to the object deviated prediction head (340) are time series data, the object deviated prediction score must be consistent, so the control unit (120) can determine the moving average value of the object deviated prediction scores for a specific period as the final object deviated prediction score used when determining whether an object has deviated, and can determine whether an object has deviated by comparing the final object deviated prediction score with the threshold. In addition, the threshold used for object deviated prediction can be determined by reflecting the change in the final object deviated prediction score.

[0069] The control unit (120) can train the object tracking model (300) based on data collected in the environment in which the robot will be used. Specifically, the control unit (120) inputs the collected image data into the backbone (310), inputs the original features of the frame image output from the backbone (310) and the template features of the target object into the transformer encoder (320), and trains the bounding box prediction head (350) and the template update prediction head (360) using the output values ​​of the transformer encoder (320) and the transformer decoder (330), and at the same time trains the object departure prediction head (340) using the original features of the frame image. This is because the performance of the prediction heads is better when the object deviation prediction head (340), the boundary box prediction head (350), and the template update prediction head (360) are trained simultaneously than when the two-step training method first trains the bounding box prediction head (350) and then trains the template update prediction head (360) and the object deviation prediction head (340). This will be described later in this regard.

[0070] Table 1 shows the experimental results for identifying the most efficient features for object departure prediction for input images. In Table 1, EXOT (EXit-aware Object Tracker) refers to an object tracking model (300) according to an embodiment illustrated in FIG. 3, and EXOTm, EXOTm-s, EXOT-s, EXOT-e, and EXOT-tq are object tracking models having the same configuration as EXOT. However, EXOT and EXOTm are the original features of the frame image. While EXOT-s and EXOTm-s used the similarity score as the input of the object departure prediction head (340), similar to the boundary prediction head (350), EXOT-e used the output of the transformer encoder. is input to the object departure prediction head (340), and EXOT-tq is the feature of the target query. was used as input.

[0071] And EXOTm and EXOT have differences in the learning method of the object tracking model. EXOTm learns the object departure prediction head (340), the bounding box prediction head (350), and the template update prediction head (360) simultaneously when learning the object tracking model, while EXOT uses a two-step learning method, that is, after learning the bounding box prediction head (350), the object departure prediction head (340) and the template update prediction head (360) are learned. Similarly, EXOTm-s and EXOT-s are the same. EXOTm-s learns the prediction heads simultaneously when learning, while EXOT-s follows a two-step learning method.

[0072] [Table 1]

[0073]

[0074] Fig. 5 is a graph comparing the performance of an object tracking model according to one embodiment and an object tracking model that does not include an object departure prediction head. In Fig. 5, the solid line indicates whether the object tracking model predicted whether an object would depart, and the dotted line indicates whether an object actually departed (absent) from the frame image. The more similar the shapes of the two graphs are, the higher the accuracy of object departure prediction. Referring to Fig. 5, it can be seen that the object departure prediction accuracy of the object tracking model including an object departure prediction head according to one embodiment is higher.

[0075] Similarly, FIGS. 6 to 8 are diagrams illustrating the performance of an object tracking model according to one embodiment. FIG. 6 illustrates a case where a target object, a block, is located within a frame image. Referring to FIG. 6, if the target object is predicted to be within the frame image, a rectangular bounding box is displayed around the perimeter of the target object, the block.

[0076] Figure 7 is a diagram illustrating the results of an object departure prediction using an object tracking model according to one embodiment. Referring to Figure 7, it can be seen that the target object block is accurately predicted to be absent from the frame image, and thus a rectangular bounding box is not displayed.

[0077] Figure 8 is a diagram illustrating the object departure prediction results of an object tracking model that does not include an object departure prediction head. Referring to Figure 8, it can be seen that even though the target object block is not within the frame image, the target object is determined to exist within the frame image and a rectangular bounding box is displayed.

[0078] According to the above, the object tracking device according to one embodiment can explicitly determine whether a target object has deviated from a frame image of a video captured by a camera mounted on a robot, and when the departure of the target object is detected, transmit a signal to stop the operation of the robot, thereby preventing malfunction of the robot and maintaining a safe operating environment by preventing an accident that may occur due to malfunction of the robot.

[0079] The term '~ unit' used in the above embodiments means a software or hardware component such as an FPGA (field programmable gate array) or an ASIC, and the '~ unit' performs certain roles. However, the '~ unit' is not limited to software or hardware. The '~ unit' may be configured to be on an addressable storage medium or may be configured to play one or more processors. Thus, as an example, the '~ unit' includes components such as software components, object-oriented software components, class components, and task components, processes, functions, attributes, procedures, subroutines, segments of program code, drivers, firmware, microcode, circuits, data, databases, data structures, tables, arrays, and variables.

[0080] The functionality provided within the components and '~sub-components' may be combined into fewer components and '~sub-components' or separated into additional components and '~sub-components'.

[0081] Additionally, components and '~parts' may be implemented to regenerate one or more CPUs within a device or secure multimedia card.

[0082] Meanwhile, FIG. 9 is a flowchart illustrating an object tracking method according to one embodiment. The object tracking method of FIG. 9 includes steps that are processed in a time-series manner in the object tracking device (100) illustrated in FIGS. 1 to 8. Therefore, even if omitted below, the content described above regarding the object tracking device (100) illustrated in FIGS. 1 to 8 can also be applied to the object tracking method according to the embodiment illustrated in FIG. 9.

[0083] Referring to FIG. 9, the object tracking device (100) receives an image captured from a camera, acquires original features of the frame image based on the frame image of the received image, and acquires template features of the target object based on the initial template and dynamic template of the target object (S710). At this time, the object tracking device (100) may preprocess the frame image through a perturbation process using Equation 3, and then acquire original features of the preprocessed frame image.

[0084] Next, the object tracking device (100) can acquire features of the frame image based on the original features of the frame image and the template features of the target object, and can acquire features of the target object query based on the features of the acquired frame image and the target object query (S720).

[0085] Then, the object tracking device (100) can predict the position coordinates of the bounding box of the target object based on the features of the frame image and the features of the target object query, and can predict whether the dynamic template needs to be updated based on the features of the target object query (S730). The object tracking device (100) can calculate a template update prediction score using a multi-layer perceptron (MLP) based on the features of the target object query, and can predict that the dynamic template needs to be updated if the calculated template update prediction score is greater than a threshold. If it is predicted that the dynamic template needs to be updated, the object tracking device (100) can update the target object image of the corresponding frame image with the dynamic template, and the updated dynamic template can be added to the existing template list.

[0086] Meanwhile, the object tracking device (100) predicts whether the target object has deviated (absent) from the frame image based on the original features of the frame image (S740), and if it is predicted that the target object has deviated from the frame image, the object tracking device (100) can transmit a control signal to the robot control device to stop (stop) the operation of the robot (S750). The object tracking device (100) obtains an object deviated prediction score based on Equations 1 and 2, compares the predicted object deviated prediction score with a threshold value, and if the object deviated prediction score is less than the threshold value, it can predict that the target object does not exist in the corresponding frame image. Although step S740 is shown as a separate step from step S730, the two steps can be performed simultaneously.

[0087] The object tracking method according to the embodiment described through FIG. 9 may also be implemented in the form of a computer-readable medium that stores computer-executable commands and data. In this case, the commands and data may be stored in the form of program code, and when executed by a processor, may generate a predetermined program module to perform a predetermined operation. In addition, the computer-readable medium may be any available medium that can be accessed by a computer, and includes both volatile and nonvolatile media, removable and non-removable media. In addition, the computer-readable medium may be a computer recording medium, and the computer recording medium may include both volatile and nonvolatile, removable and non-removable media implemented by any method or technology for storing information such as computer-readable commands, data structures, program modules, or other data. For example, the computer recording medium may be a magnetic storage medium such as an HDD or an SSD, an optical recording medium such as a CD, a DVD, and a Blu-ray disc, or a memory included in a server that is accessible via a network.

[0088] In addition, the object tracking method according to the embodiment described through FIG. 9 may be implemented as a computer program (or computer program product) including computer-executable instructions. The computer program includes programmable machine instructions processed by the processor, and may be implemented in a high-level programming language, an object-oriented programming language, an assembly language, a machine language, etc. In addition, the computer program may be recorded on a tangible computer-readable recording medium (e.g., a memory, a hard disk, a magnetic / optical medium, or a solid-state drive (SSD), etc.).

[0089] Accordingly, the object tracking method according to the embodiment described through FIG. 9 can be implemented by executing the above-described computer program on a computing device. The computing device may include at least some of a processor, memory, a storage device, a high-speed interface connecting the memory and a high-speed expansion port, and a low-speed interface connecting the low-speed bus and the storage device. Each of these components is interconnected using various buses and may be mounted on a common motherboard or in another suitable manner.

[0090] Here, the processor can process instructions within the computing device, such as instructions stored in a memory or storage device to display graphical information for providing a graphical user interface (GUI) on an external input / output device, such as a display connected to a high-speed interface. In another embodiment, multiple processors and / or multiple buses may be utilized, as appropriate, together with multiple memories and memory types. The processor may also be implemented as a chipset comprising multiple independent analog and / or digital processors.

[0091] Memory also stores information within a computing device. For example, memory may consist of volatile memory units or a collection of volatile memory units. For another example, memory may consist of nonvolatile memory units or a collection of nonvolatile memory units. Memory may also be another form of computer-readable media, such as magnetic or optical disks.

[0092] And memory can provide a large amount of storage space for a computing device. Memory can be a computer-readable medium or a configuration including such a medium, and can include devices within a storage area network (SAN) or other configurations, such as a floppy disk drive, a hard disk drive, an optical disk drive, a tape drive, flash memory, or other similar semiconductor memory devices or arrays of devices.

[0093] The embodiments described above are provided for illustrative purposes only, and those skilled in the art will readily appreciate that the embodiments described above can be readily modified into other specific forms without altering the technical concepts or essential characteristics of the embodiments described above. Therefore, the embodiments described above should be understood as illustrative in all respects and not restrictive. For example, components described as being single may be implemented in a distributed manner, and similarly, components described as being distributed may be implemented in a combined manner.

[0094] The scope of protection sought through this specification is indicated by the claims described below rather than the detailed description above, and should be interpreted to include all changes or modifications derived from the meaning and scope of the claims and their equivalent concepts.

Claims

1. In an object tracking device for a robot that manipulates a moving object, Memory for storing data and object tracking models; and A control unit including one or more processors and using the object tracking model to determine whether a target object has escaped from a frame image, The above frame image is a frame image of a video captured from a camera attached to the robot, The above object tracking model is, A transformer encoder that receives original features of the frame image extracted from the frame image, an initial template of the target object, and template features extracted from a dynamic template of the target object as input and outputs features of the frame image; A transformer decoder that receives features of the above frame image and a target query as input and outputs features of the target object query; A bounding box prediction head that predicts the position coordinates of the bounding box of the target object within the frame image; A template update prediction head that predicts whether the dynamic template of the target object needs to be updated; and An object tracking device including an object departure prediction head that predicts whether the target object has departed from the frame image.

2. In paragraph 1, An object tracking device, wherein the above object tracking model is a single object tracking model that tracks one target object.

3. In paragraph 1, The above control unit, An object tracking device that simultaneously trains the object departure prediction head, the bounding box prediction head, and the template update prediction head.

4. In paragraph 1, An object tracking device in which the object departure prediction head calculates an object departure prediction score based on the original features of the frame image, and predicts that the target object has departed if the calculated object departure prediction score is less than a threshold value.

5. In paragraph 1, The above control unit, An object tracking device that transmits a control signal to a robot control device to stop the operation of the robot when the target object is predicted to have departed from the frame image from the object departure prediction head.

6. In a method for tracking an object performed in an object tracking device, A step of extracting original features of a frame image extracted from a frame image of a video captured from a camera attached to a robot, an initial template of a target object, and a template feature from a dynamic template of the target object; A step of obtaining features of the frame image based on original features and template features of the frame image; A step of obtaining features of a target object query based on features of the above frame image and a target query; A step of predicting the position coordinates of a bounding box of the target object within a video image based on features of the frame image and features of a target object query; A step of predicting whether a dynamic template of the target object needs to be updated based on the characteristics of the target object query; and An object tracking method, comprising a step of predicting whether the target object has deviated from the frame image of the video based on the original features.

7. In paragraph 6, An object tracking method, wherein the steps of determining whether the target object has deviated, predicting whether the dynamic template of the target object needs to be updated, and predicting the position coordinates of the bounding box of the target object are performed simultaneously.

8. In paragraph 6, An object tracking method, wherein an object deviation prediction score is calculated based on the above original features, and if the calculated object deviation prediction score is less than a threshold value, the target object is predicted to have deviated from the frame image of the video.

9. In paragraph 6, The above object tracking method is, An object tracking method further comprising a step of transmitting a control signal to a robot control device to stop the operation of the robot when, in the object departure prediction step, the target object is predicted to have departed from the frame image.

10. A computer program that is performed by an object tracking device and performs the method described in paragraph 6.

11. A computer-readable recording medium having recorded thereon a computer program for performing the method described in Article 6.

Citation Information

Patent Citations

  • The object tracking system of video images

    KR101912569B1

  • Pattern inspection method and pattern inspection apparatus

    KR1020170049457A

  • KR20210068337A

Cited By

  • Visual target tracking method and device based on efficient sequence generation and electronic equipment

    CN119785355A

  • Target tracking method and device, computer readable storage medium and electronic equipment

    CN121437560A