Robot, target tracking method and device thereof and program product

Abstract

The invention relates to the field of target tracking, in particular to a robot, a target tracking method and device thereof and a program product. The method comprises the following steps: acquiring a visual image for tracking a target object; determining a hand detection frame in the visual image according to a pre-trained hand detection model, and detecting and determining a target object detection frame in the visual image according to a pre-trained target object detection model; determining a first intersection-to-union ratio of the hand detection frame and the target object detection frame; and when the first intersection-to-union ratio is greater than a preset first threshold value, determining tracking information of the target object in the tracker according to the hand detection frame. Even if the target object is shielded, the tracking information can still be effectively determined, so that object missing detection can be effectively reduced, and grabbing operation can be more reliably completed.

Description

Technical Field

[0001] The present application relates to the field of target tracking, and in particular to a robot and its target tracking method, device and program product. Background Art

[0002] With the rapid development of artificial intelligence and robotics, robots and robotic arms can now perform increasingly complex actions. Grasping is one of the most fundamental robotic actions, fundamental to advanced tasks such as sorting, object delivery, and tool manipulation. Obtaining the coordinates of the target object is crucial for accurate and reliable grasping.

[0003] Current target tracking methods typically store tracking information in a tracker. During a grasping operation, the position of the target to be tracked in the current frame is typically detected using deep learning or image processing methods. The target previously in the tracker is matched with each currently detected target, and the tracker is updated based on the matching results to obtain the tracking result for the current frame. However, in grasping scenarios, the gripper often overlaps with the target, making it easy to miss objects and hindering reliable grasping operations. Summary of the Invention

[0004] In view of this, the embodiments of the present application provide a robot and its target tracking method, device, equipment and program product to solve the problem in the prior art that when grasping, the gripper usually overlaps with the target, which easily leads to missed detection of objects and affects the effectiveness of the grasping action.

[0005] A first aspect of an embodiment of the present application provides a target tracking method for a robot, the method comprising:

[0006] Acquire visual images for target object tracking;

[0007] Determine a hand detection frame in the visual image based on a pre-trained hand detection model, and determine a target object detection frame in the visual image based on a pre-trained target object detection model;

[0008] Determining a first intersection-over-union (IoU) of the hand detection frame and the target object detection frame;

[0009] When the first intersection-over-union ratio is greater than a predetermined first threshold, tracking information of the target object in the tracker is determined according to the hand detection frame.

[0010] In conjunction with the first aspect, in a first possible implementation of the first aspect, when the first intersection-over-union ratio is greater than a predetermined first threshold, determining tracking information of the target object in the tracker based on the hand detection frame includes:

[0011] When the first intersection-over-union ratio is greater than a predetermined first threshold, determining, based on hand features in the visual image, that the hand type in the hand detection frame is a human hand type, and entering a human hand occlusion mode;

[0012] In the human hand occlusion mode, the tracking information of the target object in the tracker is updated according to the hand detection frame determined by the hand detection model.

[0013] In conjunction with the first possible implementation of the first aspect, in a second possible implementation of the first aspect, after entering the hand occlusion mode, the method further includes:

[0014] When the target object detection model redetects the target object detection frame and the first intersection-over-union ratio of the target object detection frame and the hand detection frame is less than or equal to the first threshold, exit the human hand occlusion mode and update the tracking information of the target object in the tracker according to the target object detection frame.

[0015] In combination with the first aspect, in a third possible implementation of the first aspect, when the first intersection-over-union ratio is greater than a predetermined first threshold, determining tracking information of the target object in the tracker based on the hand detection frame includes:

[0016] When the first intersection-over-union ratio is greater than a predetermined first threshold, determining, based on hand features in the visual image, that the hand type in the hand detection frame is a robotic hand type, updating the tracking information of the target object in the tracker based on the target object detection frame or the hand detection frame, and entering a robotic hand occlusion mode;

[0017] In the robotic arm occlusion mode, updating of the tracking information of the target object in the tracker is stopped, and the robotic arm is continued to be controlled to grasp the target object.

[0018] In conjunction with the third possible implementation manner of the first aspect, in a fourth possible implementation manner of the first aspect, before stopping updating the tracking information of the target object in the tracker according to the type of the robotic hand and continuing to control the robotic hand to grasp the target object, the method further includes:

[0019] Acquire a first spatial frame of the target object, and acquire a second spatial frame of the hand;

[0020] detecting a second intersection-over-union ratio (IOR) of the first spatial frame and the second spatial frame;

[0021] When the second intersection-over-union ratio is greater than a preset second threshold, stopping updating the tracking information of the target object in the tracker according to the type of the robotic arm is executed, and continuing to control the robotic arm to grasp the target object.

[0022] In combination with the first aspect, in a fifth possible implementation of the first aspect, before determining the hand detection frame in the visual image based on the pre-trained hand detection model, the method further includes:

[0023] Acquire a first predetermined number of first sample images including images of the hand, and acquire a second predetermined number of second sample images including images of the target object;

[0024] extracting a hand image from the first sample image;

[0025] Performing feature fusion on the hand image and the second sample image to obtain a third sample image;

[0026] The hand detection model is trained according to the third sample image to obtain a trained hand detection model.

[0027] In combination with the fifth possible implementation manner of the first aspect, in a sixth possible implementation manner of the first aspect, performing feature fusion on the hand image and the second sample image to obtain a third sample image includes:

[0028] determining a pasting position of the hand image in the second sample image according to a position of the target object in the second sample image;

[0029] Adjusting the pasting angle of the hand image according to a predetermined angle range;

[0030] The hand image is feature-fused with the second sample image according to the image feature information of the second sample image to obtain the third sample image.

[0031] A second aspect of the embodiments of the present application provides a target tracking device for a robot, characterized in that the device includes:

[0032] An image acquisition unit, configured to acquire visual images for target object tracking;

[0033] a detection frame determining unit, configured to determine a hand detection frame in the visual image based on a pre-trained hand detection model, and to determine a target object detection frame in the visual image based on a pre-trained target object detection model;

[0034] a first IoU determining unit, configured to determine a first IoU between the hand detection frame and the target object detection frame;

[0035] A tracking information determining unit is configured to determine tracking information of a target object in a tracker according to the hand detection frame when the first intersection-over-union ratio is greater than a predetermined first threshold.

[0036] In conjunction with the second aspect, in a first possible implementation manner of the second aspect, the tracking information determining unit includes:

[0037] a human hand type determination subunit, configured to determine, when the first intersection-over-union ratio is greater than a predetermined first threshold, based on hand features in the visual image, that the hand type in the hand detection frame is a human hand type and enter a human hand occlusion mode;

[0038] The first tracking information updating subunit is configured to update the tracking information of the target object in the tracker according to the hand detection frame determined by the hand detection model in the human hand occlusion mode.

[0039] In conjunction with the first possible implementation manner of the second aspect, in a second possible implementation manner of the second aspect, the apparatus further includes:

[0040] The second tracking information updating subunit is used to exit the hand occlusion mode and update the tracking information of the target object in the tracker according to the target object detection frame when the target object detection model re-detects the target object detection frame and the first intersection-over-union ratio of the target object detection frame and the hand detection frame is less than or equal to the first threshold.

[0041] In conjunction with the second aspect, in a third possible implementation manner of the second aspect, the tracking information determining unit includes:

[0042] a robotic hand type determination subunit, configured to, when the first intersection-over-union ratio is greater than a predetermined first threshold, determine, based on hand features in the visual image, that the hand type in the hand detection frame is a robotic hand type, update the tracking information of the target object in the tracker based on the target object detection frame or the hand detection frame, and enter a robotic hand occlusion mode;

[0043] The stop updating subunit is used to stop updating the tracking information of the target object in the tracker in the occlusion mode of the robot arm, and continue to control the robot arm to grasp the target object.

[0044] In combination with the third possible implementation manner of the second aspect, in a fourth possible implementation manner of the second aspect, the apparatus further includes:

[0045] a space frame acquisition subunit, configured to acquire a first space frame of the target object and a second space frame of the hand;

[0046] a second IoU determining subunit, configured to detect a second IoU ratio between the first spatial frame and the second spatial frame;

[0047] The comparing subunit is configured to execute the stopping updating subunit when the second intersection-over-union ratio is greater than a preset second threshold.

[0048] In conjunction with the second aspect, in a fifth possible implementation of the second aspect, the apparatus further includes:

[0049] a sample image acquisition unit, configured to acquire a first predetermined number of first sample images including hand images, and acquire a second predetermined number of second sample images including target object images;

[0050] a hand image extraction unit, configured to extract a hand image from the first sample image;

[0051] a fusion unit, configured to perform feature fusion on the hand image and the second sample image to obtain a third sample image;

[0052] A training unit is used to train the hand detection model according to the third sample image to obtain a trained hand detection model.

[0053] In conjunction with the fifth possible implementation manner of the second aspect, in a sixth possible implementation manner of the second aspect, the fusion unit includes:

[0054] a position determining subunit, configured to determine a pasting position of the hand image in the second sample image according to a position of the target object in the second sample image;

[0055] An angle adjustment subunit, used for adjusting the pasting angle of the hand image according to a predetermined angle range;

[0056] The feature fusion subunit is configured to perform feature fusion on the hand image and the second sample image according to the image feature information of the second sample image to obtain the third sample image.

[0057] A third aspect of an embodiment of the present application provides a robot, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the robot implements the method described in any one of the first aspects.

[0058] A fourth aspect of the embodiments of the present application provides a computer program product, which, when executed on a computer, enables the computer to execute the method in the above-mentioned first aspect or its various implementations.

[0059] A fifth aspect of an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the method described in any one of the first aspects are implemented.

[0060] A sixth aspect of the present application provides a chip for implementing the methods described in each implementation of the first aspect. Specifically, the chip includes a processor configured to retrieve and execute a computer program from a memory, causing a device equipped with the chip to execute the methods described in the first aspect or its implementations.

[0061] The beneficial effects of the embodiments of the present application compared with the prior art are: after acquiring the visual image for target object tracking, the embodiments of the present application obtain a hand detection frame through a hand detection model, obtain a target object detection frame through a target object detection model, and based on the comparison result of the first intersection-and-union ratio of the hand detection frame and the target object detection frame with a predetermined first threshold, it is determined that the current target object is occluded when the first intersection-and-union ratio is greater than the first threshold, thereby determining the tracking information of the target object in the tracker based on the hand detection frame. Even if the target object is occluded, the tracking information can still be effectively determined, thereby effectively reducing missed object detections and facilitating more reliable completion of the grasping operation. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the embodiments or descriptions of the prior art. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0063] Figure 1 This is a schematic diagram of the implementation flow of a target tracking method for a robot provided in an embodiment of the present application;

[0064] Figure 2 This is a schematic diagram of an implementation flow of a hand detection model training method provided in an embodiment of the present application;

[0065] Figure 3 This is a schematic diagram of an implementation flow of a sample image acquisition method provided in an embodiment of the present application;

[0066] Figure 4 This is a schematic diagram of a hand grasping state provided in an embodiment of the present application;

[0067] Figure 5 This is another schematic diagram of a hand grasping state provided in an embodiment of the present application;

[0068] Figure 6 This is another schematic diagram of a hand grasping state provided in an embodiment of the present application;

[0069] Figure 7 is a schematic diagram of a target tracking device for a robot provided in an embodiment of the present application;

[0070] Figure 8 This is a schematic diagram of a robot provided in an embodiment of the present application. DETAILED DESCRIPTION

[0071] In the following description, specific details such as specific system structures and techniques are provided for purposes of illustration rather than limitation to facilitate a thorough understanding of the embodiments of the present application. However, it will be apparent to those skilled in the art that the present application may be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to avoid obscuring the description of the present application with unnecessary detail.

[0072] In order to illustrate the technical solution described in this application, specific embodiments are provided below.

[0073] Artificial intelligence and robotics are experiencing rapid development. Against this backdrop, robots and robotic arms are becoming increasingly powerful, capable of performing increasingly complex and precise movements. Grasping, as a fundamental and critical robotic action, is of paramount importance. It forms the core foundation for robots to accomplish a range of advanced and complex tasks, such as sorting objects, delivering them, and operating various tools.

[0074] In the process of a robot performing a grasping operation, accurately obtaining the coordinate information of the target object to be grasped plays a vital role. It is the key to ensuring that the grasping operation can be completed accurately, reliably and efficiently.

[0075] As for the common target tracking methods currently used, the basic model is usually to store various types of information data generated during the tracking process in the tracker. When the robot is ready to perform a grasping action, it often relies on deep learning algorithms or image processing technology to detect and determine the specific position information of the target to be tracked in the current picture frame. Subsequently, the target information previously stored in the tracker is matched one by one with each target currently detected and identified. Based on the matching results, the data in the tracker is updated accordingly to obtain accurate tracking results for the current frame, thereby guiding the robot's grasping action.

[0076] However, in actual grasping applications, the gripper is prone to overlapping and occluding the target object during the grasping process, which can easily lead to missed object detection. Once missed detection occurs, the robot cannot accurately grasp the target object's real-time position and status, which seriously affects the reliability and success rate of the grasping operation and is not conducive to the robot's stable and efficient completion of the designated grasping task.

[0077] To solve the above problems, the present invention proposes a target tracking method for a robot. Figure 1 The following is a flowchart of the implementation of this method, which is detailed as follows:

[0078] In S101 , a visual image for target object tracking is acquired.

[0079] The visual images in the embodiments of the present application may be images captured by a visual sensor (e.g., a camera) equipped with the robot, and used to observe and analyze the target object. Different camera types can produce different types of visual image content. For example, an RGB camera can be used to capture images including color information, while a depth camera can capture images including depth information. The acquired visual images can be used to perceive scene information in real time.

[0080] In S102, a hand detection frame in the visual image is determined according to a pre-trained hand detection model, and a target object detection frame in the visual image is determined according to a pre-trained target object detection model.

[0081] Among them, the hand detection box is the bounding box drawn by the hand detection model to detect the hand (human hand or robot hand) appearing in the visual image, which is used to determine the position and size of the hand in the image. The hand detection model can be trained based on a deep learning target detection algorithm (such as the YOLO series or other convolutional neural network models) to identify and frame the hand area. For example, by training on an image dataset containing various hand movements and postures, it can accurately locate the hand in real scenes.

[0082] The target object detection frame can be a bounding box defined by the target object detection model to detect the target object in the visual image. The target object detection frame can be used to determine the position, size, and other information of the target object in the image. The target object detection model can be trained based on the target detection algorithm and can identify and locate the target object based on its characteristics (such as shape, texture, color, etc.). In the embodiment of the present application, the model can be trained for different types of target objects (such as bottles, boxes, etc.) to ensure the accuracy and reliability of its detection.

[0083] In the embodiment of the present application, in order to improve the efficiency of collecting sample data and the training efficiency of the hand detection model, the training process of the hand detection model in the embodiment of the present application can be as follows: Figure 2 Shown, including:

[0084] In S201 , a first predetermined number of first sample images including hand images are acquired, and a second predetermined number of second sample images including target object images are acquired.

[0085] A first sample image containing a hand and a second sample image containing a target object may be obtained from the data set, wherein the number of the first sample images is a first predetermined number and the number of the second sample images is a second predetermined number.

[0086] Sample images can come from public datasets or user-defined image data. For example, the first sample image can include images of a gripper (including human or robotic hands) in different motions, postures, and lighting conditions, with the hand position annotated in the first sample image; the second sample image can include images of various target objects such as tools and parts, with the target object position and category annotated in the second sample image.

[0087] In S202, a hand image is extracted from the first sample image.

[0088] To facilitate generation of a scene including the target object, an image of the target hand region, i.e., a hand image, can be extracted from the first sample image. When extracting the hand image, an image segmentation method can be employed to separate the hand image from the background. For example, a semantic segmentation algorithm can be employed to extract the hand image from the first sample image to form a mask image of the hand.

[0089] In S203 , feature fusion is performed on the hand image and the second sample image to obtain a third sample image.

[0090] In order to effectively synthesize the hand image and the target object image into the required third sample image, it is necessary to perform feature fusion processing on the second sample image and the hand image, so that the style of the fused image is more diverse and the number is larger, so as to meet the training requirements of the hand detection model. The specific process of feature fusion can be as follows: Figure 3 Shown, including:

[0091] In S301 , a pasting position of the hand image in the second sample image is determined according to the position of the target object in the second sample image.

[0092] The appropriate location for pasting the hand image is determined based on the target object's position in the second sample image. The selection of the pasting location must consider the appropriate spatial relationship between the hand and the target object, ensuring that the hand is aligned with the target object. This reduces invalid samples generated when the hand image deviates too far from the target object image, while also increasing sample diversity and authenticity. For example, for a second sample image in which the target object is located at the center of the image, the hand image can be pasted to an area within a predetermined range of the center, simulating the different images that appear when the hand grasps the target object.

[0093] For example, the hand image may be pasted by randomly offsetting a predetermined distance based on the center point coordinates of the target object, such as offsetting by 20% of the image width.

[0094] In S302, the pasting angle of the hand image is adjusted according to a predetermined angle range.

[0095] During the actual grasping process, the hand may approach the target object at different angles. In order to better generate multiple posture images of the hand image when approaching the target object, the hand image can be randomly rotated according to the range of joint motion, such as within the range of -30° to +30°, to determine the pasting angle when the hand image is pasted on the second sample image.

[0096] When the hand is a robot hand, a first maximum angle range when the robot reaches out to grab the target object can be determined, and the pasting angle is randomly determined within the first maximum angle range.

[0097] When the hand is that of another person, the second angle range may be determined based on possible angles of the person at different positions, and the pasting angle may be randomly determined within the second angle range.

[0098] In S303 , the hand image is feature-fused with the second sample image according to the image feature information of the second sample image to obtain the third sample image.

[0099] When fusing a hand image with a second sample image, factors such as image brightness, contrast, and color distribution are typically considered to ensure the fused image appears natural and realistic. For example, the hand image's brightness and contrast can be adjusted to match the background of the second sample image, reducing the perceived intrusion. Alternatively, the hand image can be color-corrected to align with the color of the target object in the second sample image. Poisson fusion algorithms can also be used to eliminate edge seams.

[0100] By comprehensively adjusting the pasting position, angle and feature fusion method, more diverse, realistic and challenging training samples can be generated. Therefore, a large number of third sample images can be generated from a small number of first sample images, such as 10 or less, and a small number of second sample images, thereby effectively improving the efficiency of obtaining the third sample images, further improving the performance and robustness of the hand detection model, and enabling it to better adapt to complex scenes and hand postures in practical applications.

[0101] In S204, the hand detection model is trained according to the third sample image to obtain a trained hand detection model.

[0102] After generating a larger number of third sample images from a smaller number of first and second sample images, the hand detection model can be trained based on the third sample images. For example, the third sample images can be input into the hand detection model to obtain a hand detection frame calculated by the hand detection model. The calculated hand detection frame is then compared with the annotated hand detection frame in the third sample image to determine the difference between the two. Based on the difference, the parameters of the hand detection model are adjusted until the difference meets a predetermined requirement, such as being less than a predetermined difference threshold, or the number of optimizations reaches a predetermined number.

[0103] The hand detection model, trained on the third sample image after feature fusion, can more accurately identify and detect hand targets, reducing the need for externally labeled data. By fusing the hand image with the target object image to generate a large number of new training samples, the hand detection model can learn richer hand features during training, thereby improving detection performance.

[0104] In S103 , a first intersection-over-union (IoU) ratio between the hand detection frame and the target object detection frame is determined.

[0105] After obtaining the hand detection frame and the target object detection frame, the first intersection over union (IoU value) between the two is calculated. You can first determine the coordinate information of the two detection frames (for example, the coordinates of the upper left corner and the lower right corner, or the coordinates of the upper right corner and the lower left corner), then calculate the area of ​​the overlapping area and the area of ​​the union area, and finally use the formula to get the IoU value. For example, the coordinates of the hand detection frame are (x1, y1, x2, y2), and the coordinates of the target object detection frame are (x3, y3, x4, y4). The overlapping area of ​​the two is calculated to be S_overlap, and the total union area is S_union. Then the first intersection over union is: IoU = S_overlap / S_union.

[0106] for example Figure 4 As shown, the target object detection frame determined according to the target object and the hand detection frame determined according to the hand image do not overlap, and the first intersection-over-union ratio is 0.

[0107] In S104, when the first intersection-over-union ratio is greater than a predetermined first threshold, tracking information of the target object in the tracker is determined according to the hand detection frame.

[0108] As the hand moves toward the target object, the hand detection frame gets closer and closer to the target object detection frame. Figure 5In the detection status diagram shown, there is an intersection between the target object detection frame and the hand detection frame, and the ratio of the intersection to the union is detected for the first time, that is, the first intersection-union ratio is greater than the predetermined first threshold, indicating that there is an occlusion relationship between the hand and the target object. The hand detection frame is then used to determine the tracking information of the target object in the tracker, that is, the hand detection frame is used to replace the previous target object detection frame.

[0109] In the embodiment of the present application, the first threshold is determined based on the relative sizes of the target object and the hand. The larger the absolute value of the size difference between the target object and the hand, the smaller the first threshold, and the smaller the absolute value of the size difference, the larger the first threshold.

[0110] In the embodiment of the present application, for different types of hands, tracking information can be determined in different ways according to the hand detection frame.

[0111] After the hand detection model identifies the hand detection frame, it can further analyze the hand features to identify the hand type. For example, by training a binary classifier, it can distinguish between human and robotic hands based on the color distribution, texture features, and morphological features within the hand detection frame.

[0112] When a hand is detected as occluded, the system automatically switches to hand occlusion mode. In this mode, when the system updates tracking information, it uses the hand detection frame as the tracking information of the target object and updates it to the tracker. For example, in an automated vending machine, when a customer picks up an item with their hand, the hand and the item are occluded. At this point, the system enters hand occlusion mode, and the product tracking information in the tracker is updated based on the hand detection frame until the hand leaves the item and is no longer occluded.

[0113] Even in scenarios where a human hand blocks the target object, it can still maintain effective tracking of the object, preventing target loss due to temporary occlusion, thereby improving the robustness and practicality of the target tracking system in complex interactive environments.

[0114] When the target object becomes visible again after being obscured, the target object is re-detected and a new target object detection frame is drawn. For example, when a person's hand moves an object, the target object detection frame cannot detect the object normally due to obstruction. When the person's hand leaves and the target object re-enters the camera's field of view without obstruction, the target object detection model can detect the object again and generate a detection frame. Figure 6 As shown, when the first intersection-over-union ratio of the re-detected target object detection frame and the hand detection frame is less than or equal to the first threshold (or may be less than a third threshold, which may be different from the first threshold), the hand occlusion mode is exited and the tracking information of the target object in the tracker is updated again based on the target object detection frame.

[0115] For example, when the IoU value between the hand detection frame and the target object detection frame drops to 0.3 (lower than the first threshold 0.5), the system can determine that the occlusion is released and update the target object detection frame in the tracker to a new target object detection frame, so that the target object detection frame can accurately reflect the position of the target object.

[0116] In the embodiment of the present application, after the hand detection model identifies the hand detection frame, it further analyzes the hand features to identify the hand type within the hand detection frame as a robotic hand. For example, image features within the hand detection frame (such as metallic joints, movement trajectories, etc.) are extracted and compared with a pre-established robotic hand feature model to determine that the hand type is a robotic hand.

[0117] When the hand type is determined to be a robotic hand type, it is determined that there is a robot hand occlusion, the robot hand occlusion mode is entered, and the tracking information of the target object in the tracker is stopped from being updated. The detection frame of the robotic hand or the previous frame tracking information of the target object can be used to maintain tracking of the target object. At the same time, the system can continue to control the robot hand to complete the grasping action. For example, in the process of the robot grasping electronic components, the hand gradually approaches the component. When the hand occludes the component and causes the detection frame of the component to be lost, the system stops updating the tracking frame and continues to drive the robot hand to complete the grasping action according to the pre-planned path. When the robot hand performs the grasping task, the target object detection frame is unstable due to the movement and occlusion of the hand. The system uses the robotic hand occlusion mode to ensure the continuity and stability of the grasping action and improve the grasping success rate.

[0118] In addition, in order to further improve the accuracy and stability of target tracking and robot grasping tasks, and enable the robot to better perform tasks in complex spatial operation scenarios, the embodiment of the present application can also perform three-dimensional spatial intersection and comparison verification.

[0119] In the embodiment of the present application, the three-dimensional sensor (such as a depth camera, a lidar, etc.) equipped by the robot can be used to obtain three-dimensional data of the target object and the hand. For example, the depth camera forms a depth image by emitting and receiving infrared light after reflection on the surface of the object, thereby constructing a three-dimensional point cloud of the target object and the hand. Then, the three-dimensional point cloud is processed to extract the first spatial frame and the second spatial frame. The extraction method can use a bounding box algorithm to determine the minimum circumscribed cube of the target object and the hand as their spatial frame.

[0120] Based on the determined first and second spatial frames, the degree of overlap between them can be determined. For example, the second intersection-of-union ratio is calculated by calculating the ratio of the overlapping volume to the union volume of two three-dimensional cubes. Assuming the volume of the first spatial frame of the target object is V1, the volume of the second spatial frame of the hand is V2, the overlapping volume of the two is V_overlap, and the total union volume is V_union, then the second intersection-of-union ratio = V_overlap / V_union.

[0121] When the second intersection-over-union ratio is greater than a preset second threshold (the second threshold is determined based on the first spatial frame volume and the second spatial frame volume), it indicates that the spatial overlap between the hand and the target object is relatively high. The operation of stopping updating the target object tracking information according to the type of the robotic hand can be executed, and the robotic hand can continue to be controlled to grasp the target object.

[0122] For example, when a robot grasps a target object, the second spatial frame of the robotic hand significantly overlaps with the first spatial frame of the target object, and the second intersection-over-union ratio exceeds a threshold. At this point, the system determines that the robotic hand has grasped the target object and stops updating the tracking information of the target object. The robotic hand then continues to complete the grasping action along the predetermined path. By accurately determining the spatial relationship between the hand and the target object in three-dimensional space, the robotic hand can more accurately trigger the grasping operation, improving the robot's tracking and grasping capabilities in complex three-dimensional environments, thereby better completing grasping tasks in complex scenarios.

[0123] In a possible implementation, if the second IoU is 0, it indicates that the robot arm does not currently overlap with the target object in space, and the target object may not be the target object of the robot's current grasping task. When the target object is redetected, the tracking information of the target object can be updated. Alternatively, if the target object is the target object of the current grasping task but has not yet reached the grasping position, control is continued until the second IoU exceeds the second threshold, triggering the robot to execute a grasping action to effectively grasp the target object.

[0124] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0125] Figure 7 This is a schematic diagram of a target tracking device for a robot proposed in an embodiment of the present application, the device comprising:

[0126] An image acquisition unit 701 is used to acquire a visual image for target object tracking;

[0127] A detection frame determining unit 702 is configured to determine a hand detection frame in the visual image based on a pre-trained hand detection model, and to determine a target object detection frame in the visual image based on a pre-trained target object detection model;

[0128] A first IoU determining unit 703 is configured to determine a first IoU between the hand detection frame and the target object detection frame;

[0129] The tracking information determining unit 704 is configured to determine the tracking information of the target object in the tracker according to the hand detection frame when the first intersection-over-union ratio is greater than a predetermined first threshold.

[0130] Figure 7 The target tracking device of the robot shown in FIG. Figure 1 The target tracking method of the robot shown corresponds to this.

[0131] Figure 8 Schematic diagram of a robot provided in an embodiment of the present application. Figure 8 As shown, the robot 8 of this embodiment includes: a processor 80, a memory 81, and a computer program 82 stored in the memory 81 and executable on the processor 80, such as a robot target tracking program. When the processor 80 executes the computer program 82, the steps of the aforementioned robot target tracking method embodiments are implemented. Alternatively, when the processor 80 executes the computer program 82, the functions of the various modules / units in the aforementioned device embodiments are implemented.

[0132] For example, the computer program 82 may be divided into one or more modules / units, which are stored in the memory 81 and executed by the processor 80 to implement the present application. The one or more modules / units may be a series of computer program instruction segments capable of performing specific functions, and the instruction segments are used to describe the execution process of the computer program 82 in the robot 8.

[0133] The robot may include, but is not limited to, a processor 80 and a memory 81. Those skilled in the art will appreciate that Figure 8 This is only an example of a robot 8 and does not constitute a limitation of the robot 8. The robot may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, the robot may also include input and output devices, network access devices, buses, etc.

[0134] The processor 80 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor.

[0135] The memory 81 can be an internal storage unit of the robot 8, such as the robot 8's hard drive or memory. The memory 81 can also be an external storage device of the robot 8, such as a plug-in hard drive, a Smart Media Card (SMC), a Secure Digital (SD) card, a flash memory card, etc. equipped on the robot 8. Furthermore, the memory 81 can include both the robot 8's internal storage unit and an external storage device. The memory 81 is used to store the computer program and other programs and data required by the robot. The memory 81 can also be used to temporarily store data that has been output or is about to be output.

[0136] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated into one processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0137] In the above embodiments, the description of each embodiment has its own focus. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant description of other embodiments.

[0138] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0139] In the embodiments provided in this application, it should be understood that the disclosed devices / terminal equipment and methods can be implemented in other ways. For example, the device / terminal equipment embodiments described above are merely illustrative. For example, the division of the modules or units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0140] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0141] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0142] If the integrated module / unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the process in the above-mentioned embodiment method, and can also be completed by hardware related to computer program instructions. The computer program can be stored in a computer-readable storage medium, and when the computer program is executed by the processor, it can implement the steps of the above-mentioned various method embodiments. Wherein, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium. It should be noted that the content contained in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electric carrier signals and telecommunication signals.

[0143] In addition, an embodiment of the present application also provides a computer program product, which, when running on a computer, enables the computer to execute the methods in the above-mentioned implementation manners.

[0144] The above-described embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present application, and should all be included in the scope of protection of the present application.

Claims

1. A target tracking method for a robot, characterized in that: The method comprises: Acquire visual images for target object tracking; Determine a hand detection frame in the visual image based on a pre-trained hand detection model, and determine a target object detection frame in the visual image based on a pre-trained target object detection model; Determining a first intersection-over-union (IoU) of the hand detection frame and the target object detection frame; When the first intersection-over-union ratio is greater than a predetermined first threshold, tracking information of the target object in the tracker is determined according to the hand detection frame.

2. The method according to claim 1, characterized in that When the first intersection-over-union ratio is greater than a predetermined first threshold, determining tracking information of the target object in the tracker according to the hand detection frame includes: When the first intersection-over-union ratio is greater than a predetermined first threshold, determining, based on hand features in the visual image, that the hand type in the hand detection frame is a human hand type, and entering a human hand occlusion mode; In the human hand occlusion mode, the tracking information of the target object in the tracker is updated according to the hand detection frame determined by the hand detection model.

3. The method according to claim 2, characterized in that After entering the hand occlusion mode, the method further includes: When the target object detection model redetects the target object detection frame and the first intersection-over-union ratio of the target object detection frame and the hand detection frame is less than or equal to the first threshold, exit the human hand occlusion mode and update the tracking information of the target object in the tracker according to the target object detection frame.

4. The method according to claim 1, wherein When the first intersection-over-union ratio is greater than a predetermined first threshold, determining tracking information of the target object in the tracker according to the hand detection frame includes: When the first intersection-over-union ratio is greater than a predetermined first threshold, determining, based on hand features in the visual image, that the hand type in the hand detection frame is a robotic hand type, updating the tracking information of the target object in the tracker based on the target object detection frame or the hand detection frame, and entering a robotic hand occlusion mode; In the robotic arm occlusion mode, updating of the tracking information of the target object in the tracker is stopped, and the robotic arm is continued to be controlled to grasp the target object.

5. The method according to claim 4, characterized in that Before stopping updating the tracking information of the target object in the tracker according to the type of the robotic arm and continuing to control the robotic arm to grasp the target object, the method further includes: Acquire a first spatial frame of the target object, and acquire a second spatial frame of the hand; detecting a second intersection-over-union ratio (IOR) of the first spatial frame and the second spatial frame; When the second intersection-over-union ratio is greater than a preset second threshold, stopping updating the tracking information of the target object in the tracker according to the type of the robotic arm is executed, and continuing to control the robotic arm to grasp the target object.

6. The method according to claim 1, characterized in that Before determining a hand detection frame in the visual image according to the pre-trained hand detection model, the method further includes: Acquiring a first predetermined number of first sample images including images of the hand, and acquiring a second predetermined number of second sample images including images of the target object; extracting a hand image from the first sample image; Performing feature fusion on the hand image and the second sample image to obtain a third sample image; The hand detection model is trained according to the third sample image to obtain a trained hand detection model.

7. The method according to claim 6, characterized in that Performing feature fusion on the hand image and the second sample image to obtain a third sample image includes: determining a pasting position of the hand image in the second sample image according to a position of the target object in the second sample image; Adjusting the pasting angle of the hand image according to a predetermined angle range; The hand image is feature-fused with the second sample image according to the image feature information of the second sample image to obtain the third sample image.

8. A target tracking device for a robot, characterized in that: The device comprises: An image acquisition unit, configured to acquire visual images for target object tracking; a detection frame determining unit, configured to determine a hand detection frame in the visual image based on a pre-trained hand detection model, and to determine a target object detection frame in the visual image based on a pre-trained target object detection model; a first IoU determining unit, configured to determine a first IoU between the hand detection frame and the target object detection frame; A tracking information determining unit is configured to determine tracking information of a target object in a tracker according to the hand detection frame when the first intersection-over-union ratio is greater than a predetermined first threshold.

9. A robot comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the robot is caused to implement the method according to any one of claims 1 to 7.

10. A computer program product comprising computer program instructions, characterized in that When the computer program is executed, the method according to any one of claims 1 to 7 is performed.

Images