Robot video target bounding box tracking labeling method, device, equipment and medium

Abstract

The application relates to a robot video target bounding box tracking labeling method, device and equipment and medium, and is applied to the field of robot vision technology.The method comprises the following steps: for each video segment in robot video data, taking the first appearing target object video frame as a current video frame, obtaining mask prompt information of the target object in the current video frame, and determining the mask of the target object; the mask of the target object in the current video frame is post-processed to generate a corresponding bounding box; based on the mask of the target object in the current video frame, the mask of the target object in the next video frame is predicted and abnormality detection is performed; if there is no abnormality, the next video frame is taken as the current video frame, and the mask of the target object in the next video frame is predicted based on the mask of the target object in the current video frame; if there is abnormality, the next video frame is taken as the current video frame, and the mask prompt information of the target object in the current video frame is obtained. The accuracy of tracking labeling can be improved.

Description

Technical Field

[0001] This application relates to the field of robot vision technology, and in particular to robot video target bounding box tracking and annotation methods, devices, equipment and media. Background Technology

[0002] In the fields of robot vision, autonomous driving, and general artificial intelligence, high-quality video object annotation data is the core foundation for training vision-language-action (VLA) models and robot perception algorithms.

[0003] Currently, the annotation of bounding boxes for specific targets in long video sequences mainly relies on manual annotation or general target tracking algorithms. However, in long videos, targets are highly susceptible to occlusion, frame drops, or complex nonlinear motion, leading to tracking drift or target loss, resulting in a low annotation success rate. Summary of the Invention

[0004] To address the aforementioned technical problems, this application provides a robot video target bounding box tracking and annotation method, apparatus, electronic device, storage medium, and computer program product.

[0005] According to a first aspect of this application, a robot video target bounding box tracking and annotation method is provided, comprising: Obtain at least one video segment obtained by dividing robot video data and the task information of the robot video data; For each video segment, the first video frame in the video segment where the target object appears is taken as the current video frame, and the mask hint information of the target object in the current video frame is obtained; the target object is an object related to the task information. The mask hint information is input into the segmentation and tracking model to obtain the mask of the target object in the current video frame; Post-process the mask of the target object in the current video frame to generate the corresponding bounding box; If there is a next video frame in the video segment, predict the mask of the target object in the next video frame based on the mask of the target object in the current video frame, and perform anomaly detection on the mask of the target object in the next video frame. If the mask of the target object in the next video frame is not abnormal, the next video frame is taken as the current video frame, and the process of post-processing the mask of the target object in the current video frame to generate the corresponding bounding box is returned. If the mask of the target object in the next video frame is abnormal, the next video frame is taken as the current video frame, and the step of obtaining the mask prompt information of the target object in the current video frame is returned. If there is no next video frame in the video segment, the process ends.

[0006] Optionally, the anomaly detection of the mask of the target object in the next video frame includes: Calculate the overlap rate between the mask area of ​​the target object in the next video frame and the mask area of ​​the target object in the current video frame. If the overlap rate is greater than or equal to the preset overlap rate, then it is determined that the mask of the target object in the next video frame is not abnormal. If the overlap rate is less than the preset overlap rate, it is determined that the mask of the target object in the next video frame is abnormal.

[0007] Optionally, the anomaly detection of the mask of the target object in the next video frame includes: Obtain the confidence level of the mask of the target object in the next video frame; If the confidence level is greater than or equal to the preset confidence level, then it is determined that the mask of the target object in the next video frame is not abnormal; If the confidence level is less than the preset confidence level, it is determined that the mask of the target object in the next video frame is abnormal.

[0008] Optionally, the anomaly detection of the mask of the target object in the next video frame includes: The velocity of the target object is calculated based on the mask of the target object in the next video frame and the mask of the target object in the current video frame. The mask of the target object in the next video frame is determined based on the speed of the target object.

[0009] Optionally, before acquiring the at least one video segment, the method further includes: Feature vectors are extracted from each video frame of the robot video data, and video frames with feature mutations are determined based on the similarity between the feature vectors. The robot video data is divided into at least one video segment based on the video frames with the aforementioned feature mutations.

[0010] Optionally, obtaining the mask hint information of the target object in the current video frame includes: If the current video frame is not the first video frame in which the target object appears, obtain the mask of the target object in the previous video frame and use the mask of the target object in the previous video frame as the mask prompt information of the target object in the current video frame.

[0011] Optionally, the segmentation tracking model includes the SAM model (Segment Anything Model).

[0012] According to a second aspect of this application, a robot video target bounding box tracking and annotation device is provided, comprising: The video-related data acquisition module is used to acquire at least one video segment obtained by dividing the robot video data and the task information of the robot video data; The current video frame determination module is used to determine the first video frame in each video segment where the target object appears as the current video frame; the target object is an object related to the task information. The mask hint information acquisition module is used to acquire the mask hint information of the target object in the current video frame; The current mask determination module is used to input the mask prompt information into the segmentation and tracking model to obtain the mask of the target object in the current video frame; The target object annotation module is used to post-process the mask of the target object in the current video frame to generate the corresponding bounding box. The next mask prediction module is used to predict the mask of the target object in the next video frame based on the mask of the target object in the current video frame if there is a next video frame in the video segment. The anomaly detection module is used to detect anomalies in the mask of the target object in the next video frame; The current video frame update module is used to, if there is no abnormality in the mask of the target object in the next video frame, take the next video frame as the current video frame and return to the target object annotation module; if there is an abnormality in the mask of the target object in the next video frame, take the next video frame as the current video frame and return to the mask prompt information acquisition module to obtain the mask prompt information of the target object in the current video frame. The process termination module is used to terminate the process if there is no next video frame in the video segment.

[0013] Optionally, the anomaly detection module is specifically used to calculate the overlap rate between the mask area of ​​the target object in the next video frame and the mask area of ​​the target object in the current video frame; if the overlap rate is greater than or equal to a preset overlap rate, it is determined that the mask of the target object in the next video frame is not abnormal; if the overlap rate is less than the preset overlap rate, it is determined that the mask of the target object in the next video frame is abnormal.

[0014] Optionally, the anomaly detection module is specifically used to obtain the confidence level of the mask of the target object in the next video frame; if the confidence level is greater than or equal to a preset confidence level, it is determined that the mask of the target object in the next video frame does not have an anomaly; if the confidence level is less than the preset confidence level, it is determined that the mask of the target object in the next video frame has an anomaly.

[0015] Optionally, the anomaly detection module is specifically used to calculate the velocity of the target object based on the mask of the target object in the next video frame and the mask of the target object in the current video frame; and to determine whether there is an anomaly in the mask of the target object in the next video frame based on the velocity of the target object.

[0016] Optionally, the robot video target bounding box tracking and annotation device further includes: The video segmentation module is used to extract feature vectors from each video frame of the robot video data, and determine the feature mutation video frames based on the similarity between the feature vectors; and divide the robot video data into at least one video segment based on the feature mutation video frames.

[0017] Optionally, the mask hint information acquisition module is specifically used to acquire the mask hint information of the target object in the current video frame in the following manner: If the current video frame is not the first video frame in which the target object appears, obtain the mask of the target object in the previous video frame and use the mask of the target object in the previous video frame as the mask prompt information of the target object in the current video frame.

[0018] Optionally, the segmentation tracking model includes the SAM model.

[0019] According to a third aspect of this application, an electronic device is provided, comprising: a processor configured to execute a computer program stored in a memory, wherein the computer program, when executed by the processor, implements the method described in the first aspect.

[0020] According to a fourth aspect of this application, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the method described in the first aspect.

[0021] According to a fifth aspect of this application, a computer program product is provided that, when the computer program product is run on a computer, causes the computer to perform the method described in the first aspect.

[0022] The technical solution provided in this application has the following advantages compared with the prior art: Dividing long robot video data into multiple manageable short video segments and performing target tracking on each segment separately avoids the problem of tracking failure in long videos. The first video frame in a segment containing the target object is used as the current video frame. The mask cue information of the target object in the current video frame is obtained, and the segmentation tracking model can determine the target object's mask based on this cue information. Then, utilizing the pixel-level segmentation tracking capability of the segmentation tracking model, the mask of the target object in the next video frame is predicted. If the mask of the target object in the next video frame is normal, it is used as the current video frame, and the mask of the target object in the next video frame is directly predicted. If the mask of the target object in the next video frame is abnormal, it is used as the current video frame, and the mask cue information of the target object in the current video frame is re-obtained, and the mask of the target object is determined based on this cue information. This avoids the problem of low mask prediction accuracy due to potential errors in inter-frame mask prediction, thereby improving the accuracy of target tracking. Using pixel-level segmentation to derive the bounding box effectively addresses target deformation and local occlusion, improving the accuracy of bounding box tracking annotation. This process requires no manual intervention, improving the efficiency of target bounding box tracking and annotation. Attached Figure Description

[0023] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0024] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0025] Figure 1 This is a flowchart of a robot video target bounding box tracking and annotation method in this application embodiment; Figure 2 This is a schematic diagram of a robot video target bounding box tracking and annotation device in an embodiment of this application; Figure 3 This is a schematic diagram of the structure of an electronic device in an embodiment of this application. Detailed Implementation

[0026] To better understand the above-mentioned objectives, features, and advantages of this application, the solution of this application will be further described below. It should be noted that, unless otherwise specified, the embodiments and features described in these embodiments can be combined with each other.

[0027] Many specific details are set forth in the following description in order to provide a full understanding of this application, but this application may also be implemented in other ways different from those described herein; obviously, the embodiments in the specification are only some embodiments of this application, and not all embodiments.

[0028] See Figure 1 , Figure 1 This is a flowchart of a robot video target bounding box tracking and annotation method according to an embodiment of this application, including the following steps: S102, Obtain at least one video segment obtained by dividing the robot video data and task information of the robot video data.

[0029] Robot video data refers to video data of a robot completing a task. Different robot video data correspond to different task information. This application embodiment tracks and annotates task-related objects in the robot video data. Typically, robot video data is long video data, and directly performing target bounding box tracking and annotation on long video data results in low accuracy. Therefore, the robot video data can be divided into multiple short video segments, and target bounding box tracking and annotation can be performed on each video segment separately to improve the accuracy of target bounding box tracking and annotation.

[0030] There are several ways to segment robot video data. For example, it can be coarsely segmented manually; it can be evenly divided at fixed time intervals, such as every 5 minutes as a video segment; or it can be segmented according to task semantic boundaries, dividing the robot video data into multiple short video segments, each representing a complete subtask with clear boundaries. Individual target bounding box tracking and annotation can be performed on each video segment, improving the accuracy of target bounding box tracking and annotation. When one subtask is completed and the next subtask begins, the appearance, motion state, or ambient lighting of the target object in the video segment may change. This task-based segmentation method ensures that detection is restarted at the beginning of a new video segment, thus quickly adapting to these changes and avoiding semantic drift in long video tracking.

[0031] The task-based segmentation approach requires automatically and accurately identifying the boundaries of "sub-tasks" (the natural start and end points of robot actions) within a continuous long video, such as the moment the hand touches the cup. Specifically, a multimodal sub-task segmentation technique can be employed. This involves sequentially performing steps such as feature extraction, change point detection, boundary optimization, and post-processing. By analyzing changes in visual features, potential sub-task boundaries are discovered, and the long video is then segmented into several video segments according to these boundaries. Preferably, change point detection can be implemented by selecting keyframes based on changes in visual features, and using video segments between adjacent keyframes as video segments. For example, when the robot switches from a "moving arm" action to a "grasping" action, the speed and direction of the arm's movement change drastically at the moment the grasping begins; these changes in motion characteristics are detected as keyframes.

[0032] Optionally, feature vectors can be extracted from each video frame of the robot video data, and feature-mutated video frames can be identified based on the similarity between feature vectors. The robot video data is then divided into at least one video segment based on these feature-mutated video frames. That is, the feature-mutated video frames can be considered as the boundaries between various sub-tasks in the robot video data. Clustering and segmentation are performed based on these feature-mutated video frames; similar feature segments represent the same sub-task and are thus grouped into one video segment.

[0033] S104: For each video segment, the first video frame in the video segment where the target object appears is taken as the current video frame.

[0034] The target object is an object related to the task information. For example, if the task information is to put a cup on a table, then the target objects are the cup and the table. The first video frame in a video clip that contains the target object is the video frame to be processed. Therefore, the first video frame containing the target object is taken as the current video frame, and target tracking is performed based on the current video frame.

[0035] S106, Obtain the mask hint information of the target object in the current video frame.

[0036] Masking cue information refers to the information used to generate masks, including cue types such as dots, bounding boxes, text, and image examples. Masking cue information can include the bounding box and category of the target object. Lightweight detection models such as multimodal large models and YOLO can be used to quickly locate objects in video frames and generate bounding boxes, which are then used as masking cue information for the target object. The multimodal large model can be an existing image segmentation model. Alternatively, manual or interactive methods can be used, such as users clicking a point on the image of a video frame and telling the model whether it is the "foreground" (the object to be segmented) or the "background" (the region to be excluded) to guide the segmentation target; or users drawing a rectangle directly on the image of the video frame based on the location of the target object to obtain the masking cue information for the target object.

[0037] S108: Input the mask hint information into the segmentation and tracking model to obtain the mask of the target object in the current video frame.

[0038] Segmentation-tracking models can perform both image segmentation and tracking, possessing pixel-level segmentation and tracking capabilities. Examples of segmentation-tracking models include the SAM model. These models can generate precise pixel-level masks of the target object based on mask cues, effectively eliminating background noise interference.

[0039] S110: Post-process the mask of the target object in the current video frame to generate the corresponding bounding box.

[0040] For the mask generated from the current video frame, its minimum bounding rectangle or axis-aligned bounding box (AABB) can be calculated. For example, the AABB can be calculated using methods such as NumPy (np.where) or OpenCV (cv2.boundingRect). Through connected component analysis of the mask, broken edges caused by occlusion are automatically filtered out, ensuring that the output bounding box always tightly wraps around the target object.

[0041] S112, determine whether there is a next video frame in the video segment.

[0042] If the video segment contains a next video frame, execute S114; otherwise, the process ends.

[0043] S114, Based on the mask of the target object in the current video frame, predict the mask of the target object in the next video frame.

[0044] The segmentation tracking model can also propagate the mask of the target object in the current video frame to the next video frame. That is, it uses the mask of the target object in the current video frame as input and uses temporal correlation algorithms (such as weight aggregation or feature matching) to predict the mask of the target object in the next video frame.

[0045] Specifically, based on the mask of the current video frame (Frame_t), inter-frame mask propagation is performed to predict the mask of the next video frame (Frame_{t+1}), as follows: Encoding: The SAM model encodes the mask (Mask_t) obtained from the current video frame (Frame_t) and the image features together into a high-dimensional memory feature vector through the SAM memory encoder, and stores it in a short-term memory bank.

[0046] Query: When processing the next video frame (Frame_{t+1}), the SAM model uses its image features as "query conditions" and compares them with historical features in the memory bank through a memory attention module (specifically, a cross-attention mechanism, a concrete implementation of the time-series association algorithm). This process generates a conditional feature that integrates information from "what I am currently seeing" and "where the target should be, according to history".

[0047] Generation: Finally, the mask decoder of the SAM model combines this conditional feature with the mask (Mask_t) of the current video frame as mask hint information to predict the new mask (Mask_{t+1}) of the next video frame (Frame_{t+1}).

[0048] S116, perform anomaly detection on the mask of the target object in the next video frame.

[0049] The mask of the target object in the next video frame is predicted based on the mask of the target object in the current video frame. During inter-frame mask propagation, if the mask of the target object in any video frame is abnormal, errors will accumulate, affecting the accuracy of target object tracking for the entire video segment. Therefore, anomaly detection can be performed on the mask of the target object in the next video frame. If the mask of the target object in the next video frame is not abnormal, S118 is executed, i.e., the inter-frame mask propagation process continues; if the mask of the target object in the next video frame is abnormal, S120 is executed, i.e., the inter-frame mask propagation process is not executed to avoid error accumulation, but a reset signal is triggered, i.e., the mask hint information of the target object is reacquired, and the mask is determined based on the mask hint information, improving the accuracy of mask determination.

[0050] Optionally, the mask area of ​​the target object in the video frame can be monitored in real time, and the overlap rate between the mask area of ​​the target object in the next video frame and the mask area of ​​the target object in the current video frame can be calculated. If the overlap rate is greater than or equal to a preset overlap rate, it is determined that the mask of the target object in the next video frame is not abnormal; if the overlap rate is less than the preset overlap rate, it is determined that the mask of the target object in the next video frame is abnormal. That is, if the mask area changes abruptly between consecutive frames, indicating a possible complete occlusion or frame loss, a reset signal is triggered, and the multimodal large model prior is called again for target localization, generating new mask cue information, such as generating a new bounding box as mask cue information, to improve the accuracy of bounding box localization.

[0051] Optionally, when predicting the mask of a target object in the next video frame, the segmentation and tracking model can also output the confidence level of the target object's mask in the next video frame. A higher confidence level indicates higher accuracy of the target object's mask in the next video frame, and a lower confidence level indicates lower accuracy. Therefore, the confidence level of the target object's mask in the next video frame can be obtained; if the confidence level is greater than or equal to a preset confidence level, it is determined that the target object's mask in the next video frame is not abnormal; if the confidence level is less than the preset confidence level, it is determined that the target object's mask in the next video frame is abnormal. That is, when the segmentation and tracking model cannot accurately predict the target object's mask in the next video frame, a reset signal can also be triggered.

[0052] Optionally, the mask of the target object in the next video frame can be checked for anomalies based on kinematic rules. Specifically, the velocity of the target object can be calculated based on the mask of the target object in the next video frame and the mask of the target object in the current video frame. The velocity of the target object determines whether the mask of the target object in the next video frame is abnormal. For example, if the velocity or acceleration of the target object between adjacent frames exceeds a threshold, the mask of the target object in the next video frame is considered abnormal, and a reset signal can be triggered.

[0053] S118, take the next video frame as the current video frame, and return to S110.

[0054] S120, take the next video frame as the current video frame, and return to S106.

[0055] During the iterative execution of S106, lightweight detection models such as the multimodal large model and YOLO can be used to determine the mask cue information of the target object. If the current video frame is not the first video frame in which the target object appears, the mask of the target object in the previous video frame can also be obtained and used as the mask cue information of the target object in the current video frame. That is, the mask cue information of the target object in the current video frame can include the mask cue information determined by the large model and the mask of the target object in the previous video frame.

[0056] Through the above iterative process, the mask of the target object in the next video frame can be obtained one by one, thereby obtaining the bounding box of the target object in the next video frame.

[0057] The robot video target bounding box tracking and annotation method of this application divides long robot video data into multiple easily controllable short video segments, and performs target tracking separately for each video segment, which can avoid the problem of tracking failure in long videos. The first video frame in the video segment where the target object appears is taken as the current video frame. The mask cue information of the target object in the current video frame is obtained, and the segmentation tracking model can determine the mask of the target object based on the mask cue information. Then, the pixel-level segmentation tracking capability of the segmentation tracking model is used to predict the mask of the target object in the next video frame. If the mask of the target object in the next video frame is normal, the next video frame is taken as the current video frame, and the mask of the target object in the next video frame is directly predicted. If the mask of the target object in the next video frame is abnormal, the next video frame is taken as the current video frame, the mask cue information of the target object in the current video frame is re-obtained, and the mask of the target object is determined based on the mask cue information. In this way, by triggering a reset signal, the problem of low mask prediction accuracy due to possible errors in inter-frame mask prediction is avoided, thereby improving the accuracy of target tracking. By using pixel-level segmentation to derive bounding boxes, we can effectively address target deformation and local occlusion, improving the accuracy of bounding box tracking and annotation. This process requires no manual intervention, thus increasing the efficiency of target bounding box tracking and annotation.

[0058] It should be noted that although the steps of the method in this disclosure are described in a specific order in the accompanying drawings, this does not require or imply that the steps must be performed in that specific order, or that all the steps shown must be performed to achieve the desired result. Additional or alternative steps may be omitted, multiple steps may be combined into one step, and / or a step may be broken down into multiple steps.

[0059] This application also provides a robot video target bounding box tracking and annotation device, see [link to documentation]. Figure 2 The robot video target bounding box tracking and annotation device 200 includes: The video-related data acquisition module 202 is used to acquire at least one video segment obtained by dividing the robot video data and task information of the robot video data; The current video frame determination module 204 is used to determine the first video frame in each video segment where the target object appears as the current video frame. The target object is an object related to the task information. The mask prompt information acquisition module 206 is used to acquire the mask prompt information of the target object in the current video frame; The current mask determination module 208 is used to input the mask prompt information into the segmentation and tracking model to obtain the mask of the target object in the current video frame; The target object annotation module 210 is used to post-process the mask of the target object in the current video frame to generate the corresponding bounding box. The next mask prediction module 212 is used to predict the mask of the target object in the next video frame based on the mask of the target object in the current video frame if there is a next video frame in the video segment. The anomaly detection module 214 is used to perform anomaly detection on the mask of the target object in the next video frame; The current video frame update module 216 is used to take the next video frame as the current video frame and return to the target object annotation module 210 if there is no abnormality in the mask of the target object in the next video frame; and to take the next video frame as the current video frame and return to the mask prompt information acquisition module 206 if there is an abnormality in the mask of the target object in the next video frame. The process termination module 218 is used to terminate the process if there is no next video frame in the video segment.

[0060] Optionally, the anomaly detection module 214 is specifically used to calculate the overlap rate between the mask area of ​​the target object in the next video frame and the mask area of ​​the target object in the current video frame; if the overlap rate is greater than or equal to a preset overlap rate, it is determined that the mask of the target object in the next video frame is not abnormal; if the overlap rate is less than the preset overlap rate, it is determined that the mask of the target object in the next video frame is abnormal.

[0061] Optionally, the anomaly detection module 214 is specifically used to obtain the confidence level of the mask of the target object in the next video frame; if the confidence level is greater than or equal to a preset confidence level, it is determined that the mask of the target object in the next video frame does not have anomalies; if the confidence level is less than the preset confidence level, it is determined that the mask of the target object in the next video frame has anomalies.

[0062] Optionally, the anomaly detection module 214 is specifically used to calculate the velocity of the target object based on the mask of the target object in the next video frame and the mask of the target object in the current video frame; and to determine whether there is an anomaly in the mask of the target object in the next video frame based on the velocity of the target object.

[0063] Optionally, the robot video target bounding box tracking and annotation device 200 also includes: The video segmentation module is used to extract feature vectors from each video frame of the robot video data, and determine the feature mutation video frames based on the similarity between the feature vectors; and divide the robot video data into at least one video segment based on the feature mutation video frames.

[0064] Optionally, the mask prompt information acquisition module 206 is specifically used to acquire the mask of the target object in the previous video frame if the current video frame is not the first video frame in which the target object appears, and use the mask of the target object in the previous video frame as the mask prompt information of the target object in the current video frame.

[0065] Optionally, the segmentation tracking model includes the SAM model.

[0066] The specific details of each module or unit in the above-mentioned device have been described in detail in the corresponding methods, so they will not be repeated here.

[0067] It should be noted that although several modules or units for the device used to perform actions have been mentioned in the detailed description above, this division is not mandatory. In fact, according to the embodiments of this application, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided and embodied by multiple modules or units.

[0068] This application also provides an electronic device, including: a processor; and a memory for storing processor-executable instructions; wherein the processor is configured to execute the above-described robot video target bounding box tracking and annotation method.

[0069] Reference Figure 3 , Figure 3 This is a schematic diagram of the structure of an electronic device in an embodiment of this application. The specific embodiments of this application do not limit the specific implementation of the electronic device.

[0070] like Figure 3 As shown, the electronic device may include: a processor 302, a communication interface 304, a memory 306, and a communication bus 308.

[0071] The processor 302, communication interface 304, and memory 306 communicate with each other via communication bus 308.

[0072] Communication interface 304 is used to communicate with other electronic devices or servers.

[0073] The processor 302 is used to execute program 310, specifically the relevant steps in the above method embodiments.

[0074] Specifically, program 310 may include program code that includes computer operation instructions.

[0075] Processor 302 may be a central processing unit, a specific integrated circuit, or one or more integrated circuits configured to implement the embodiments of this application. The one or more processors included in the smart device may be processors of the same type, such as one or more CPUs; or they may be processors of different types, such as one or more CPUs and one or more ASICs.

[0076] Memory 306 is used to store program 310. Memory 306 may include high-speed RAM memory, and may also include non-volatile memory, such as at least one disk storage device.

[0077] Specifically, program 310 can be used to cause processor 302 to execute the steps in the above-described robot video target bounding box tracking and annotation method embodiment.

[0078] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working process of the devices and modules described above can be referred to the corresponding process descriptions in the foregoing method embodiments, and will not be repeated here.

[0079] In this embodiment of the application, a computer-readable storage medium is also provided, on which a computer program is stored, which, when executed by a processor, implements the above-described robot video target bounding box tracking and annotation method.

[0080] It should be noted that the computer-readable storage medium shown in this application can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory, read-only memory, erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this application, the computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. The program code contained on the computer-readable storage medium can be transmitted using any suitable medium, including but not limited to: wireless, wire, optical fiber, radio frequency, etc., or any suitable combination thereof.

[0081] In this embodiment of the application, a computer program product is also provided, which, when run on a computer, causes the computer to execute the above-described robot video target bounding box tracking and annotation method.

[0082] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0083] The above description is merely a specific embodiment of this application, enabling those skilled in the art to understand or implement this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments described herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A robot video target bounding box tracking and annotation method, characterized in that, include: Obtain at least one video segment obtained by dividing robot video data and the task information of the robot video data; For each video segment, the first video frame in which the target object appears in the video segment is taken as the current video frame, and the mask hint information of the target object in the current video frame is obtained; The target object is an object related to the task information; The mask hint information is input into the segmentation and tracking model to obtain the mask of the target object in the current video frame; Post-process the mask of the target object in the current video frame to generate the corresponding bounding box; If there is a next video frame in the video segment, predict the mask of the target object in the next video frame based on the mask of the target object in the current video frame, and perform anomaly detection on the mask of the target object in the next video frame. If the mask of the target object in the next video frame is not abnormal, the next video frame is taken as the current video frame, and the process of post-processing the mask of the target object in the current video frame to generate the corresponding bounding box is returned. If the mask of the target object in the next video frame is abnormal, the next video frame is taken as the current video frame, and the step of obtaining the mask prompt information of the target object in the current video frame is returned. If there is no next video frame in the video segment, the process ends.

2. The method according to claim 1, characterized in that, The anomaly detection of the mask of the target object in the next video frame includes: Calculate the overlap rate between the mask area of ​​the target object in the next video frame and the mask area of ​​the target object in the current video frame. If the overlap rate is greater than or equal to the preset overlap rate, then it is determined that the mask of the target object in the next video frame is not abnormal. If the overlap rate is less than the preset overlap rate, it is determined that the mask of the target object in the next video frame is abnormal.

3. The method according to claim 1, characterized in that, The anomaly detection of the mask of the target object in the next video frame includes: Obtain the confidence level of the mask of the target object in the next video frame; If the confidence level is greater than or equal to the preset confidence level, then it is determined that the mask of the target object in the next video frame is not abnormal; If the confidence level is less than the preset confidence level, it is determined that the mask of the target object in the next video frame is abnormal.

4. The method according to claim 1, characterized in that, The anomaly detection of the mask of the target object in the next video frame includes: The velocity of the target object is calculated based on the mask of the target object in the next video frame and the mask of the target object in the current video frame. The mask of the target object in the next video frame is determined based on the speed of the target object.

5. The method according to claim 1, characterized in that, Before acquiring the at least one video segment, the method further includes: Feature vectors are extracted from each video frame of the robot video data, and video frames with feature mutations are determined based on the similarity between the feature vectors. The robot video data is divided into at least one video segment based on the video frames with the aforementioned feature mutations.

6. The method according to claim 1, characterized in that, The step of obtaining the mask hint information of the target object in the current video frame includes: If the current video frame is not the first video frame in which the target object appears, obtain the mask of the target object in the previous video frame and use the mask of the target object in the previous video frame as the mask prompt information of the target object in the current video frame.

7. The method according to claim 1, characterized in that, The segmentation tracking model includes the Segmentation-All Model (SAM) model.

8. A robot video target bounding box tracking and annotation device, characterized in that, include: The video-related data acquisition module is used to acquire at least one video segment obtained by dividing the robot video data and the task information of the robot video data; The current video frame determination module is used to determine the first video frame in each video segment where the target object appears as the current video frame. The target object is an object related to the task information; The mask hint information acquisition module is used to acquire the mask hint information of the target object in the current video frame; The current mask determination module is used to input the mask prompt information into the segmentation and tracking model to obtain the mask of the target object in the current video frame; The target object annotation module is used to post-process the mask of the target object in the current video frame to generate the corresponding bounding box. The next mask prediction module is used to predict the mask of the target object in the next video frame based on the mask of the target object in the current video frame if there is a next video frame in the video segment. The anomaly detection module is used to detect anomalies in the mask of the target object in the next video frame; The current video frame update module is used to return the target object annotation module if there is no abnormality in the mask of the target object in the next video frame, and then use the next video frame as the current video frame. If the mask of the target object in the next video frame is abnormal, the next video frame is taken as the current video frame, and the process is returned to the mask prompt information acquisition module. The process termination module is used to terminate the process if there is no next video frame in the video segment.

9. An electronic device, characterized in that, include: A processor for executing a computer program stored in a memory, wherein the computer program, when executed by the processor, implements the method of any one of claims 1-7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method described in any one of claims 1-7.

Images