Tracking method and device of target object and computer equipment

Abstract

The invention relates to the technical field of computers, and discloses a target object tracking method and device and computer equipment, and the target object tracking method comprises the steps: firstly obtaining an initial frame image and a plurality of frames of to-be-detected images; secondly, determining an initial visual feature of the target object based on the initial frame image, and storing the initial visual feature in a long-term memory sequence; if the target object is detected in any one to-be-detected image, storing the intermediate visual features of the target object detected based on any one to-be-detected image into the short-term memory sequence; and finally, if the short-term memory sequence reaches the preset storage state, detecting the latest to-be-detected image, and updating the long-term memory sequence or the short-term memory sequence according to a detection result of the latest to-be-detected image so as to track the target object according to the long-term memory sequence and the short-term memory sequence. Through the synergistic effect of the long-term memory sequence and the short-term memory sequence, the adaptability of target object tracking can be effectively improved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a method, device and computer equipment for tracking a target object. Background Art

[0002] In real life or in public places, it is sometimes necessary to track important objects in a specific area. Intelligent monitoring systems can play an important role in this task by using computer vision technology. Through target detection and recognition, the system can automatically identify and mark key objects in the video, and then continue to track the location of these objects.

[0003] In the related art, the visual features of objects learned through a fixed data set are fixed and unchanged, which makes the monitoring system unable to adapt to changes in the monitoring environment during the tracking process, resulting in omissions or errors in the tracking process. Therefore, the target object tracking method in the related art needs to be improved. Summary of the invention

[0004] The present application provides a target object tracking method, device and computer equipment, which solve the technical problem that the adaptability of the target object tracking method in the related art needs to be improved, and achieve a more stable tracking effect for the target object.

[0005] In order to achieve the above objectives, the main technical solutions adopted in this application include:

[0006] In a first aspect, an embodiment of the present application provides a method for tracking a target object, the method comprising:

[0007] Acquire an initial frame image and several frames of images to be detected; wherein the shooting time of the initial frame image is earlier than the shooting time of the images to be detected;

[0008] Determining initial visual features of the target object based on the initial frame image, and storing the initial visual features in a long-term memory sequence;

[0009] If the target object is detected in any image to be detected, the intermediate visual features of the target object detected based on any image to be detected are stored in a short-term memory sequence;

[0010] If the short-term memory sequence has reached a preset storage state, a detection is performed on whether the target object exists in the latest image to be detected, and the long-term memory sequence or the short-term memory sequence is updated according to the detection result of the latest image to be detected, so as to track the target object according to the long-term memory sequence and the short-term memory sequence.

[0011] Optionally, updating the long-term memory sequence according to the detection result of the latest image to be detected includes:

[0012] If the detection result indicates that the target object is not detected in the latest image to be detected, the latest image to be detected is discarded, and the intermediate visual feature with the shortest storage time in the short-term memory sequence is stored in the long-term memory sequence.

[0013] Optionally, updating the short-term memory sequence according to the detection result of the latest image to be detected includes:

[0014] If the detection result indicates that the target object is detected in the latest image to be detected, the intermediate visual features with the longest storage time are discarded from the short-term memory sequence, and the latest visual features of the target object obtained based on the latest image to be detected are updated to the short-term memory sequence.

[0015] Optionally, before acquiring the initial frame image and the plurality of frames of images to be detected, the method further includes:

[0016] In response to an interactive operation of a target limb part on an object to be tracked, the object to be tracked is determined as the target object; wherein the initial frame image is a video image frame obtained by shooting a process of the interactive operation.

[0017] Optionally, after acquiring the initial frame image, the method further includes:

[0018] Limb segmentation and posture detection are performed on the initial frame image to obtain intersection pixel coordinates of the target limb part and the target object, and the intersection pixel coordinates are determined as prompt information.

[0019] Optionally, determining the initial visual features of the target object based on the initial frame image includes:

[0020] According to the prompt information, feature extraction is performed on the initial frame image to obtain the initial position feature and the initial visual feature of the target object in the initial frame image; and the initial position feature is stored in a long-term memory sequence.

[0021] Optionally, if the target object is detected in any image to be detected, storing the intermediate visual features of the target object detected based on any image to be detected into a short-term memory sequence includes:

[0022] A similarity comparison is performed on any image to be detected based on the initial visual features and the initial position features of the target object. If the similarity comparison result indicates that the target object is detected in any image to be detected, the intermediate visual features of any image to be detected and the intermediate position features of the target object are stored in the short-term memory sequence.

[0023] Optionally, updating the long-term memory sequence or the short-term memory sequence according to the detection result of the latest image to be detected includes:

[0024] If the target object is not detected in the latest image to be detected, the latest image to be detected is discarded, and the intermediate visual features and intermediate position features with the shortest storage time in the short-term memory sequence are stored in the long-term memory sequence;

[0025] If the target object is detected in the latest image to be detected, the intermediate visual features and intermediate position features with the longest storage time are discarded from the short-term memory sequence, and the latest visual features and latest position features of the latest image to be detected are updated to the short-term memory sequence.

[0026] In a second aspect, an embodiment of the present application provides a tracking device for a target object, the device comprising:

[0027] An image acquisition module, used to acquire an initial frame image and a plurality of frames of images to be detected; wherein the shooting time of the initial frame image is earlier than the shooting time of the images to be detected;

[0028] A visual feature determination module, used for determining the initial visual features of the target object based on the initial frame image, and storing the initial visual features in a long-term memory sequence;

[0029] A visual feature storage module, configured to store intermediate visual features of the target object detected based on any image to be detected into a short-term memory sequence if the target object is detected in any image to be detected;

[0030] A target object detection module is used to detect whether the target object exists in the latest image to be detected if the short-term memory sequence has reached a preset storage state, and to update the long-term memory sequence or the short-term memory sequence according to the detection result of the latest image to be detected, so as to track the target object according to the long-term memory sequence and the short-term memory sequence.

[0031] In a third aspect, an embodiment of the present application provides a computer device, including:

[0032] A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the method described in any one of the above embodiments by executing the computer instructions.

[0033] In the present application, firstly, an initial frame image and several frames of images to be detected are obtained; secondly, the initial visual features of the target object are determined based on the initial frame image, and the initial visual features are stored in a long-term memory sequence; then, if the target object is detected in any image to be detected, the intermediate visual features of the target object detected based on any image to be detected are stored in a short-term memory sequence; finally, if the short-term memory sequence has reached a preset storage state, whether the target object exists in the latest image to be detected is detected, and the long-term memory sequence or the short-term memory sequence is updated according to the detection result of the latest image to be detected, so as to track the target object according to the long-term memory sequence and the short-term memory sequence. Through the synergistic effect of the long-term memory sequence and the short-term memory sequence, the adaptability of target object detection and tracking can be effectively improved, especially when the environment changes, a relatively stable tracking effect can still be maintained. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] In order to more clearly illustrate the specific implementation methods of the present application or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0035] Figure 1 A flowchart of a target object tracking method provided in an embodiment of this specification;

[0036] Figure 2 A schematic diagram of the effect of limb segmentation of the YOLO v8 model provided in the embodiments of this specification;

[0037] Figure 3 A schematic diagram of a holistic model detection deviation provided in an embodiment of this specification;

[0038] Figure 4 A schematic diagram of a method for correcting holistic model detection deviation provided in an embodiment of this specification;

[0039] Figure 5 A schematic diagram of a tracking scenario provided in an embodiment of this specification;

[0040] Figure 6 A schematic diagram of a target object tracking device provided in an embodiment of this specification;

[0041] Figure 7 A schematic diagram of the structure of a computer device provided in an embodiment of this specification. DETAILED DESCRIPTION

[0042] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present application.

[0043] The development of intelligent security monitoring technology has brought great convenience to people's lives and made it possible to track important objects. In daily life, especially in public places, people need to track certain key objects in real time through monitoring systems. However, related technologies usually rely on fixed data sets to learn the visual features of objects, so these features are fixed during the tracking process, which makes the system unable to flexibly respond to changes in the monitoring environment. For example, when the light, object appearance or background in the scene changes, the system may not be able to accurately identify and continue to track the target, resulting in omissions or errors, and these problems may continue to recur in subsequent tracking processes.

[0044] Based on this, the present application provides a method for tracking a target object, firstly obtaining an initial frame image and several frames of images to be detected; secondly determining the initial visual features of the target object based on the initial frame image, and storing the initial visual features in a long-term memory sequence; then if the target object is detected in any image to be detected, storing the intermediate visual features of the target object detected based on any image to be detected in a short-term memory sequence; finally, if the short-term memory sequence has reached a preset storage state, detecting whether there is a target object in the latest image to be detected, and updating the long-term memory sequence or the short-term memory sequence according to the detection result of the latest image to be detected, so as to track the target object according to the long-term memory sequence and the short-term memory sequence. Through the synergistic effect of the long-term memory sequence and the short-term memory sequence, the accuracy and robustness of target object detection and tracking can be effectively improved, especially when the environment changes, it can ensure that the system can track continuously and stably.

[0045] According to an embodiment of the present application, an embodiment of a method for tracking a target object is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0046] See also Figure 1 The target object tracking method provided in the embodiment of the present application comprises the following steps:

[0047] S110, acquiring an initial frame image and several frames of images to be detected.

[0048] S120, determining initial visual features of the target object based on the initial frame image, and storing the initial visual features in a long-term memory sequence.

[0049] S130: If a target object is detected in any image to be detected, the intermediate visual features of the target object detected based on any image to be detected are stored in a short-term memory sequence.

[0050] S140. If the short-term memory sequence has reached a preset storage state, detect whether there is a target object in the latest image to be detected, and update the long-term memory sequence or the short-term memory sequence according to the detection result of the latest image to be detected, so as to track the target object according to the long-term memory sequence and the short-term memory sequence.

[0051] The initial frame image may be the first image obtained by the intelligent monitoring system in which the target object to be tracked exists. In this image, the area where the target object is located may be selected to help the monitoring system identify the target object and perform subsequent detection and tracking. The images to be detected may be subsequent images continuously collected by the camera after the intelligent monitoring system recognizes the target object in the initial frame image. These images are used by the system to further detect and track the target object.

[0052] The initial visual features can be obtained by extracting features from the initial frame image, which is an abstract representation of the appearance of the target object. It should be noted that the visual features of the target object extracted in the intelligent monitoring system are related to the color, texture, shape, edge and other attributes of the target object, but do not directly correspond to the physical entity, but are high-level abstract features extracted from the image by the deep learning model. The visual features extracted in this way are helpful for identifying and tracking the target object in the image. The intermediate visual features can be obtained by extracting features from the image when the target object is detected in the image to be detected.

[0053] The long-term memory sequence stores the visual features of the target object extracted from the key images. These key images usually include the initial frame image and other important images. For example, if the target object is not detected in a certain image to be detected, the image in which the target object was last detected can be used as an important image, and the visual features of the target object extracted from the image are stored in the long-term memory sequence. The visual feature information in the long-term memory sequence does not expire over time, but has long-term validity. These visual features provide a deep understanding of the target object, helping the intelligent monitoring system to detect the target object more effectively in the subsequent monitoring process. Even in a dynamic environment, when the appearance of the target object changes, the system can perform effective detection based on the long-term memory sequence.

[0054] The short-term memory sequence is used to temporarily store the intermediate visual features of the detected target object. This is a storage structure with limited capacity. It only stores a certain number of intermediate visual features instead of storing all historical intermediate visual features. The number can be 1, 2, 3... any value. A larger number can bring higher detection accuracy, but the calculation complexity will also be higher. It is necessary to find a balance between the two. When the number of stored intermediate visual features reaches the maximum value, the intermediate visual features extracted from the image of the latest detected target object will be stored in the short-term memory sequence, and the intermediate visual features with the longest storage time in the short-term memory sequence will be removed. The short-term memory sequence helps the system refer to the latest visual features of the target object during detection, thereby improving the accuracy of target object detection.

[0055] The preset storage state may be a state in which the number of intermediate visual features stored in the short-term memory sequence reaches a maximum.

[0056] For example, the features of the target object can be learned by fine-tuning a deep learning model, such as a pre-trained YOLO v8 model, and then the model can be applied to the tracking of the target object. When the intelligent monitoring system wants to track the target object, it first needs to obtain the initial frame image of the target object in real-time monitoring, and then extract features from the initial frame image through the model to obtain the initial visual features of the target object, which are stored in the long-term memory sequence. This is also the first visual feature in the long-term memory sequence of the target object. After starting to track the target object, the intelligent monitoring system needs to detect the target object on each image to be detected that is subsequently acquired. When detecting the first image to be detected, the model can be used to extract features of the image to be detected to obtain the first intermediate visual feature, and then the intermediate visual feature is compared with the initial visual feature for similarity, which can be calculated by cosine similarity or Euclidean distance. When the similarity reaches a preset similarity threshold, it is determined that the target object is detected, and the intermediate visual feature is stored in the short-term memory sequence. It should be noted that when several intermediate visual features have been stored in the short-term memory sequence, when detecting the next image to be detected, the intermediate visual features of the image to be detected and the initial frame image features stored in the long-term memory sequence and the similarity of several intermediate visual features stored in the short-term memory sequence can be comprehensively considered, so that the target object can be identified more accurately. For example, the intermediate visual features of the image to be detected are first compared with the initial visual features and the intermediate visual features in each short-term memory sequence for similarity, and then these similarity comparison results are added according to the weight to obtain a comprehensive similarity value, and finally compared with the preset similarity threshold. It is understandable that the weight of each similarity comparison result can be obtained by trying different weight combinations and comparing the image detection effects under different weight combinations. Since the initial frame image is manually determined and is relatively reliable, the initial visual features are more reliable than the intermediate visual features. Therefore, the weight corresponding to the similarity comparison result of the initial visual features can be set to the maximum value of all weights.

[0057] For example, if the short-term memory sequence is set to store up to 6 intermediate visual feature data, when the number of stored features reaches 6, if the target object is successfully detected in the next image to be detected, it is necessary to first delete the intermediate visual features with the longest storage time, and then store the intermediate visual features of the current image into the short-term memory sequence. This helps the system to quickly respond and update the visual features of the target object during real-time processing, thereby ensuring the accuracy of target object detection.

[0058] For example, if the target object exists in the next image to be detected but the system fails to detect it due to environmental changes (such as light getting stronger or darker), the intermediate visual feature with the shortest storage time in the short-term memory sequence can be stored in the long-term memory sequence. In the calculation of the comprehensive similarity value, since the similarity comparison result of the intermediate visual feature of the image to be detected and the visual feature information of the long-term memory can be set to account for a larger proportion, and the similarity comparison result with the visual feature information of the short-term memory can be set to account for a smaller proportion, after the intermediate visual feature with the shortest storage time in the short-term memory sequence is stored in the long-term memory sequence, when the target object is tracked according to the long-term memory sequence and the short-term memory sequence, the comprehensive similarity value can be improved so that it can exceed the threshold, thereby improving the accuracy of detection and achieving a more stable tracking effect.

[0059] In the above embodiment, firstly, an initial frame image and several frames of images to be detected are obtained; secondly, the initial visual features of the target object are determined based on the initial frame image, and the initial visual features are stored in the long-term memory sequence; then, if the target object is detected in any image to be detected, the intermediate visual features of the target object detected based on any image to be detected are stored in the short-term memory sequence; finally, if the short-term memory sequence has reached a preset storage state, whether the target object exists in the latest image to be detected is detected, and the long-term memory sequence or the short-term memory sequence is updated according to the detection result of the latest image to be detected, so as to track the target object according to the long-term memory sequence and the short-term memory sequence. Through the synergistic effect of the long-term memory sequence and the short-term memory sequence, the adaptability of target object detection and tracking can be effectively improved, especially when the environment changes, with a more stable tracking effect.

[0060] In some embodiments, the long-term memory sequence is updated according to the detection result of the latest image to be detected, including: if the detection result shows that the target object is not detected in the latest image to be detected, discarding the latest image to be detected, and storing the intermediate visual feature with the shortest storage time in the short-term memory sequence in the long-term memory sequence.

[0061] Specifically, when several intermediate visual features have been stored in the short-term memory sequence, the target object detection can be performed on the image to be detected by calculating the comprehensive similarity. First, the intermediate visual features of the image to be detected are compared with the initial visual features in the long-term memory sequence and other important visual features transferred from the short-term memory sequence to the long-term memory sequence, as well as the intermediate visual features in each short-term memory sequence. Then, these similarity comparison results are weighted according to the preset weights to obtain a comprehensive similarity value. Then, the comprehensive similarity value is compared with the pre-set similarity threshold to determine whether the target object is detected. It should be noted that the weight corresponding to the similarity comparison result between the intermediate visual features of the image to be detected and the visual features in the long-term memory sequence is larger, while the weight corresponding to the similarity comparison result with the intermediate visual features in the short-term memory sequence is smaller.

[0062] Exemplarily, the detection result shows that the target object is not detected in the latest image to be detected. This may be due to the fact that there is really no target object in the latest image to be detected due to occlusion. It may also be that when the similarity between the intermediate visual features of the target object and the visual features in the long-term memory sequence becomes lower and lower due to environmental changes (such as light changes), the comprehensive similarity value will become lower and lower until it is lower than the preset similarity threshold, causing the system to mistakenly judge that the target object is not detected. If the target object is blocked, the intermediate visual features with the shortest storage time in the short-term memory sequence are stored in the long-term memory sequence. When the target object appears again, it is more conducive to accurate detection of the target object. If the target object feature changes greatly due to environmental changes, the intermediate visual features with the shortest storage time in the short-term memory sequence, which are also the intermediate visual features closest to the intermediate visual features in the image to be detected, are stored in the long-term memory sequence. This can increase the comprehensive similarity value and make it higher than the preset similarity threshold, thereby achieving accurate detection of the target object. It should be noted that, no matter which of the above situations is used, the latest image to be detected is discarded and its corresponding intermediate visual features are not stored in the short-term memory sequence to avoid feature dilution.

[0063] In the above embodiment, when the detection result indicates that the target object is not detected in the latest image to be tested, the memory ability of the target object feature is enhanced by converting the last target object feature in the short-term memory sequence into long-term memory. When the detection result is wrong, it can be corrected through long-term memory, thereby improving the adaptability of the target object tracking method.

[0064] In some embodiments, the short-term memory sequence is updated according to the detection results of the latest image to be detected, including: if the detection results indicate that the target object is detected in the latest image to be detected, discarding the intermediate visual features with the longest storage time from the short-term memory sequence, and updating the latest visual features of the target object obtained based on the latest image to be detected to the short-term memory sequence.

[0065] Specifically, the number of intermediate visual features stored in the short-term memory sequence is limited, and the number can be 1, 2, 3... any value. A larger number can bring higher detection accuracy, but the calculation complexity will also be higher, and a balance needs to be found between the two. When the number of stored intermediate visual features reaches the maximum value, if the next image to be detected successfully detects the target object, it is necessary to first delete the intermediate visual features with the longest storage time, and then store the latest visual features corresponding to the latest image to be detected in the short-term memory sequence. This helps the system refer to the latest visual features of the target object during detection, thereby improving the accuracy of target object detection.

[0066] In some embodiments, before acquiring an initial frame image and several frames of images to be detected, the target object tracking method also includes: determining the object to be tracked as a target object in response to an interactive operation of a target limb part with respect to the object to be tracked; wherein the initial frame image is a video image obtained by capturing the process of the interactive operation.

[0067] Among them, the target limb part can be a limb part used by the intelligent monitoring system to interact with the target object to be tracked, such as a hand, foot, head, etc. The interactive operation can be an action acting on the object to be tracked through the target limb part, such as pointing a finger at the object to be tracked, stepping on the object to be tracked, etc. Specifically, by monitoring the interactive operation between the target limb part and the object to be tracked, the intelligent monitoring system can determine the target object. For example, when the target limb part is a finger and a finger is detected in the video image pointing to an object to be tracked, the object can be determined to be the target object based on the intersection of the pixel coordinates of the fingertip and the object. Since the target object is included in the video image captured during the interactive operation, it can be used as the initial frame image.

[0068] In the above embodiment, in response to the interactive operation of the target limb part on the object to be tracked, the object to be tracked is determined as the target object. This interactive operation method based on the target limb part can effectively improve the usability of the system and provide a good user experience.

[0069] In some embodiments, after acquiring the initial frame image, the target object tracking method further includes: performing limb segmentation and posture detection on the initial frame image to obtain the intersection pixel coordinates of the target limb part and the target object, and determining the intersection pixel coordinates as prompt information.

[0070] The prompt information may be a coordinate point, a box, text information, etc., which is used to enable the model to determine the target object to be tracked, such as the SAM2 model.

[0071] Posture detection can be achieved by identifying and analyzing the key points of the human body through computer vision technology, and then inferring the postures of various parts of the body. For example, the holistic human posture detection model of mediapipe can be used to perform human posture detection on the initial frame image to obtain the intersection pixel coordinates of the target body part and the target object.

[0072] Limb segmentation can be done by using computer vision technology to accurately segment various parts of the human body (such as head, upper limbs, lower limbs, etc.) from images or videos so that each part can be analyzed or processed separately. For example, the YOLOv8 model can be used to segment the initial frame image. The result is as follows: Figure 2 shown.

[0073] For example, on the initial frame image, the holistic human posture detection model is used to detect and output the human skeleton, and the YOLO v8 model is used to perform limb segmentation on the initial frame image. If the skeleton falls within the limb segmentation area, it means that the holistic detection of the skeleton is relatively accurate. When the person is on the left side of the image and the target object is on the right side of the image, the rightmost pixel coordinates of the intersection of the fingertip and the target object can be obtained as prompt information.

[0074] For example, on the initial frame image, the holistic human posture detection model is used to detect and output the human skeleton, and the YOLO v8 model is used to segment the limbs of the initial frame image. If the skeleton does not fall within the limb segmentation area, it means that when the holistic human posture detection model is used to detect the human body, the skeleton output result has a relatively large deviation, resulting in the inability to detect the correct fingertips, such as Figure 3 As shown in the figure, the arm skeleton output is not on the human body at all. In this case, the limb segmentation results of the limb segmentation model YOLO v8 can be combined and corrected to a certain extent through geometric calculation. Figure 4 In most cases, since the posture detection is relatively accurate for the relative position and angle between joints, but there will be some deviations in the position, the steps of the correction method can be:

[0075] 1) Starting from the elbow joint and wrist joint output by the posture detection, two intersection points are generated along the normal direction with the segmentation boundary. The center of the two intersection points can be used as the corrected elbow joint and wrist joint;

[0076] 2) Starting from the corrected wrist joint, query the segmentation boundary farthest from the human body within a certain angle above and below the line connecting the elbow joint and the wrist joint, which is the intersection pixel coordinates of the fingertip and the target object.

[0077] For example, in the SAM2 model, based on the input intersection pixel coordinates, the weight of the area near the coordinates can be enhanced through the attention mechanism to achieve accurate segmentation of the target object and provide accurate target object information for subsequent tracking.

[0078] Exemplarily, the target object may be selected by clicking or selecting a frame with a mouse on a terminal of the intelligent tracking system, such as a computer client or a mobile phone client.

[0079] In the above embodiment, by performing limb segmentation and posture detection on the initial frame image, the intersection pixel coordinates of the target limb part and the target object are obtained, and the intersection pixel coordinates are determined as prompt information, so that the target object to be tracked can be accurately determined.

[0080] In some embodiments, determining the initial visual features of the target object based on the initial frame image includes: extracting features from the initial frame image according to prompt information to obtain initial position features and initial visual features of the target object in the initial frame image; and storing the initial position features in a long-term memory sequence.

[0081] Among them, the initial position feature can be a representation of the initial spatial position of the target object. After the target object is segmented in the initial frame image, it can be encoded according to the position of the target object in the image. It is the basis for the model to understand and process the spatial position of the target object.

[0082] Exemplarily, the target object is tracked by the SAM2 model. After the initial frame image is acquired through the camera, the initial frame image is firstly feature extracted to obtain hierarchical initial frame image features with different granularity (one layer contains the features of the target object). Secondly, because the same layer may also contain features of other objects, the prompt information, such as the pixel coordinates of the intersection of the fingertip and the target object, is input into the SAM2 model. The model determines the level and position of the initial visual features of the target object in the initial frame image features through the prompt information. Then, the obtained initial visual features of the target object are segmented and decoded to obtain the segmentation result of the target object in the initial frame image. Then, the position of the target object is encoded according to the segmentation result to obtain the initial position features of the target object. Finally, the initial position features and initial visual features are stored in the long-term memory sequence. It should be noted that SAM2 is used to continuously detect the target object in the image to be detected to achieve tracking of the target object. Although SAM2 focuses more on the visual features at the pixel level during detection, in some complex scenes, the position features can allow the model to understand the spatial relationship between different objects, making the detection results more accurate. Therefore, before tracking begins, the initial position features and initial visual features of the target object are stored together in the long-term memory sequence.

[0083] In the above embodiment, features are extracted from the initial frame image according to the prompt information to obtain the initial position features and initial visual features of the target object in the initial frame image. The initial position features are then stored in a long-term memory sequence, which together with the initial visual features provide the most accurate reference and comparison basis for the detection of target objects in subsequent images.

[0084] In some embodiments, if a target object is detected in any image to be detected, the intermediate visual features of the target object detected based on any image to be detected are stored in a short-term memory sequence, including: performing a similarity comparison on any image to be detected based on the initial visual features and initial position features of the target object; if the similarity comparison result indicates that the target object is detected in any image to be detected, the intermediate visual features of any image to be detected and the intermediate position features of the target object are stored in the short-term memory sequence.

[0085] Among them, the intermediate position feature can be obtained by encoding its position according to the segmentation result after the target object is segmented in the image to be detected.

[0086] Exemplarily, the target object is tracked by the SAM2 model, and the pixel coordinates of the intersection of the limb part and the target object are used to give prompt information on the initial frame image. The model extracts features from the initial frame image to obtain the initial position features and initial visual features of the target object in the initial frame image, and stores the initial position features and initial visual features in the long-term memory sequence. When the next frame of the image to be detected is obtained, it is not necessary to provide prompt information to the model again. The model detects the target object based on the memory of the initial frame image, mainly the initial visual features and initial position features. The detection is mainly based on the visual features for similarity comparison. If the similarity exceeds the preset similarity threshold, it is judged that the target object is detected, and the intermediate visual features and intermediate position features of the frame image are stored in the short-term memory sequence. If the target object is not detected, the frame of the image to be detected is discarded, and the intermediate visual features and intermediate position features will not be stored in the short-term memory sequence to ensure that the short-term memory sequence stores the feature data of the target object to prevent feature dilution.

[0087] It should be noted that the pre-trained SAM2 model adopts the Transformer architecture and uses a streaming memory mechanism for efficient video processing and context memory. In order to enhance the interactive ability and generalization ability of the model, user interaction scenarios are simulated in the training, and data enhancement is performed by randomly sampling click points and reversing the time sequence. The above mechanism can greatly improve the accuracy of the SAM2 model in detecting the target object, but the premise is that the complete video sequence must be obtained and all images are feature extracted. Therefore, the pre-trained SAM2 model is suitable for detecting and tracking the target object in the offline video, but it cannot be directly applied to the real-time video stream acquired by the camera for target detection. In this embodiment, the memory mechanism of the pre-trained SAM2 model is simplified. Specifically, the features of the target object extracted by the SAM2 model are stored in a short-term memory sequence. Only a limited number of the latest intermediate visual features and intermediate position features can be saved in this sequence, instead of memorizing all the historical intermediate visual features and intermediate position features, and the features of the images where the target object is not detected will not be memorized, ensuring that the short-term memory sequence is the features of the target object, which can effectively avoid the problem of feature dilution. By simplifying the memory mode of the original SAM2 model, the complexity of the model calculation can be reduced, which is conducive to real-time tracking and detection of the target object. In addition, the mechanism of the short-term memory sequence that only saves the features of the image of the detected target object can make up for the accuracy loss caused by the simplified memory mode, which is beneficial to the success rate of SAM2 model tracking.

[0088] Exemplarily, if one or more short-term memory data, that is, one or more pairs of intermediate visual features and intermediate position features, have been stored in the short-term memory sequence, when the model obtains the next frame of image to be detected, it detects the target object based on the memory feature data and short-term memory feature data of the initial frame image, and the SAM2 model performs similarity comparison through its attention mechanism, comprehensively considering the comparison results of the feature data of the image to be detected and the long-term memory feature data and the short-term memory feature data, among which the comparison results with the long-term memory data account for a larger proportion, and the comparison results with the short-term memory data account for a smaller proportion.

[0089] In some embodiments, updating the long-term memory sequence or the short-term memory sequence according to the detection result of the latest image to be detected includes:

[0090] If the target object is not detected in the latest image to be detected, the latest image to be detected is discarded, and the intermediate visual features and intermediate position features with the shortest storage time in the short-term memory sequence are stored in the long-term memory sequence;

[0091] If the target object is detected in the latest image to be detected, the intermediate visual features and intermediate position features with the longest storage time are discarded from the short-term memory sequence, and the latest visual features and latest position features of the latest image to be detected are updated to the short-term memory sequence.

[0092] Specifically, see Figure 5 , the toy car is the target object. The intelligent monitoring system tracks the target object toy car through the SAM2 model. First, it needs to be initialized. For example, the user points his finger to the toy car and uses the voice input command "track this object" through the client of the intelligent monitoring system. The model obtains the initial position features and initial visual features of the toy car according to the user prompts, and stores them in the long-term memory sequence, and then obtains the latest image to be detected and starts to detect the target object toy car. The SAM2 model detects the toy car in the latest image to be detected based on the visual features and position features of the toy car stored in the long-term memory sequence and the short-term memory sequence. If the toy car is not detected in the latest image to be detected and the target object is lost, the latest image to be detected is discarded, and the intermediate visual features and intermediate position features with the shortest storage time in the short-term memory sequence are stored in the long-term memory sequence; if the target object is detected in the latest image to be detected, the intermediate visual features and intermediate position features with the longest storage time are discarded from the short-term memory sequence, and the latest visual features and latest position features of the latest image to be detected are updated to the short-term memory sequence.

[0093] Exemplarily, when the target object is tracked by the SAM2 model, when there is only one memory data in the long-term memory sequence, that is, only the initial visual features and initial position features of the initial frame image, a maximum of 6 memory data can be stored in the short-term memory sequence, that is, 6 pairs of intermediate visual features and intermediate position features. The current state of the short-term memory sequence is that 6 memory data are already stored. When the model obtains the next frame of the image to be detected, it comprehensively considers the similarity comparison results of the feature data of the image to be detected and the long-term memory feature data and the short-term memory feature data through its attention mechanism. The similarity obtained does not reach the preset similarity threshold, and it is determined that the target object is not detected. It may be due to occlusion that there is really no target object in the latest image to be detected, or there may be a target object in the image, but it cannot be detected by the current detection method. For example, when the similarity between the intermediate visual features of the target object and the visual features in the long-term memory sequence becomes lower and lower due to environmental changes (such as light changes), the comprehensive similarity value will become lower and lower until it is lower than the preset similarity threshold. At this time, the system will determine that the target object is not detected. In this case, the intermediate visual features and intermediate position features with the shortest storage time in the short-term memory sequence are stored in the long-term memory sequence. The SAM2 model can strengthen the memory based on the features in all the long-term memory sequences, and then improve the comprehensive similarity value of the target object to make it higher than the preset similarity threshold, thereby adaptively achieving accurate detection of the target object.

[0094] Illustratively, in the above example, when environmental changes (such as light changes) cause the intermediate visual features of the target object to become less and less similar to the visual features in the long-term memory sequence, resulting in a lower and lower comprehensive similarity value, tracking errors may also occur. By storing the intermediate visual features and intermediate position features with the shortest storage time in the short-term memory sequence in the long-term memory sequence, the SAM2 model strengthens the memory based on the features in all long-term memory sequences, and then the problem of tracking errors can be corrected.

[0095] It should be noted that in the pre-trained SAM2 model, when the features of the target object gradually change and become increasingly different from the visual features and position features of the target object in memory, resulting in subsequent failure to detect, the user clicks on the target object again, that is, prompts the model again, so that the model can update and strengthen the memory of the target object and detect the target object again. This method is improved in this embodiment, and the user only needs to give prompt information on the initial frame. When the subsequent detection of the target object fails, the model updates and strengthens the memory of the target object by storing the intermediate visual features and intermediate position features with the shortest storage time in the short-term memory sequence in the long-term memory sequence, thereby reducing the need for manual intervention.

[0096] When the target object is not detected, the latest image to be detected can be discarded, and its corresponding intermediate visual features are not stored in the short-term memory sequence to avoid the problem of feature dilution.

[0097] For example, if in the above example, the model detects the target object as the result of the latest image to be detected, then the latest visual features and the latest position features of the image need to be stored in the short-term memory sequence. This approach helps the model to only contain the latest valid information in the short-term memory sequence, so as to focus more on the current detection results and target tracking. It is understandable that since the amount of memory data stored in the short-term memory sequence is limited, it is also necessary to discard the intermediate visual features and intermediate position features with the longest storage time to store the latest visual features and the latest position features.

[0098] In the above embodiment, when the target object is not detected in the latest image to be detected, the intermediate visual features and intermediate position features with the shortest storage time in the short-term memory sequence are stored in the long-term memory sequence. By converting the last target object feature in the short-term memory sequence into long-term memory, the memory ability of the target object feature is enhanced. When an error occurs in the detection result, it can be corrected through long-term memory, thereby improving the adaptability of the target object tracking method.

[0099] See also Figure 6 In an embodiment of the present application, a target object tracking device 600 is further provided. The target object tracking device 600 includes:

[0100] The image acquisition module 610 is used to acquire an initial frame image and a plurality of frames of images to be detected; wherein the shooting time of the initial frame image is earlier than the shooting time of the images to be detected;

[0101] A visual feature determination module 620, configured to determine initial visual features of a target object based on an initial frame image, and store the initial visual features in a long-term memory sequence;

[0102] A visual feature storage module 630 is used to store the intermediate visual features of the target object detected based on any image to be detected into a short-term memory sequence if the target object is detected in any image to be detected;

[0103] The memory update module 640 is used to detect whether there is a target object in the latest image to be detected if the short-term memory sequence has reached a preset storage state, and update the long-term memory sequence or the short-term memory sequence according to the detection result of the latest image to be detected, so as to track the target object according to the long-term memory sequence and the short-term memory sequence.

[0104] In some embodiments, the memory update module 640 is also used to discard the latest image to be detected if the detection result shows that the target object is not detected in the latest image to be detected, and store the intermediate visual feature with the shortest storage time in the short-term memory sequence in the long-term memory sequence.

[0105] In some embodiments, the memory update module 640 is also used to discard the intermediate visual features with the longest storage time from the short-term memory sequence if the detection result indicates that the target object is detected in the latest image to be detected, and update the latest visual features of the target object obtained based on the latest image to be detected to the short-term memory sequence.

[0106] In some implementations, the target object tracking device 600 further includes a target object determination module:

[0107] The target object determination module is used to determine the object to be tracked as the target object in response to the interactive operation of the target limb part to the object to be tracked.

[0108] In some implementations, the image acquisition module 610 further includes a prompt information determination module:

[0109] The prompt information determination module is used to perform limb segmentation and posture detection on the initial frame image to obtain the intersection pixel coordinates of the target limb part and the target object, and determine the intersection pixel coordinates as prompt information.

[0110] In some embodiments, the visual feature determination module 620 is further used to extract features from the initial frame image according to the prompt information, to obtain the initial position features and initial visual features of the target object in the initial frame image; and to store the initial position features in the long-term memory sequence.

[0111] In some embodiments, the visual feature storage module 630 is also used to perform a similarity comparison on any image to be detected based on the initial visual features and initial position features of the target object. If the similarity comparison result indicates that the target object is detected in any image to be detected, the intermediate visual features of any image to be detected and the intermediate position features of the target object are stored in the short-term memory sequence.

[0112] In some embodiments, the memory update module 640 is also used to discard the latest image to be detected if the target object is not detected in the latest image to be detected, and store the intermediate visual features and intermediate position features with the shortest storage time in the short-term memory sequence into the long-term memory sequence; if the target object is detected in the latest image to be detected, discard the intermediate visual features and intermediate position features with the longest storage time from the short-term memory sequence, and update the latest visual features and latest position features of the latest image to be detected into the short-term memory sequence.

[0113] The further functional description of each of the above modules and units is the same as that of the above corresponding embodiments and will not be repeated here.

[0114] The target object tracking device in this embodiment is presented in the form of a functional unit, where the unit refers to an ASIC (Application Specific Integrated Circuit) circuit, a processor and memory that executes one or more software or fixed programs, and / or other devices that can provide the above functions.

[0115] See also Figure 7 , Figure 7 is a schematic diagram of the structure of a computer device provided in an embodiment of the present application, such as Figure 7 As shown, the computer device includes: one or more processors 10, a memory 20, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. Various components are connected to each other using different buses for communication, and can be installed on a common mainboard or installed in other ways as needed. The processor can process the instructions executed in the computer device, including instructions stored in or on the memory to display the graphical information of the GUI on an external input / output device (such as, a display device coupled to the interface). In some optional embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple computer devices can be connected, and each device provides some necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Figure 7 A processor 10 is taken as an example.

[0116] The processor 10 may be a central processing unit, a network processor or a combination thereof. The processor 10 may further include a hardware chip. The hardware chip may be a dedicated integrated circuit, a programmable logic device or a combination thereof. The programmable logic device may be a complex programmable logic device, a field programmable gate array, a general purpose array logic or any combination thereof.

[0117] The memory 20 stores instructions executable by at least one processor 10, so that the at least one processor 10 executes the method shown in the above embodiment.

[0118] The memory 20 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the computer device, etc. In addition, the memory 20 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some optional embodiments, the memory 20 may optionally include a memory remotely arranged relative to the processor 10, and these remote memories may be connected to the computer device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0119] The memory 20 may include a volatile memory, such as a random access memory; the memory may also include a non-volatile memory, such as a flash memory, a hard disk or a solid state drive; the memory 20 may also include a combination of the above types of memory.

[0120] The computer device also includes an input device 30 and an output device 40. The processor 10, the memory 20, the input device 30 and the output device 40 may be connected via a bus or other means. Figure 7 The example of connecting through bus is taken in the following.

[0121] The input device 30 can receive input digital or character information, and generate key signal input related to the user settings and function control of the computer device, such as a touch screen, a keypad, a mouse, a track pad, a touch pad, an indicator bar, one or more mouse buttons, a trackball, a joystick, etc. The output device 40 may include a display device, an auxiliary lighting device (e.g., an LED) and a tactile feedback device (e.g., a vibration motor), etc. The above-mentioned display device includes but is not limited to a liquid crystal display, a light emitting diode, a display and a plasma display. In some optional embodiments, the display device can be a touch screen.

[0122] The embodiment of the present application also provides a computer-readable storage medium. The above method according to the embodiment of the present application can be implemented in hardware, firmware, or can be implemented as a computer code that can be recorded in a storage medium, or can be implemented as a computer code that is originally stored in a remote storage medium or a non-temporary machine-readable storage medium and will be stored in a local storage medium through a network download, so that the method described herein can be stored in such software processing on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only storage memory, a random access memory, a flash memory, a hard disk or a solid-state hard disk, etc.; further, the storage medium can also include a combination of the above types of memories. It can be understood that a computer, a processor, a microprocessor controller, or programmable hardware includes a storage component that can store or receive software or computer code. When the software or computer code is accessed and executed by a computer, a processor, or hardware, the method shown in the above embodiment is implemented.

[0123] The embodiment of the present application provides a computer program product, which includes computer instructions, which are stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the method of any embodiment of the present application.

[0124] Although the embodiments of the present application are described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present application, and such modifications and variations are all within the scope defined by the appended claims.

[0125] The systems, devices, modules or units described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, the computer may be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0126] For the convenience of description, the above device is described in various units according to their functions. Of course, when implementing the present application, the functions of each unit can be implemented in the same or multiple software and / or hardware.

[0127] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0128] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0129] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0130] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0131] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0132] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0133] The above is only an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.

[0134] Although the embodiments of the present application have been described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present application, and such modifications and variations are all within the scope defined by the appended claims.

Claims

1. A method for tracking a target object, characterized in that: The method comprises: Acquire an initial frame image and several frames of images to be detected; wherein the shooting time of the initial frame image is earlier than the shooting time of the images to be detected; Determining initial visual features of the target object based on the initial frame image, and storing the initial visual features in a long-term memory sequence; If the target object is detected in any image to be detected, the intermediate visual features of the target object detected based on any image to be detected are stored in a short-term memory sequence; If the short-term memory sequence has reached a preset storage state, a detection is performed on whether the target object exists in the latest image to be detected, and the long-term memory sequence or the short-term memory sequence is updated according to the detection result of the latest image to be detected, so as to track the target object according to the long-term memory sequence and the short-term memory sequence.

2. The method according to claim 1, characterized in that The long-term memory sequence is updated according to the detection result of the latest image to be detected, including: If the detection result indicates that the target object is not detected in the latest image to be detected, the latest image to be detected is discarded, and the intermediate visual feature with the shortest storage time in the short-term memory sequence is stored in the long-term memory sequence.

3. The method according to claim 1, characterized in that The short-term memory sequence is updated according to the detection result of the latest image to be detected, including: If the detection result indicates that the target object is detected in the latest image to be detected, the intermediate visual features with the longest storage time are discarded from the short-term memory sequence, and the latest visual features of the target object obtained based on the latest image to be detected are updated to the short-term memory sequence.

4. The method according to claim 1, characterized in that: Before acquiring the initial frame image and the plurality of frames of images to be detected, the method further includes: In response to an interactive operation of a target limb part on an object to be tracked, the object to be tracked is determined as the target object; wherein the initial frame image is a video image frame obtained by shooting a process of the interactive operation.

5. The method according to claim 4, characterized in that After acquiring the initial frame image, the method further includes: Limb segmentation and posture detection are performed on the initial frame image to obtain intersection pixel coordinates of the target limb part and the target object, and the intersection pixel coordinates are determined as prompt information.

6. The method according to claim 5, characterized in that The determining the initial visual features of the target object based on the initial frame image includes: According to the prompt information, feature extraction is performed on the initial frame image to obtain the initial position feature and the initial visual feature of the target object in the initial frame image; and the initial position feature is stored in a long-term memory sequence.

7. The method according to claim 6, characterized in that If the target object is detected in any image to be detected, storing the intermediate visual features of the target object detected based on any image to be detected into a short-term memory sequence includes: A similarity comparison is performed on any image to be detected based on the initial visual features and the initial position features of the target object. If the similarity comparison result indicates that the target object is detected in any image to be detected, the intermediate visual features of any image to be detected and the intermediate position features of the target object are stored in the short-term memory sequence.

8. The method according to claim 7, characterized in that The updating of the long-term memory sequence or the short-term memory sequence according to the detection result of the latest image to be detected includes: If the target object is not detected in the latest image to be detected, the latest image to be detected is discarded, and the intermediate visual features and intermediate position features with the shortest storage time in the short-term memory sequence are stored in the long-term memory sequence; If the target object is detected in the latest image to be detected, the intermediate visual features and intermediate position features with the longest storage time are discarded from the short-term memory sequence, and the latest visual features and latest position features of the latest image to be detected are updated to the short-term memory sequence.

9. A tracking device for a target object, characterized in that: The device comprises: An image acquisition module, used to acquire an initial frame image and a plurality of frames of images to be detected; wherein the shooting time of the initial frame image is earlier than the shooting time of the images to be detected; A visual feature determination module, used for determining the initial visual features of the target object based on the initial frame image, and storing the initial visual features in a long-term memory sequence; A visual feature storage module, configured to store intermediate visual features of the target object detected based on any image to be detected into a short-term memory sequence if the target object is detected in any image to be detected; A target object detection module is used to detect whether the target object exists in the latest image to be detected if the short-term memory sequence has reached a preset storage state, and to update the long-term memory sequence or the short-term memory sequence according to the detection result of the latest image to be detected, so as to track the target object according to the long-term memory sequence and the short-term memory sequence.

10. A computer device, characterized in that: include: A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the method according to any one of claims 1 to 8 by executing the computer instructions.

Images