Target tracking method and device based on visual key-value vector, equipment and medium

Abstract

Embodiments of the present disclosure disclose a target tracking method and device based on a visual key-value vector, an equipment and a medium. A specific implementation of the method comprises: using a target tracking visual language large model to perform visual coding on coded video stream information to obtain a mapping information set and a visual key-value feature information set; determining a target object tracking template for preset object information to be tracked; determining a feature similarity set of a target visual feature information corresponding to a subsequent video frame and the target object tracking template; and determining target fine position information of the preset object information to be tracked in the subsequent video frame according to the feature similarity set. The implementation can realize real-time full target tracking of target objects of different sizes without additional inference computing power, improve the accuracy and efficiency of target tracking, and reduce tracking latency by using the key-value vector generated by the target tracking visual language large model in the inference process.

Description

Technical Field

[0001] Embodiments of this disclosure relate to the field of computer technology, and more specifically to target tracking methods, apparatus, devices, and media based on visual key-value vectors. Background Technology

[0002] The goal of AI glasses (Artificial Intelligence Glasses) in industrial applications, such as video semantic understanding, is to not only understand the semantic information in videos but also to annotate and track key information. With the increasing prevalence of Vision-Language Models (VLMs) in video understanding tasks, collaborating with existing tracking technologies (such as optical flow tracing algorithms and filtering algorithms) to perform target tracking has become a new direction. However, while VLMs have strong semantic reasoning capabilities, the need for them to re-detect or locate targets in each frame leads to long tracking inference latency and high computational costs, making real-time tracking impossible. The common approach for tracking targets in video streams is to input the encoded video stream into a VLM to obtain the positional information of the target object in subsequent videos, thus achieving target tracking.

[0003] However, in practice, it has been found that when using the above method to track targets in video streams, the following technical problems often arise: Since VLM does not pay attention to the number of video frames in the video stream during inference, if inference tracking is performed on every video frame in the video stream, it will increase the latency of target tracking and the power consumption of the server running the VLM model. In addition, a large amount of intermediate semantic feature information will be discarded after each inference, resulting in a waste of computing and storage resources. At the same time, when the background of the video stream is complex (e.g., there is occlusion, interference from similar objects, etc.), the multi-level feature information output by the intermediate VLM is not utilized, resulting in low target tracking accuracy and difficulty in adapting to the balance between tracking efficiency and tracking accuracy for different sizes. Summary of the Invention

[0004] The summary portion of this disclosure is intended to provide a brief overview of the concepts, which will be described in detail in the detailed description portion later. This summary portion is not intended to identify key or essential features of the claimed technical solutions, nor is it intended to limit the scope of the claimed technical solutions.

[0005] Some embodiments of this disclosure propose target tracking methods, apparatuses, devices, and media based on visual key-vectors to address one or more of the technical problems mentioned in the background section above.

[0006] In a first aspect, some embodiments of this disclosure provide a target tracking method based on visual key-value vectors, comprising: using an acquired target tracking visual language large model to visually encode acquired video stream information to obtain a mapping information set and a visual key-value feature information set; determining a target object tracking template for a preset object to be tracked based on the mapping information set, the visual key-value feature information set, and preset model selection hierarchical information; determining a target visual feature information set corresponding to subsequent video frames and a feature similarity set of the target object tracking template, wherein the subsequent video frames are any video frames located after the target video frame containing the preset object to be tracked, and the target visual feature information set is obtained based on the visual key-value feature information set corresponding to the subsequent video frames; and determining the target precision position information of the preset object to be tracked in the subsequent video frames based on the feature similarity set.

[0007] Optionally, the aforementioned preset object information to be tracked is information about multiple target objects to be tracked; and the aforementioned method further includes: constructing a template cache queue for each of the multiple target objects to be tracked to obtain a template cache queue set; performing parallel target tracking on the multiple target objects to be tracked according to the template cache queue set to obtain an initial bounding box information set; and performing bounding box suppression processing on the initial bounding box information set to obtain a target precision location information set for the multiple target objects to be tracked.

[0008] Optionally, determining the target precision location information of the preset object to be tracked in the subsequent video frames based on the feature similarity set includes: generating object bounding box information for the preset object to be tracked in response to determining that the number of visual features of the visual key-value feature information corresponding to the preset object to be tracked satisfies a first preset target size condition; determining pixel-level location information of the preset object to be tracked in the subsequent video frames based on the video stream information and the target precision location information of the preset object to be tracked in the target video frames; and fusing the pixel-level location information and the object bounding box information to obtain the target precision location information.

[0009] Optionally, determining the target precision location information of the preset object to be tracked in the subsequent video frame based on the feature similarity set includes: in response to determining that the number of visual features of the visual key-value feature information corresponding to the preset object to be tracked satisfies a second preset target size condition, selecting at least one feature similarity that satisfies a preset similarity threshold condition from the feature similarity set; generating a video frame binary mask for the subsequent video frame based on the at least one feature similarity; performing spatial connected component aggregation on the video frame binary mask to obtain candidate target region information; and determining the minimum bounding rectangle of the candidate target region information as the target precision location information.

[0010] Optionally, determining the target precision location information of the preset object to be tracked in the subsequent video frames based on the feature similarity set includes: in response to determining that the number of visual features of the visual key value feature information corresponding to the preset object to be tracked meets the third preset target size condition, filtering the feature similarity set to obtain a filtering vector similarity index; and determining the target precision location information based on the mapping information set and the filtering vector similarity index.

[0011] Optionally, the method further includes: updating the target object tracking template and frame buffer queue frame by frame according to the visual key-value feature information set corresponding to the target fineness location information, to obtain the updated target object tracking template and the updated frame buffer queue, so as to perform target tracking on subsequent video frame sets, wherein the frame buffer queue is a queue that caches the visual key-value feature information set and corresponding mapping information of a preset number of video frames located before the subsequent video frames; in response to determining that preset object information to be tracked is detected in subsequent video frames, determining the target fineness location information according to the updated target object tracking template and the updated frame buffer queue; in response to determining that preset object information to be tracked is not detected in subsequent video frames, determining the target fineness location information according to the frame buffer queue and the target object tracking template.

[0012] Optionally, determining the pixel-level position information of the preset object to be tracked in the subsequent video frames based on the video stream information and the target precision position information of the preset object to be tracked in the target video frame includes: in response to determining that motion vector information of the target video frame exists in the video stream information, determining the pixel-level position information based on the target precision position information on the target video frame and the motion vector information, or determining the object bounding box information as pixel-level position information.

[0013] Optionally, determining the pixel-level position information of the preset object to be tracked in the subsequent video frames based on the video stream information and the target precision position information of the preset object to be tracked in the target video frame includes: determining optical flow motion field information between the target video frame and the subsequent video frames in response to determining that there is no motion vector information of the target video frame in the video stream information; and determining the pixel-level position information based on the target precision position information on the target video frame and the optical flow motion field information.

[0014] Optionally, the above-mentioned use of the acquired target tracking visual language big model to visually encode the acquired video stream information to obtain a mapping information set and a visual key-value feature information set includes: in response to detecting that the target tracking visual language big model satisfies the dynamic resolution support condition, determining the pixel affine mapping information set of the video stream information and the target video stream information, wherein the target video stream information is video stream information that meets the input requirements of the target tracking visual language big model; and using the target tracking visual language big model to visually encode each video frame included in the target video stream information to obtain video frame blocks. The process involves: 1) determining a set of video frame block visual vectors and a set of visual key-value vectors; 2) determining a set of pixel video block mapping information for the set of video frame block visual vectors and the set of visual key-value vectors; 3) determining a set of key-value video block mapping information for the set of video frame block visual vectors and the set of visual key-value vectors; 4) defining the set of pixel affine mapping information, the set of pixel video block mapping information, and the set of key-value video block mapping information as a mapping information set, and defining the set of video frame block visual vectors and the set of visual key-value vectors as a set of visual key-value feature information, and 5) caching the set of mapping information and the set of visual key-value vectors into a frame buffer queue.

[0015] Optionally, determining the target object tracking template for the preset object to be tracked based on the mapping information set, the visual key-value feature information set, and the preset model selection level information includes: in response to detecting that the target tracking visual language large model meets the dynamic resolution support condition, extracting at least one visual key-value feature information corresponding to the preset object to be tracked from the visual key-value feature information set according to the pixel affine mapping information set and the pixel video block mapping information set included in the mapping information set, as the target visual key-value feature information set; and performing vector fusion on the target visual key-value feature information set according to the key-value video block mapping information set included in the mapping information set and the preset model selection level information to obtain the target object tracking template.

[0016] Optionally, the above determination of the target visual feature information set corresponding to the subsequent video frame and the feature similarity set of the target object tracking template includes: determining the target visual feature information set based on the visual key value feature information corresponding to the subsequent video frame; for each target visual feature information, performing the following weighted fusion steps: determining the similarity between the target visual feature information and the target object tracking template at the preset model selection level information to obtain a similarity set; and performing weighted fusion on the similarity set to obtain feature similarity.

[0017] Secondly, some embodiments of this disclosure provide a target tracking device based on visual key-value vectors, comprising: a visual encoding unit configured to use an acquired target tracking visual language large model to visually encode acquired video stream information to obtain a mapping information set and a visual key-value feature information set; a first determining unit configured to determine a target object tracking template for a preset object to be tracked based on the mapping information set, the visual key-value feature information set, and a preset model selection level information; a second determining unit configured to determine a target visual feature information set corresponding to a subsequent video frame and a feature similarity set of the target object tracking template, wherein the subsequent video frame is any video frame located after the target video frame containing the preset object to be tracked, and the target visual feature information set is obtained based on the visual key-value feature information set corresponding to the subsequent video frame; and a third determining unit configured to determine the target precision position information of the preset object to be tracked in the subsequent video frame based on the feature similarity set.

[0018] Thirdly, some embodiments of this disclosure provide an electronic device, including: one or more processors; and a storage device having one or more programs stored thereon, such that when the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any implementation of the first aspect.

[0019] Fourthly, some embodiments of this disclosure provide a computer-readable medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the method as described in any implementation of the first aspect.

[0020] The above embodiments of this disclosure have the following beneficial effects: The target tracking method based on visual key-value vectors in some embodiments of this disclosure can achieve real-time full target tracking of target objects of different sizes without increasing inference computing power by utilizing the key-value vectors generated during the inference process of the target tracking visual language large model. This improves the accuracy and efficiency of target tracking and reduces tracking latency. Specifically, the reasons for increased target tracking latency and power consumption, wasted computing and storage resources, lower target tracking accuracy, and difficulty in balancing tracking efficiency and accuracy across different screen sizes are as follows: Since the Visual Key Vector (VLM) does not consider the number of video frames in the video stream during inference, performing inference and tracking on every single frame increases target tracking latency and the power consumption of the server running the VLM model. Furthermore, a large amount of intermediate semantic feature information is discarded after each inference, wasting computing and storage resources. Simultaneously, when the video stream background is complex (e.g., with occlusion, interference from similar objects, etc.), the multi-level feature information output by the VLM is not utilized, resulting in lower target tracking accuracy and difficulty in balancing tracking efficiency and accuracy across different screen sizes. Based on this, some embodiments of the target tracking method based on visual key vectors in this disclosure can first use an acquired target tracking visual language large model to visually encode the acquired video stream information, obtaining a mapping information set and a visual key feature information set. Therefore, when the large-scale visual language model for target tracking satisfies dynamic resolution support, the aforementioned mapping information set can include coordinate mapping information before and after the model processes video stream information. When dynamic resolution support is not met, the aforementioned mapping information set may not include coordinate mapping information before and after the model processes video stream information. This ensures compatibility with the dynamic resolution processing of the large model and effectively solves the output coordinate mapping problem caused by the network structure of the large model itself. The visual key-value feature information set is used to subsequently determine the target object tracking template. Secondly, based on the aforementioned mapping information set, the aforementioned visual key-value feature information set, and the preset model selection level information, a target object tracking template is determined for the preset object information to be tracked. Thus, by using the preset model selection level information and performing multi-layer semantic fusion on the visual key-value feature information set, a target object tracking template is obtained. This integrates shallow spatial cues and high-level semantic cues, making it less susceptible to interference from similar textures and complex backgrounds, thus improving the accuracy of subsequent target tracking, i.e., location information localization. Then, the target visual feature information set corresponding to the subsequent video frame and the feature similarity set of the target object tracking template are determined. The subsequent video frame is any video frame located after the target video frame where the preset object information to be tracked is located. The target visual feature information set is obtained based on the visual key value feature information set corresponding to the subsequent video frame.Therefore, by reusing visual key-value feature information for similarity calculation, the waste of computational resources can be reduced, and the feature similarity set facilitates subsequent target weight tracking. Finally, based on the aforementioned feature similarity set, the target fine-grained position information of the preset object to be tracked in the subsequent video frames is determined. Thus, the target fine-grained position information is used to determine the target position at different fine-grained levels (e.g., pixel level, image block level) by using the number of visual vectors of the video frame block occupied by the preset object to be tracked. This supports full target tracking of different sizes from small to large, effectively balancing tracking efficiency and tracking accuracy. In summary, this target tracking method based on visual key-value vectors reuses the intermediate vectors, i.e., visual key-value feature information, of the large-scale visual language model for target tracking. It eliminates the need for additional convolutional networks or detectors for inference, thus ignoring additional computational overhead. This enables long battery life, time-saving, efficiency, accuracy, and energy efficiency on edge devices, such as head-mounted displays. Furthermore, it achieves full target tracking of targets of different sizes through similarity matching and fuses visual key-value feature information from different levels. This integrates shallow spatial cues and high-level semantic cues, reducing the waste of computational and storage resources, decreasing power consumption during inference, improving the accuracy and efficiency of target tracking, and lowering tracking latency. Attached Figure Description

[0021] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and elements are not necessarily drawn to scale.

[0022] Figure 1 This is a flowchart of some embodiments of the target tracking method based on visual key vectors according to the present disclosure; Figure 2 This is a schematic diagram illustrating the number of visual vectors covering a video frame block with preset object information to be tracked, according to some embodiments of the target tracking method based on visual key vectors of this disclosure. Figure 3 This is a schematic diagram of the entire processing flow from acquiring video stream information to determining the fine-grained location information of the target based on visual key vectors according to some embodiments of the target tracking method of this disclosure; Figure 4 This is a flowchart of some other embodiments of the target tracking method based on visual key vectors according to the present disclosure; Figure 5 These are schematic diagrams illustrating the structure of some embodiments of a target tracking device based on visual key vectors according to the present disclosure; Figure 6 This is a schematic diagram of the structure of an electronic device suitable for implementing some embodiments of the present disclosure. Detailed Implementation

[0023] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0024] It should also be noted that, for ease of description, only the parts relevant to the invention are shown in the accompanying drawings. Unless otherwise specified, the embodiments and features described in this disclosure can be combined with each other.

[0025] It should be noted that the concepts of "first" and "second" mentioned in this disclosure are used only to distinguish different devices, modules or units, and are not used to limit the order of functions performed by these devices, modules or units or their interdependencies.

[0026] It should be noted that the terms "a" and "a plurality of" used in this disclosure are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0027] The names of messages or information exchanged between multiple devices in the embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of such messages or information.

[0028] This disclosure will now be described in detail with reference to the accompanying drawings and embodiments.

[0029] Figure 1 A flow 100 of some embodiments of a visual key-vector-based target tracking method according to the present disclosure is shown. This visual key-vector-based target tracking method includes the following steps: Step 101: Using the acquired target tracking visual language big model, the acquired video stream information is visually encoded to obtain a mapping information set and a visual key-value feature information set.

[0030] In some embodiments, the execution entity (e.g., a head-mounted display device) of the aforementioned target tracking method based on visual key-value vectors can employ an acquired target tracking visual language large model to visually encode the acquired video stream information, obtaining a mapping information set and a visual key-value feature information set. The execution entity can be a head-mounted display device integrating components such as sensors, processors, and microdisplays, as well as a device for processing video stream information. For example, the head-mounted display device can be AI glasses or AR glasses (Augmented Reality Glasses). The aforementioned target tracking visual language large model can be a VLM model that performs image block visual encoding on the input video stream information, outputting a visual key-value feature information set. For example, the aforementioned target tracking visual language large model can be one of the following: GLM-4.5V model, MiniCPM-V model. The aforementioned video stream information can be information that requires target tracking, follows common video encoding formats (e.g., H.264, H.265, AV1, or AV9), and includes both video and text prompts. The aforementioned video stream information can also be information that only includes the encoded video. The visual key-value feature information in the aforementioned visual key-value feature information set can be in the form of vectors, representing the visual and semantic features of the video stream information. For example, the aforementioned visual key-value feature information may include: video frame block feature vectors, i.e., visual tokens, and key-value vectors (KV Cache). When the aforementioned target tracking visual language large model needs to scale the input video stream information, the aforementioned mapping information set may include: an affine matrix recording the coordinate mapping relationship between video frames before and after image scaling, and index mapping information recording the various feature vectors output by the target tracking visual language large model. For example, the aforementioned index mapping information may include: index mapping information between video frame block feature vectors and key-value vectors, and mapping relationship information between video frame blocks obtained after the target tracking visual language large model has scaled the video stream and video frame block feature vectors. When the aforementioned target tracking visual language large model does not need to scale the input video stream information, the aforementioned mapping information set may include: index mapping information recording the various feature vectors output by the target tracking visual language large model.

[0031] In some optional implementations of certain embodiments, the above-mentioned use of the acquired target tracking visual language large model to visually encode the acquired video stream information to obtain a mapping information set and a visual key-value feature information set may include the following steps: The first step, in response to the detection that the target tracking visual language large model meets the dynamic resolution support condition, is to determine the pixel affine mapping information set of the aforementioned video stream information and the target video stream information. The target video stream information is the video stream information that meets the input requirements of the aforementioned target tracking visual language large model. The aforementioned dynamic resolution support condition can be a condition under which the aforementioned target tracking visual language large model needs to perform image scaling processing on the input video stream information. The pixel affine mapping information in the aforementioned pixel affine mapping information set can be a 2*3 affine matrix describing the coordinate mapping relationship between each video frame included in the aforementioned video stream information and the corresponding video frame in the target video stream information. The aforementioned coordinate mapping relationship can be expressed as: .

[0032] in, This represents the x-coordinate in the processed coordinates of the target video stream information obtained after image scaling by the aforementioned target tracking visual language large model. This represents the scaling factor when the above-mentioned target tracking visual language large model performs image scaling. This represents the x-coordinate of the coordinates in the video frames included in the aforementioned video stream information. This indicates the offset along the horizontal axis. This represents the x-coordinate and y-coordinate of the processed coordinates in the target video stream information obtained after image scaling in the above target tracking visual language large model. This represents the x-coordinate of the coordinates in the video frames included in the aforementioned video stream. This indicates the offset along the vertical axis.

[0033] The second step involves using the aforementioned target tracking visual language model to visually encode each video frame in the target video stream information, resulting in a video frame block set, a video frame block visual vector set, and a visual key-value vector set. The video frame blocks in the video frame block set can be non-overlapping video frame blocks obtained by dividing each video frame according to a partition size using the aforementioned target tracking visual language model. The video frame block set can be a video frame block obtained by partitioning a single video frame. For example, the partition size can be 14 pixels * 14 pixels or 16 pixels * 16 pixels. The video frame block visual vectors in the video frame block visual vector set can be feature vectors representing the visual spatial features of the video frame blocks, i.e., visual tokens. These video frame block visual vectors can also be feature vectors obtained by the aforementioned target tracking visual language model using ViT (Vision Transformer) to perform linear projection and positional encoding on the video frame block set. The visual key-value vectors in the visual key-value vector set can be feature vectors representing the semantic features of the video frame blocks, i.e., key vectors (K vectors) and value vectors (V vectors). The aforementioned visual key-value vector can be a feature vector obtained by using the L-layer Transformer included in the target tracking visual language large model to perform deep feature extraction on the visual vectors of the aforementioned video frame blocks.

[0034] The third step is to determine the pixel video block mapping information set for the aforementioned video frame block group set and the aforementioned video frame block visual vector set. The pixel video block mapping information in the aforementioned pixel video block mapping information set can be mapping information describing the processed coordinates of the image-scaled video frame block group and the patch index of the corresponding video frame block visual vector set. The aforementioned pixel video block mapping information can be represented as: .

[0035] in, Indicates the division dimensions. This represents the row index of the visual vector of a video frame block. The column index represents the visual vector of a video frame block. This represents the linear index of the video frame visual vector within the aforementioned video frame visual vector. The video stream information includes video frame segmentation to obtain the first... A video frame block, that is, the order of blocks from left to right and from top to bottom.

[0036] The fourth step is to determine the key-value video block mapping information set for the aforementioned video frame block visual vector set and the aforementioned visual key-value vector set. The key-value video block mapping information set can be used to describe the mapping information between the Transformer layer number and token index between the visual vectors of multiple video frame blocks included in a video frame and the corresponding multiple visual key-value vectors. For example, using a Transformer layer located at layer L, each video frame block visual vector token idx will generate a dimension of... key vector Sum value vector If the visual key-value vector is a key-value video block mapping information, then the key-value video block mapping information can be the key vector. Value vector Information composed of token index.

[0037] The fifth step involves defining the aforementioned pixel affine mapping information set, pixel video block mapping information set, and key-value video block mapping information set as mapping information sets, and defining the aforementioned video frame block visual vector set and visual key-value vector set as visual key-value feature information sets. The mapping information set and visual key-value vector set are then cached in a frame cache queue. This frame cache queue can be a queue that caches a preset number of video frames that have already undergone target tracking. The preset number of video frames can be a pre-defined number. For example, the preset number of video frames can be determined based on specific circumstances and is not limited here. Target tracking can be the tracking of the position information of the target object already obtained in the corresponding video frame. When the frame cache queue has no cache space and there are new video frame mapping information sets and visual key-value vector sets that need to be cached, the frame cache queue deletes the mapping information set and visual key-value vector set of the first cached video frame and caches the mapping information set and visual key-value vector set of the latest video frame. It should be noted that when the target tracking visual language large model does not meet the dynamic resolution support condition, the above mapping information set only includes: the above pixel video block mapping information set and the above key-value video block mapping information set.

[0038] In addressing the aforementioned technical challenges by employing technical solutions, the target tracking scenario—a complex interaction scenario based on edge devices—often presents the following technical problems: Based on a large language model, target tracking requires the full storage of the visual key-value feature information set extracted from each Transformer layer. Because this set contains a large amount of redundant feature information, it demands significant storage resources, prolongs target tracking time, and reduces accuracy and efficiency. Considering the following requirements for this application scenario: adaptability to target tracking on edge devices, adaptability to high-concurrency multi-target tracking and recognition, adaptability to key-value feature caching, and adaptability to high resolution, we have decided to adopt the following solution: In some optional implementations of certain embodiments, after using the acquired target tracking visual language large model to visually encode the acquired video stream information to obtain a mapping information set and a visual key-value feature information set, the above method may further include the following steps: The first step involves performing model-level grouping processing on the aforementioned visual key-value feature information set to obtain a visual key-value grouped feature information set. This set may include: feature information representing the information of the preset object to be tracked, and feature information formed by the background image interacting with the preset object to be tracked. In practice, the execution entity can perform the following filtering steps for each Transformer layer in the aforementioned target tracking visual language large model: First, determine the attention entropy of the Transformer layer. This attention entropy can measure the uncertainty or uniformity of the attention weight distribution of the query feature vector relative to the key feature vector set in the Transformer layer. The lower the attention entropy value, the more concentrated the attention is on the key video frame block visual vector set; the higher the value, the more evenly the attention is distributed across the video frame block visual vectors. Second, normalize the attention entropy as a layer-level budget ratio. Subsequently, determine the target video frame block visual vector set of the target video frames included in the video stream information. This target video frame block visual vector set can be the visual vector set of the target video frame group covered by the bounding box of the previous video frame. Next, the dot product of the visual vector set of the target video frame block and the transpose of the key vector set included in the Transformer layer is determined to obtain the attention matrix set. Then, the query dimension of each attention matrix in the attention matrix set is summed to obtain the attention cumulative sum set. Next, the difference between the product of the layer budget ratio and the number of video frame blocks included in the target video frame and the number of video frame blocks corresponding to the preset object information to be tracked is determined as the remaining selection quantity. Finally, using the attention cumulative sum set in descending order, the remaining selection quantity of visual key value features representing the background is selected from the above visual key value feature information set as the background visual key value feature information, forming the visual key value group feature information group.

[0039] The second step involves performing hierarchical perceptual budget retention on the aforementioned visual key-value grouped feature information set to obtain a visual key-value retained feature information set. The visual key-value retained feature information in this set can be obtained by further retaining feature information based on the visual key-value grouped feature information set according to the key-value retention ratio determined for different tracking types in the tracking scene (e.g., occlusion, deformation, rapid movement, etc.). In practice, the executing entity can utilize a tracking type mapping form to perform hierarchical perceptual budget retention on the aforementioned visual key-value grouped feature information set to obtain the visual key-value retained feature information set. This tracking type mapping form can be a form used to record the feature information retention ratios corresponding to different types of tracking types.

[0040] The third step is to perform low-rank compression on the aforementioned set of visual key-value preserved feature information to obtain a compressed set of key-value feature information. In practice, the aforementioned execution entity can use the LoRC (Low Rank Compensation) algorithm to perform low-rank compression on the aforementioned set of visual key-value preserved feature information to obtain a compressed set of key-value feature information.

[0041] The fourth step is to quantize the compressed key-value feature information set to obtain a quantized key-value feature information set. The quantized key-value feature information in this set can be obtained by performing channel quantization on the key feature information included in the compressed key-value feature information set and INT4 quantization on the included value feature information.

[0042] The fifth step involves performing quantization compensation processing on the quantized key-value feature information set to obtain a compensated key-value feature information set. In practice, the execution entity can use the GEAR (Generative Inference with Approximation Error Reduction) algorithm to perform quantization compensation processing on the compressed key-value feature information set to obtain a compensated key-value feature information set.

[0043] The sixth step involves caching the compensated key-value feature information set to obtain a visual key-value cached feature information set, which serves as the visual key-value feature information set. The visual key-value cached feature information in this set can be used to record the association between the clipped key-value tensor set and the initial object boundary position information set, and is cached in fixed-size blocks to improve memory utilization. In practice, the execution entity can utilize the PagedAttention algorithm to cache the quantized key-value feature information set to obtain the visual key-value cached feature information set, which serves as the visual key-value feature information set.

[0044] The above technical solution, combined with steps 102-104 and related content, serves as an inventive point of this disclosure, solving the technical problem of "leading to the need for large amounts of storage resources, extending the target tracking time, and reducing the accuracy and efficiency of target tracking." The factors leading to the need for large amounts of storage resources, extending the target tracking time, and reducing the accuracy and efficiency of target tracking are often as follows: Based on a large language model, in target tracking scenarios, the visual key-value feature information set extracted from each Transformer layer needs to be fully stored. Since the full set of visual key-value feature information contains a large amount of redundant feature information, it leads to the need for large amounts of storage resources, extending the target tracking time, and reducing the accuracy and efficiency of target tracking. If the above factors are solved, the effect of reducing storage resource consumption, shortening the target tracking time, and improving the accuracy and efficiency of target tracking can be achieved. To achieve this effect, this disclosure firstly groups the model layers using the budget ratio obtained by normalizing the attention entropy of each layer. When attention is concentrated, a small amount of background is retained; when attention is dispersed, a larger amount of background is retained. This avoids the loss of key information caused by global uniform pruning. Each layer only retains the feature information of the object to be tracked and the background with high attention accumulation, which can initially reduce the amount of feature information data. Secondly, hierarchical perception budget retention employs different retention ratios for different tracking types, preserving background feature information interacting with the target in high-complexity tracking scenarios, improving tracking performance in complex environments, and further reducing the storage space occupied by feature information. Next, low-rank compression performs low-rank decomposition on the visual key-value feature information set, further reducing storage while retaining key information. Following this, quantization and quantization compensation processes are applied. Quantization allows for higher-precision channel quantization of distribution-sensitive key features, avoiding target loss due to quantization, while compressing value features to 4 bits further reduces storage. Quantization compensation reduces the impact of quantization on tracking accuracy. Then, cache optimization divides the compensated key-value feature information set into fixed-size pages, dynamically allocating and releasing them to reduce memory fragmentation. The paging mechanism can handle target tracking of long video frame sequences, ensuring accuracy and efficiency in target tracking while further compressing feature information, shortening target tracking time, and improving the battery life of edge devices, i.e., head-mounted displays.

[0045] Step 102: Select hierarchical information based on the mapping information set, visual key value feature information set and preset model, and determine the target object tracking template for the preset object information to be tracked.

[0046] In some embodiments, the execution entity can determine a target object tracking template for a preset object to be tracked based on the mapping information set, the visual key-value feature information set, and the preset model selection level information. The preset model selection level information can be a pre-defined level of at least one Transformer layer included in the target tracking visual language large model. For example, the preset model selection level information may include the first layer, the second L-th layer, and the L-th layer of the Transformer layer set included in the target tracking visual language large model. L can be the number of Transformer layers included in the Transformer layer set. The preset object to be tracked information can be pre-defined information about the object to be tracked in the video stream. This preset object to be tracked information can be object information obtained by the user through natural language description prompts, object information determined by clicking a specified method, or object information obtained from the detection bounding box output by the target tracking visual language large model. The target object tracking template can be template feature information formed by fusing the visual key-value feature information set output by the Transformer layer set corresponding to the selected preset model selection level information. The aforementioned target object tracking template can be a vector template obtained by fusing a set of key vectors included in the visual key-value feature information set, or a vector template obtained by fusing a set of included value vectors, or a vector template obtained by fusing a set of concatenated key vectors and value vectors. If the preset model selects a single-layer layer of layer information, then the aforementioned target object tracking template can be a template for the layer information selected by the preset model. If the preset model selects multiple layers of layer information, then the aforementioned target object tracking template can be multiple templates for each layer in the layer information selected by the preset model.

[0047] In some optional implementations of certain embodiments, determining the target object tracking template for the preset object to be tracked based on the mapping information set, the visual key feature information set, and the preset model selection hierarchy information may include the following steps: The first step involves, in response to the detection that the target tracking visual language large model meets the dynamic resolution support condition, extracting at least one visual key-value feature information corresponding to the preset object information from the aforementioned visual key-value feature information set, based on the pixel affine mapping information set and pixel video block mapping information set included in the aforementioned mapping information set, as the target visual key-value feature information set. In practice, the executing entity can first determine the video frame block set corresponding to the preset object information from the aforementioned video frame block set using the pixel affine mapping information corresponding to the target video frame where the preset object information is located, as the target video frame block set. Then, using the pixel video block mapping information corresponding to the pixel video block mapping information set, selecting at least one video frame block visual vector corresponding to the preset object information from the aforementioned video frame block visual vector set, as the target visual key-value feature information.

[0048] The second step involves vector fusion of the target visual key-value feature information set based on the key-value video block mapping information set included in the aforementioned mapping information set and the aforementioned preset model selection hierarchy information, to obtain the target object tracking template. In practice, the executing entity can first select a set of video frame key-value vectors corresponding to the target video key-value feature information set from the aforementioned video frame key-value vector set using the key-value video block mapping information corresponding to the aforementioned key-value video block mapping information set, as the target video frame key-value vector set. Then, it selects at least one target video frame key-value vector from the aforementioned target video frame key-value vector set that corresponds to the aforementioned preset model selection hierarchy information. Afterward, using the key vector template construction formula, it performs hierarchical fusion of the at least one target video frame key-value vector to obtain the target object tracking template. The key vector template construction formula can be expressed as: .

[0049] in, This represents a target object tracking template composed of a set of key vectors included in the target video frame key value vector set. This represents the key vector with token index idx output by the Lth Transformer layer. express The quantity. This represents the token index sequence.

[0050] It should be noted that when the target tracking visual language large model does not meet the dynamic resolution support condition, firstly, by using the pixel video block mapping information set and the key-value video block mapping information set included in the mapping information set, at least one video frame key-value vector corresponding to the preset template selection level information is selected from the aforementioned video frame key-value vector set. Then, vector fusion is performed on the aforementioned at least one video frame key-value vector to obtain the target object tracking template.

[0051] Step 103: Determine the target visual feature information set and the feature similarity set of the target object tracking template corresponding to the subsequent video frames.

[0052] In some embodiments, the execution entity may determine the target visual feature information set corresponding to subsequent video frames and the feature similarity set of the target object tracking template. The subsequent video frames are any video frames located after the target video frame containing the preset object information to be tracked. The target visual feature information set is obtained based on the visual key-value feature information set corresponding to the subsequent video frames. The target visual feature information in the target visual feature information set may be a key vector, a value vector, or a concatenated vector of key and value vectors corresponding to the visual vectors (tokens) of each video frame block output by the target tracking visual language large model for the subsequent video frames. The feature similarity in the feature similarity set may be the calculated similarity between the target visual feature information and the target object tracking template. The feature similarity may be at least the following: cosine similarity, the reciprocal of the Euclidean distance, the reciprocal of the Mahalanobis distance, or a normalized value obtained by multiplying the dot products.

[0053] In some optional implementations of certain embodiments, determining the target visual feature information set corresponding to subsequent video frames and the feature similarity set of the target object tracking template may include the following steps: The first step is to determine the target visual feature information set based on the visual key-value feature information corresponding to the subsequent video frames. Here, the target visual feature information in the target visual feature information set can be a visual key-value vector corresponding to a video frame block.

[0054] The second step is to perform the following weighted fusion step for each target visual feature information: Sub-step 1: Determine the similarity between the aforementioned target visual feature information and the aforementioned target object tracking template at each preset model selection level, thus obtaining a similarity set. The aforementioned similarity can characterize the degree of similarity between the aforementioned target visual feature information and the aforementioned target object tracking template. The aforementioned similarity can be at least the following: cosine similarity, the reciprocal of Euclidean distance, the reciprocal of Mahalanobis distance, or a normalized value after multiplying the dot products. In practice, the executing entity can use the hierarchical similarity calculation formula to determine the similarity between the aforementioned target visual feature information and the aforementioned target object tracking template at each preset model selection level, thus obtaining a similarity set. When the aforementioned preset model selection level information is a single-layer level, the similarity set can be a single similarity. When the aforementioned preset model selection level information is the hierarchical information of multiple model layers, the similarity set can include multiple similarities. The aforementioned hierarchical similarity calculation formula can be expressed as: .

[0055] in, This indicates that, in the preset model, the selected level information is L, and the level is L, the first... The similarity between the visual feature information of a target and the target object tracking template. Indicates the first The visual feature information of a target is a feature vector at level L. In the similarity calculation formula at this level, the feature vector can be a key vector. Indicates the first The magnitude of each key vector in layer L. This indicates the modulus of the target object tracking template in layer L.

[0056] Sub-step 2 involves weighted fusion of the aforementioned similarity sets to obtain feature similarity. In practice, the executing entity can utilize a weighted fusion formula to weightedly fuse the aforementioned feature similarity sets to obtain feature similarity. The weighted fusion formula can be expressed as: .

[0057] in, Indicates feature similarity. This indicates that the preset model selection includes all model levels. The weights represent the similarity of the Lth layer. These weights can be obtained through a multilayer perceptron, and the sum of the weights of all model layers is 1. For example, if the preset model selects the Lth layer, L / 2 layer, and Lth layer, then based on expert experience, the weights of the Lth layer, L / 2 layer, and Lth layer can be set to 0.2, 0.3, and 0.5, respectively.

[0058] In addressing the aforementioned technical challenges in implementing technical solutions, the following challenges arise in the target tracking scenario: tracking targets in video streams with complex, occluded backgrounds and on edge devices. These challenges stem from the fact that shallow network layers of the target tracking visual language model extract feature vectors containing more texture information, while deep networks extract more semantic information. Determining feature similarity requires dynamically determining the weights of different network layers. However, traditional sparrow search algorithms are prone to getting stuck in local optima, resulting in lower weight accuracy and reduced feature similarity. Consequently, the accuracy of tracking the location of the target object in subsequent video frames is also low. To address the following requirements for this application scenario: adaptability to multiple fitness functions, adaptability to complex video frame backgrounds (e.g., numerous objects similar to the target in texture and color), adaptability to geometric deformation of the target, adaptability to decoupling of semantic and spatial features (i.e., shallow features deform while deep features remain stable), adaptability to resource-constrained terminal devices, and adaptability to frequent occlusion of the target, we have decided to adopt the following solution: In some optional implementations of certain embodiments, the weighted fusion of the aforementioned similarity sets to obtain feature similarity may include the following steps: The first step is to generate a localization similarity fitness function. This function can be a function that assigns weights to each model network layer within the hierarchical information to ensure that the fused similarity scores are as close as possible, thus accurately locating the position of the preset object to be tracked in subsequent video frames. This localization similarity fitness function can be a fitness function composed of the sum of a localization accuracy function and a localization discriminative function, along with a constraint function. The localization accuracy function can be a function that determines the localization accuracy by calculating the intersection-union ratio (IoU) between the predicted bounding box and the ground truth bounding box of the preset object to be tracked in subsequent video frames. The localization discriminative function can be a function used to accurately distinguish the preset object information region from the background region of the video frame; that is, the similarity of the target region should be significantly higher than that of the background region, and the similarity distribution of the background region should be concentrated. The constraint function can be a constraint function where the sum of the weights of each model network layer is 1.

[0059] The second step involves generating an initial hierarchical weight population based on the hierarchical information and target visual feature information of the aforementioned target tracking visual language large model. Each initial hierarchical weight individual in this population can be a set of weight values ​​for each level corresponding to the localization similarity fitness function. The initial hierarchical weight population may also include a population parameter set. This set may include, but is not limited to, at least one of the following: a preset population size, a preset execution threshold, a preset discoverer ratio, a preset scout ratio, the function dimension of the localization similarity fitness function, and an individual boundary range. The individual boundary range can be a boundary range composed of the maximum and minimum values ​​of the initial hierarchical weight individuals. The preset discoverer ratio can be the proportion of individuals guiding the population towards potentially high-quality solutions, typically taking any value from (0.1, 0.2). The preset scout ratio can be the proportion of individuals guiding the population to avoid risks and increase population diversity, typically taking any value from (0.1, 0.2). The preset population size can be a pre-defined number of individuals included in the initial hierarchical weight population. The aforementioned preset execution threshold can be the maximum number of pre-defined loop executions. In practice, the aforementioned execution entity can utilize the Chebyshev chaotic mapping algorithm to generate an initial hierarchical weight population based on the hierarchical information of the aforementioned target tracking visual language large model and the target visual feature information.

[0060] The third step involves performing the following weighted fusion steps based on the initial hierarchical weight population: Sub-step 1: Based on a preset discoverer ratio, randomly select a discoverer weight population and a follower weight population from the initial hierarchical weight population. The discoverer weight population can be a population randomly selected from the initial hierarchical weight population whose proportion of the initial hierarchical weight population is equal to the preset discoverer ratio. The follower weight population can be the population remaining after removing the discoverer weight population from the initial hierarchical weight population. In practice, the executing entity can first determine the number of discoverers in the discoverer weight population by multiplying the preset population size and the preset discoverer ratio. Then, randomly select the specified number of initial hierarchical weight individuals from the initial hierarchical weight population to obtain the discoverer weight population. Finally, the population obtained by removing the discoverer weight population from the initial hierarchical weight population is determined as the follower weight population.

[0061] Sub-step 2 involves filtering the follower weight population and the initial level weight population according to the location similarity fitness function to obtain target follower weight individuals, first-level weight individuals, and second-level weight individuals. The target follower weight individual can be the follower weight individual with the largest location similarity fitness function value in the follower weight population. The first-level weight individual can be the individual with the largest corresponding location similarity fitness function value in the initial level weight population. The second-level weight individual can be the individual with the smallest corresponding location similarity fitness function value in the initial level weight population. In practice, the executing entity can first input the follower weight population into the location similarity fitness function to obtain the target fitness function value set. Then, the follower weight individual corresponding to the largest target fitness function value in the target fitness function value set is determined as the target follower individual. Finally, the initial level weight population is input into the location similarity fitness function, and the initial level weight individuals corresponding to the smallest and largest location similarity fitness function values ​​are filtered out to obtain the first target initial individual and the second target initial individual.

[0062] Sub-step 3 involves updating the global and local equilibrium distribution of the follower weight population based on the target follower weight individuals and the first-level weight individuals, resulting in the updated follower weight population. The updated follower weight individuals in this updated follower weight population can be individuals that incorporate an adaptive t-distribution mutation strategy into the existing sparrow search algorithm. This strategy dynamically adjusts the degree-of-freedom parameters to intelligently regulate the search range. Specifically, a smaller degree-of-freedom parameter (i.e., the number of executions) is used in the early stages of the loop (exhibiting heavy-tailed characteristics) to enhance global exploration capabilities. As the loop iterates, the degree-of-freedom parameter is gradually increased (approaching a Gaussian distribution) to improve local exploration accuracy.

[0063] Sub-step 4: Based on the second-level weighted individuals, perform wavelet mutation position updates on each discoverer weighted individual in the discoverer weighted population to obtain the updated discoverer weighted population. The updated discoverer weighted individuals included in the updated discoverer weighted population can be, when the warning value is less than the safety threshold, the product of the position vector of the i-th sparrow in the j-th dimension and the sum of 1 and the wavelet mutation coefficient; when the warning value is greater than or equal to the safety threshold, the product of the position vector of the i-th sparrow in the j-th dimension and the product of a random number following a normally distributed [0, 1] and the identity matrix.

[0064] Sub-step 5: Based on the location similarity fitness function and the second-level weighted individuals, perform normal random position updates on the updated follower weight population and the updated discoverer weight population to obtain the reconnaissance weight population. The reconnaissance weighted individuals included in the reconnaissance weight population can be individuals obtained through normal random position updates using the reconnaissance update function of an existing sparrow search algorithm. In practice, the executing entity can first randomly select an initial reconnaissance weight population from the updated follower weight population and the updated discoverer weight population. The ratio of the initial reconnaissance weight population to the initial level weight population can be a preset reconnaissance ratio. Then, each initial reconnaissance weighted individual and the second-level weighted individuals included in the initial reconnaissance weight population are input into the reconnaissance update function to obtain the reconnaissance weight population.

[0065] Sub-step 6 involves performing a hybrid mutation on the updated follower weight population, the updated discoverer weight population, and the scout weight population to obtain a mutated weight population. The mutated weight individuals included in this mutated weight population can be individuals obtained by performing Gaussian mutation on the updated follower weight population, the updated discoverer weight population, and the scout weight population, followed by a nonlinear perturbation update.

[0066] Sub-step 7: In response to the determination that the number of times the weighted fusion step has been executed is greater than or equal to a preset execution threshold, the aforementioned similarity set is weighted and fused according to the localization similarity fitness function and the mutation weight population to obtain feature similarity. In practice, the executing entity can first input the mutation weight population into the localization similarity fitness function to obtain the target fitness function value set. Secondly, the value group corresponding to the mutation weight individual with the largest target fitness function value is selected from the target fitness function value set and determined as the weight coefficient group for each level. Then, the aforementioned similarity set is weighted and fused according to the weight coefficient group for each level to obtain feature similarity.

[0067] Fourth, in response to the determination that the number of times the weighted fusion step has been executed is less than a preset execution threshold, the mutated weight population is determined as the initial hierarchical weight population, and the sum of the number of executions and the preset value is determined as the number of executions, so that the weighted fusion step described above can be executed again. The preset value can be a pre-set value, and its value can be 1.

[0068] The above technical solution, combined with step 104 and related content, serves as an inventive point of this disclosure, solving the technical problem of "low accuracy of weights, leading to reduced feature similarity, and further reducing the accuracy of tracking the location information of the preset object to be tracked in subsequent video frames." The factors causing low accuracy of weights, reduced feature similarity, and further reducing the accuracy of tracking the location information of the preset object to be tracked in subsequent video frames are often as follows: Since the feature vectors extracted by the shallow network layers of the target tracking visual language large model include more texture information, and the feature vectors extracted by the deep network layers include more semantic information, determining feature similarity requires dynamically determining the weights of different network layers. However, when using the traditional sparrow search algorithm for determination, there is a problem of easily getting trapped in local optima, resulting in low accuracy of weights, reduced feature similarity, and further reducing the accuracy of tracking the location information of the preset object to be tracked in subsequent video frames. If the above factors are solved, the accuracy of weights can be improved, feature similarity can be increased, and the accuracy of tracking the location information of the preset object to be tracked in subsequent video frames can be improved. To achieve this effect, this disclosure first generates a localization similarity fitness function. This fitness function simultaneously considers localization accuracy and anti-interference discriminative power, combined with a constraint function with a total weight of 1. The function considering localization accuracy ensures that the bounding box only covers the area where the preset object information is located, but the boundaries of the bounding box are unclear. The discriminative power function maximizes the feature difference between the target and the background, preventing tracking drift. Secondly, an initial hierarchical weight population is generated using the Chebyshev chaotic mapping algorithm. Chebyshev chaotic mapping has excellent ergodicity and aperiodicity, effectively avoiding the local optimum trap caused by the population clustering in a certain region of the search space in the early stages. This improves the uniformity and diversity of the initially generated population distribution, covers a wider area, enhances the population's global search capability, and reduces search blind spots.Next, based on the initial hierarchical weight population, the following processes are iteratively executed: Global and local equilibrium distribution position updates for the follower weight population are performed, incorporating an adaptive t-distribution mutation strategy. This dynamically adjusts the number of executions (degrees of freedom) to intelligently regulate the search range. Initially, the smaller degree of freedom enhances global exploration capabilities, while increasing it improves local development accuracy. Finally, the discoverer weight population undergoes wavelet mutation position updates. This wavelet mutation strategy leverages the scaling and translation properties of wavelet functions to achieve multi-scale exploration and development of the solution space. This method leverages the multi-scale properties of wavelet functions to refine the neighborhood of the current optimal solution, thereby improving the quality and diversity of solutions. It performs normal position updates on the detection weight population and introduces hybrid mutations into the updated population. This involves fine-tuning high-fitness individuals near the current solution to enhance local search capabilities, and applying nonlinear perturbations to the optimal solution individual to help it escape local convergence regions, increasing the generation of new potential solutions from high-fitness regions. This process continues until a termination condition is met (the number of executions exceeds a preset threshold), resulting in optimal weight coefficient sets for each level. These sets are then used for weighted fusion to improve weight accuracy and feature similarity. Finally, the feature similarity set is used to determine the fine-grained location information of the target, improving the accuracy of tracking the location information of the preset object to be tracked in subsequent video frames.

[0069] Step 104: Based on the feature similarity set, determine the target precision location information of the preset object to be tracked in subsequent video frames.

[0070] In some embodiments, the execution entity can determine the target precision location information of the preset object to be tracked in the subsequent video frames based on the feature similarity set. The target precision location information can be pixel-level or video frame block-level location information of the preset object to be tracked in the subsequent video frames, obtained by using different positioning methods based on the size of the object corresponding to the preset object to be tracked. The target precision location information can also be the location information of the preset object to be tracked on the subsequent video frames included in the video stream. The size can be the number of tokens covered by the preset object to be tracked in the target video frame. Figure 3 As shown, Figure 3 It shows the entire processing flow from acquiring the video stream to determining the fine-grained location information of the target. Figure 3 Whether an initialized target exists or not can be considered the process of target tracking for the first video frame in a video stream. Figure 3 The process involves tracking a target from an initialized target to a result, which can be a process for tracking targets in video frames other than the first video frame. Figure 3The process involves selecting three paths based on the visual feature quantity selection strategy, which correspond to the subsequent processing of the target precision location information by satisfying the first, second, and third preset target size conditions.

[0071] In some optional implementations of certain embodiments, determining the target precision location information of the preset object to be tracked in the subsequent video frames based on the feature similarity set may include the following steps: The first step involves, in response to determining that the number of visual features corresponding to the preset object-to-be-tracked information satisfies a second preset target size condition, selecting at least one feature similarity from the feature similarity set that satisfies a preset similarity threshold condition. The corresponding visual key-value feature information can be the visual vectors of video frame blocks corresponding to each video frame covered by the preset object-to-be-tracked information. For example... Figure 2 As shown, Figure 2 The diagram shows the number of visual vectors covering the video frame block in the image, as shown in the figure, which represents the preset information about the object to be tracked. Figure 2 The entire grid can represent subsequent video frames processed by the aforementioned large-scale visual language model for target tracking, while small squares can represent video frame blocks. The second preset target size condition can be a condition where the number of visual features is greater than or equal to a preset feature number threshold. This preset feature number threshold can be a pre-set value. For example, the preset feature number threshold can be 16. The preset similarity threshold condition can be a condition where the feature similarity is greater than or equal to a preset feature similarity threshold. This preset feature similarity threshold can be a pre-set critical value used to determine whether the corresponding visual key-value feature information needs to be selected. For example, the preset feature similarity threshold can be 0.7.

[0072] The second step involves generating a binary video frame mask for the subsequent video frames based on at least one of the aforementioned feature similarities. This binary video frame mask can be a two-dimensional mask image that uses 0s and 1s to represent partial information of a video frame block in the subsequent video frame, including information about a preset object to be tracked.

[0073] As an example, the aforementioned execution entity can first determine the video frame block set corresponding to the visual vector set of the video frame block corresponding to at least one feature similarity, using the pixel video block mapping information set and the pixel affine mapping information set included in the aforementioned mapping information set, as the subsequent video frame block set. Then, a binary mask with the values ​​of the subsequent video frame block set set being 1 and the values ​​of the remaining video frame blocks being 0 is determined on the aforementioned subsequent video frames as the video frame binary mask map. The aforementioned remaining video frame block set can be the set obtained by removing the subsequent video frame block set from the aforementioned subsequent video frames.

[0074] The third step involves spatial connectivity aggregation of the aforementioned video frame binary mask to obtain candidate target region information. This candidate target region information can be regions whose values ​​are 1 in the aforementioned video frame binary mask and whose included subsequent video frame blocks satisfy an 8-adjacent connectivity domain. In practice, the executing entity can utilize an 8-adjacent connectivity domain algorithm to perform spatial connectivity aggregation on the aforementioned video frame binary mask to obtain candidate target region information.

[0075] The fourth step is to determine the minimum bounding rectangle of the aforementioned candidate target region information, which will serve as the target's refined location information. In practice, the executing entity can utilize the ROI (Region of Interest) algorithm to determine the coordinates of the minimum bounding rectangle of the candidate target region information. The coordinates of this minimum bounding rectangle can be its position on the processed video frame corresponding to the subsequent video frame in the target video stream. These coordinates can be expressed as: .

[0076] in, This represents the minimum x-coordinate of the aforementioned minimum bounding rectangle in the processed video frame. This represents the maximum x-coordinate of the minimum bounding rectangle in the processed video frame. This represents the minimum ordinate of the aforementioned minimum bounding rectangle in the processed video frame. This represents the maximum ordinate of the minimum bounding rectangle in the processed video frame. This represents a two-dimensional index set of video frame block visual vector tokens corresponding to the aforementioned candidate target region information. Then, in response to the detection that the target tracking visual language large model satisfies the dynamic resolution support condition, the pixel affine mapping information set included in the mapping information set is used to determine the target's fine-grained position information in subsequent video frames. Finally, in response to the detection that the target tracking visual language large model does not satisfy the dynamic resolution support condition, the coordinate information of the minimum bounding rectangle is determined as the target's fine-grained position information.

[0077] In some optional implementations of certain embodiments, determining the target precision location information of the preset object to be tracked in the subsequent video frames based on the feature similarity set may include the following steps: The first step involves filtering the feature similarity set to obtain a filtered vector similarity index, in response to the determination that the number of visual features corresponding to the preset object information to be tracked meets the third preset target size condition. The third preset target size condition can be the condition that the number of visual features is 1. The filtered vector similarity can be the index of the largest similarity value in the token sequence within the feature similarity set. In practice, the executing entity can use a similarity index formula to filter the feature similarity set to obtain the filtered vector similarity index. The similarity index formula can be expressed as: .

[0078] in, This represents the vector similarity index after filtering. Indicates that When taking the maximum value The value is the index.

[0079] The second step involves determining the target fineness location information based on the aforementioned mapping information set and the aforementioned filtering vector similarity index. In practice, the executing entity can first perform reverse calculation using the pixel affine mapping information set included in the aforementioned mapping information set to determine the video frame block corresponding to the filtered vector similarity index in the processed video frame, which is then used as the processed video frame block. Then, in response to the detection that the target tracking visual language large model does not meet the dynamic resolution support condition, the location information corresponding to the aforementioned processed video frame block is determined as the target fineness location information. Finally, in response to the detection that the target tracking visual language large model meets the dynamic resolution support condition, reverse calculation is performed using the pixel video block mapping information set included in the aforementioned mapping information set to determine the video frame block corresponding to the processed video frame block in the aforementioned subsequent video block, which is then used as the target video frame block. The location information corresponding to the aforementioned target video frame block is then determined as the target fineness location information.

[0080] Optionally, the aforementioned preset object information to be tracked may be information on multiple target objects to be tracked.

[0081] After error code 104, the aforementioned executing entity can also perform the following steps: The first step involves constructing a template cache queue for each of the multiple target objects to be tracked, resulting in a template cache queue set. This template cache queue can be the frame cache queue mentioned above, used to store the visual key-value feature information set and the corresponding target object tracking template for each target object. For a detailed implementation of this step, please refer to [link to relevant documentation]. Figure 1 The specific implementation methods of steps 101 and 102 in the corresponding embodiments will not be described again here.

[0082] The second step involves parallel target tracking of the multiple target objects based on the aforementioned template cache queue set, yielding an initial bounding box information set. The initial bounding box information in this set can be the positional information of the bounding box used to select each target object. For a detailed implementation of this step, please refer to [reference needed]. Figure 1 The specific implementation methods of steps 103 and 104 in the corresponding embodiments will not be repeated here.

[0083] The third step involves applying bounding box suppression to the initial bounding box information set to obtain a set of detailed target location information for multiple objects to be tracked. In practice, the execution entity can utilize the NMS (Non-Maximum Suppression) algorithm to perform bounding box suppression on the initial bounding box information set to obtain a set of detailed target location information for multiple objects to be tracked. Therefore, without increasing additional inference computing power, parallel and simultaneous tracking of multiple targets across the entire video frame can be achieved, improving the efficiency of multi-target tracking and significantly reducing its duration.

[0084] Optionally, after error 104, the aforementioned executing entity may also perform the following steps: The first step involves updating the target object tracking template and frame buffer queue frame by frame based on the visual key-value feature information set corresponding to the target's fine-grained location information. This results in an updated target object tracking template and frame buffer queue, which are then used for target tracking on subsequent video frames. The frame buffer queue caches the visual key-value feature information set and its corresponding mapping information for a predetermined number of video frames preceding the subsequent video frames. This predetermined number can be a pre-set maximum number of video frames that the frame buffer queue can store. For example, the predetermined number could be 5 or 10. In practice, the execution entity can first use a template update formula to update the target object tracking template frame by frame based on the visual key-value feature information set corresponding to the target's fine-grained location information, resulting in an updated target object tracking template. The template update formula can be expressed as: .

[0085] in, This represents the attenuation coefficient, which can take a value of 0.9.

[0086] Then, in response to the detection that the frame buffer queue is not full, the visual key-value feature information set and the corresponding mapping information set corresponding to the target fineness location information are cached into the frame buffer queue. Finally, in response to the detection that the frame buffer queue is full, the visual key-value feature information set and the corresponding mapping information corresponding to the earliest cached video frame are deleted from the frame buffer queue, and the visual key-value feature information set and the corresponding mapping information set corresponding to the target fineness location information are cached into the frame buffer queue.

[0087] The second step, in response to the detection of the preset object to be tracked in subsequent video frames, determines the target's fine-grained position information based on the updated target object tracking template and the updated frame buffer queue. For a detailed implementation of this step, please refer to [link to relevant documentation]. Figure 1 The specific implementation methods of steps 103 and 104 in the corresponding embodiments will not be repeated here.

[0088] The third step involves determining the target's fine-grained position information based on the frame buffer queue and the target object tracking template, in response to the determination that no preset object information is detected in subsequent video frames. This "no preset object information detected in subsequent video frames" could mean that the preset object information is occluded or lost during tracking. In practice, the execution entity can, in response to the determination that no preset object information is detected in subsequent video frames, use a Kalman filter algorithm to determine the processed position information based on the frame buffer queue and the target object tracking template. Then, a reverse calculation is performed using the mapping information set to determine the original image position information of the processed position information in the subsequent video frames, which serves as the target's fine-grained position information. Therefore, by recording the complete coordinate mapping chain from the original image to the processed image, and then to the visual vector token index of the video frame block, it is expected that the content of this disclosure can be directly embedded into the VLM inference pipeline that supports dynamic resolution through the historical caching of a limited number of frames cached by the frame cache queue, and the memory usage is controllable. This can realize real-time target tracking scenarios for video streams, and the frame cache queue can maintain the stability and robustness of long-term tracking by caching historical information, which can prevent the target object tracking template mutation problem caused by target occlusion or instantaneous deformation.

[0089] Further reference Figure 4 The diagram illustrates flow 400 of some other embodiments of a visual key-vector-based target tracking method according to the present disclosure. This visual key-vector-based target tracking method includes the following steps: Step 401: Using the acquired target tracking visual language big model, the acquired video stream information is visually encoded to obtain a mapping information set and a visual key-value feature information set.

[0090] Step 402: Select hierarchical information based on the mapping information set, visual key value feature information set and preset model, and determine the target object tracking template for the preset object information to be tracked.

[0091] Step 403: Determine the target visual feature information set and the feature similarity set of the target object tracking template corresponding to the subsequent video frames.

[0092] In some embodiments, the specific implementation of steps 401-403 and the resulting technical effects can be found in [reference needed]. Figure 1 Steps 101-103 in the corresponding embodiments will not be repeated here.

[0093] Step 404: In response to determining that the number of visual features of the visual key value feature information corresponding to the preset object information to be tracked meets the first preset target size condition, object bounding box information for the preset object information to be tracked is generated.

[0094] In some embodiments, the execution entity may generate object bounding box information for the preset object to be tracked in response to determining that the number of visual vectors of the visual key-value feature information corresponding to the preset object to be tracked satisfies a first preset target size condition. The first preset target size condition may be a condition where the number of visual features is greater than 1 and less than or equal to a preset feature number threshold. The object bounding box information may be the position coordinate information of the determined bounding box at the video frame block level in the subsequent video frame for the preset object to be tracked. In practice, the execution entity may first select at least one feature similarity from the feature similarity set that satisfies a preset similarity threshold condition. Secondly, based on the at least one feature similarity, a video frame binary mask image for the subsequent video frame is generated. Then, spatial connected component aggregation is performed on the video frame binary mask image to obtain candidate target region information. Finally, based on the mapping information set, the minimum bounding rectangle of the candidate target region information is determined as the object bounding box information.

[0095] Step 405: Based on the video stream information and the target precision position information of the preset object to be tracked in the target video frame, determine the pixel-level position information of the preset object to be tracked in subsequent video frames.

[0096] In some embodiments, the execution entity can determine the pixel-level position information of the preset object to be tracked in the subsequent video frames based on the video stream information and the target precision position information of the preset object to be tracked in the target video frame. The target precision position information of the preset object to be tracked in the target video frame can be the position information of the tracked preset object to be tracked in the previous video frame of the subsequent video frames. The pixel-level position information can be the position information of the preset object to be tracked in the subsequent video frames represented in pixel-level format.

[0097] In some optional implementations of certain embodiments, determining the pixel-level position information of the preset object to be tracked in the subsequent video frames based on the video stream information and the target precision position information of the preset object to be tracked in the target video frame may include the following steps: In response to determining that motion vector information of the target video frame exists in the aforementioned video stream information, the pixel-level position information is determined based on the target's fine-grained position information on the target video frame and the aforementioned motion vector information, or the object bounding box information is determined as pixel-level position information. The aforementioned motion vector information can be the distance and directional displacement information of the coding block (macroblock) or coding unit of the subsequent video frame relative to the corresponding position block in the target video frame, obtained during the decoding process of the aforementioned video stream information. In practice, the executing entity can use a pixel-level position prediction formula to determine the pixel-level position information based on the target's fine-grained position information on the target video frame and the aforementioned motion vector information. The pixel-level position prediction formula can be expressed as: .

[0098] in, It represents pixel-level positional information. Indicates in The motion vector information. This indicates the target's precise location information within the target video frame.

[0099] In some optional implementations of certain embodiments, determining the pixel-level position information of the preset object to be tracked in the subsequent video frames based on the video stream information and the target precision position information of the preset object to be tracked in the target video frame may include the following steps: The first step, in response to the determination that motion vector information of the target video frame is absent from the aforementioned video stream information, is to determine the optical flow motion field information between the target video frame and the subsequent video frames. This optical flow motion field information can be a two-dimensional motion vector, i.e., velocity and direction information, between each pixel in the target video frame and the subsequent video frames. In practice, the executing entity can utilize an optical flow algorithm to determine the optical flow motion field information between the target video frame and the subsequent video frames.

[0100] The second step is to determine the pixel-level position information based on the target's detailed position information and the optical flow motion field information in the target video frame. In practice, the executing entity can use an optical flow position prediction formula to determine the pixel-level position information based on the target's detailed position information and the motion vector information in the target video frame. The optical flow position prediction formula can be a formula where the motion vector information in the pixel-level position prediction formula is replaced with optical flow motion field information.

[0101] Step 406: The pixel-level position information and the object bounding box information are fused to obtain the target fine-grained position information.

[0102] In some embodiments, the execution entity may perform a fusion process on the pixel-level position information and the object bounding box information to obtain target precision position information. This fusion process may be a fusion based on the union of the bounding boxes corresponding to the pixel-level position information and the bounding boxes corresponding to the object bounding box information, or a fusion based on a weighted average of the coordinates of the pixel-level position information and the object bounding box information.

[0103] from Figure 4 It can be seen from this that, with Figure 1 Compared to the description of some corresponding embodiments, Figure 4 The flowchart 400 of the target tracking method based on visual key-value vectors in some corresponding embodiments embodies the step of determining the target precision location information in the subsequent video frames based on the aforementioned feature similarity set. Thus, the schemes described in these embodiments, under the condition that the number of visual features meets the first preset target size, fuse the vector information (motion vector information or optical flow motion field information) between each pixel in the target video frame and subsequent video frames to achieve pixel-level precise alignment of the visual vectors of video frame blocks within a video block. This can bridge the gap between token-level tracking and pixel-level requirements for video frame block visual vectors, achieving pixel-level target tracking, improving the accuracy of target tracking location information determination, and making the bounding box of the tracked target precision location information more closely match the real contour of the object.

[0104] like Figure 5As shown, a target tracking device 500 based on visual key-value vectors includes: a visual encoding unit 501, a first determining unit 502, a second determining unit 503, and a third determining unit 504. The visual encoding unit 501 is configured to: use an acquired target tracking visual language large model to visually encode the acquired video stream information to obtain a mapping information set and a visual key-value feature information set. The first determining unit 502 is configured to: determine a target object tracking template for a preset object to be tracked based on the mapping information set, the visual key-value feature information set, and preset model selection level information. The second determining unit 503 is configured to: determine the target visual feature information set corresponding to subsequent video frames and the feature similarity set of the target object tracking template, wherein the subsequent video frames are any video frames located after the target video frame containing the preset object to be tracked, and the target visual feature information set is obtained based on the visual key-value feature information set corresponding to the subsequent video frames. The third determining unit 504 is configured to: determine the target precision position information of the preset object to be tracked in the subsequent video frames based on the feature similarity set.

[0105] It is understandable that the units described in the target tracking device 500 based on visual key vectors are related to the reference... Figure 1 The steps in the described method correspond accordingly. Therefore, the operations, features, and beneficial effects described above for the method are also applicable to the target tracking device 500 based on visual key-value vectors and the units contained therein, and will not be repeated here.

[0106] The following is for reference. Figure 6 It shows a schematic diagram of the structure of an electronic device (e.g., an electronic device) 600 suitable for implementing some embodiments of the present disclosure. Figure 6 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments of this disclosure.

[0107] like Figure 6 As shown, electronic device 600 may include a processing device (e.g., a central processing unit, a graphics processor, etc.) 601, which can perform various appropriate actions and processes according to a program stored in read-only memory (ROM) 602 or a program loaded from storage device 608 into random access memory (RAM) 603. RAM 603 also stores various programs and data required for the operation of electronic device 600. Processing device 601, ROM 602, and RAM 603 are interconnected via bus 604. Input / output (I / O) interface 605 is also connected to bus 604.

[0108] Typically, the following devices can be connected to I / O interface 605: input devices 606 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 607 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 608 including, for example, magnetic tapes, hard disks, etc.; and communication devices 609. Communication device 609 allows electronic device 600 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 6 An electronic device 600 with various devices is shown; however, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively. Figure 6 Each box shown can represent a device or multiple devices as needed.

[0109] In particular, according to some embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, some embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device 609, or installed from a storage device 608, or installed from a ROM 602. When the computer program is executed by the processing device 601, it performs the functions defined above in the methods of some embodiments of this disclosure.

[0110] It should be noted that, in some embodiments of this disclosure, the computer-readable medium described above may be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium may be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In some embodiments of this disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In some embodiments of this disclosure, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0111] In some implementations, clients and servers can communicate using any currently known or future-developed network protocol such as HTTP (Hypertext Transfer Protocol) and can interconnect with digital data communication (e.g., communication networks) of any form or medium. Examples of communication networks include local area networks (“LANs”), wide area networks (“WANs”), the Internet (e.g., the Internet of Things), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any currently known or future-developed networks.

[0112] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device. The aforementioned computer-readable medium carries one or more programs that, when executed by the electronic device, cause the electronic device to: employ an acquired target tracking visual language model to visually encode the acquired video stream information, obtaining a mapping information set and a visual key-value feature information set; determine a target object tracking template for a preset object to be tracked based on the mapping information set, the visual key-value feature information set, and preset model selection hierarchy information; determine the target visual feature information set corresponding to subsequent video frames and the feature similarity set of the target object tracking template, wherein the subsequent video frames are any video frames located after the target video frame containing the preset object to be tracked, and the target visual feature information set is obtained based on the visual key-value feature information set corresponding to the subsequent video frames; and determine the target precision position information of the preset object to be tracked in the subsequent video frames based on the feature similarity set.

[0113] Computer program code for performing operations of some embodiments of this disclosure can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0114] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0115] The units described in some embodiments of this disclosure can be implemented in software or hardware. The described units can also be housed in a processor; for example, a processor may be described as including a visual encoding unit, a first determining unit, a second determining unit, and a third determining unit. The names of these units do not necessarily limit the specific unit itself; for example, a visual encoding unit may also be described as "a unit that uses an acquired target tracking visual language large model to visually encode the acquired video stream information to obtain a mapping information set and a visual key-value feature information set."

[0116] The functions described above in this document can be performed at least in part by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), system-on-a-chip (SoCs), complex programmable logic devices (CPLDs), and so on.

[0117] The above description is merely a selection of preferred embodiments of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of the invention involved in the embodiments of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described inventive concept. For example, technical solutions formed by substituting the above-described features with (but not limited to) technical features with similar functions disclosed in the embodiments of this disclosure.

Claims

1. A target tracking method based on visual key-value vectors, characterized in that, include: The acquired target tracking visual language large model is used to visually encode the acquired video stream information to obtain a mapping information set and a visual key-value feature information set. Based on the mapping information set, the visual key value feature information set, and the preset model selection level information, a target object tracking template is determined for the preset object information to be tracked. Determine the target visual feature information set corresponding to the subsequent video frame and the feature similarity set of the target object tracking template, wherein the subsequent video frame is any video frame located after the target video frame where the preset object information to be tracked is located, and the target visual feature information set is obtained based on the visual key value feature information set corresponding to the subsequent video frame; Based on the feature similarity set, the target precision location information of the preset object to be tracked in the subsequent video frames is determined.

2. The method according to claim 1, characterized in that, The preset object information to be tracked consists of information about multiple target objects to be tracked. The method also includes: For each of the multiple target objects to be tracked, a template cache queue is constructed to obtain a template cache queue set; Based on the template cache queue set, parallel target tracking is performed on the multiple target objects to be tracked to obtain an initial bounding box information set; The initial bounding box information set is subjected to bounding box suppression processing to obtain a target precision location information set for multiple target objects to be tracked.

3. The method according to claim 1, characterized in that, The step of determining the target precision location information of the preset object to be tracked in the subsequent video frames based on the feature similarity set includes: In response to determining that the number of visual features of the visual key value feature information corresponding to the preset object information to be tracked meets the first preset target size condition, object bounding box information for the preset object information to be tracked is generated. Based on the video stream information and the target precision position information of the preset object to be tracked in the target video frame, determine the pixel-level position information of the preset object to be tracked in the subsequent video frames; The pixel-level position information and the object bounding box information are fused to obtain the target fine-grained position information.

4. The method according to claim 1, characterized in that, The step of determining the target precision location information of the preset object to be tracked in the subsequent video frames based on the feature similarity set includes: In response to determining that the number of visual features of the visual key value feature information corresponding to the preset object information to be tracked meets the second preset target size condition, at least one feature similarity that meets the preset similarity threshold condition is selected from the feature similarity set. Based on the at least one feature similarity, generate a video frame binary mask image for the subsequent video frames; Spatial connected component aggregation is performed on the binary mask image of the video frame to obtain candidate target region information; The minimum bounding rectangle of the candidate target region information is determined as the target precision location information.

5. The method according to claim 1, characterized in that, The step of determining the target precision location information of the preset object to be tracked in the subsequent video frames based on the feature similarity set includes: In response to determining that the number of visual features of the visual key value feature information corresponding to the preset object information to be tracked meets the third preset target size condition, the feature similarity set is filtered to obtain the filtered vector similarity index. The target precision location information is determined based on the mapping information set and the filtering vector similarity index.

6. The method according to claim 1, characterized in that, The method further includes: Based on the visual key value feature information set corresponding to the target precision location information, the target object tracking template and frame buffer queue are updated frame by frame to obtain the updated target object tracking template and the updated frame buffer queue, so as to perform target tracking on subsequent video frame sets. The frame buffer queue is a queue that caches the visual key value feature information set and corresponding mapping information of a preset number of video frames located before the subsequent video frames. In response to determining that preset object information to be tracked is detected in subsequent video frames, the target fine-grained position information is determined according to the updated target object tracking template and the updated frame buffer queue; In response to determining that no preset object information is detected in subsequent video frames, the target precision position information is determined based on the frame buffer queue and the target object tracking template.

7. The method according to claim 3, characterized in that, The step of determining the pixel-level position information of the preset object to be tracked in the subsequent video frames based on the video stream information and the target precision position information of the preset object to be tracked in the target video frame includes: In response to determining that motion vector information of the target video frame exists in the video stream information, the pixel-level position information is determined based on the target precision position information on the target video frame and the motion vector information, or the object bounding box information is determined as pixel-level position information.

8. The method according to claim 3, characterized in that, The step of determining the pixel-level position information of the preset object to be tracked in the subsequent video frames based on the video stream information and the target precision position information of the preset object to be tracked in the target video frame includes: In response to determining that there is no motion vector information of the target video frame in the video stream information, optical flow motion field information between the target video frame and the subsequent video frame is determined; The pixel-level position information is determined based on the target precision position information on the target video frame and the optical flow motion field information.

9. The method according to claim 1, characterized in that, The method employs the acquired target tracking visual language large model to perform visual encoding on the acquired video stream information, obtaining a mapping information set and a visual key-value feature information set, including: In response to the detection that the target tracking visual language big model meets the dynamic resolution support condition, a pixel affine mapping information set of the video stream information and the target video stream information is determined, wherein the target video stream information is video stream information that meets the input requirements of the target tracking visual language big model; Using the target tracking visual language big model, each video frame included in the target video stream information is visually encoded to obtain a video frame block set, a video frame block visual vector set, and a visual key value vector set; Determine the pixel video block mapping information set of the video frame block group set and the video frame block visual vector set; Determine the key-value video block mapping information set of the video frame block visual vector set and the visual key-value vector set; The pixel affine mapping information set, the pixel video block mapping information set, and the key-value video block mapping information set are determined as a mapping information set, and the video frame block visual vector set and the visual key-value vector set are determined as a visual key-value feature information set. The mapping information set and the visual key-value vector set are then cached in the frame buffer queue.

10. The method according to claim 1, characterized in that, The step of determining a target object tracking template for a preset object to be tracked based on the mapping information set, the visual key-value feature information set, and the preset model selection hierarchy information includes: In response to the detection that the target tracking visual language large model meets the dynamic resolution support condition, at least one visual key value feature information corresponding to the preset object information to be tracked is extracted from the visual key value feature information set according to the pixel affine mapping information set and the pixel video block mapping information set included in the mapping information set, and used as the target visual key value feature information set. Based on the key-value video block mapping information set included in the mapping information set and the preset model selection level information, the target visual key-value feature information set is fused into vectors to obtain the target object tracking template.

11. The method according to claim 1, characterized in that, The step of determining the target visual feature information set corresponding to subsequent video frames and the feature similarity set of the target object tracking template includes: The target visual feature information set is determined based on the visual key value feature information corresponding to the subsequent video frames; For each target visual feature, perform the following weighted fusion step: Determine the similarity between the target visual feature information and the target object tracking template at the selected hierarchical information of the preset model to obtain a similarity set; The similarity set is weighted and fused to obtain the feature similarity.

12. A target tracking device based on visual key-value vectors, characterized in that, include: The visual encoding unit is configured to use the acquired target tracking visual language big model to visually encode the acquired video stream information to obtain a mapping information set and a visual key-value feature information set. The first determining unit is configured to determine a target object tracking template for preset object information based on the mapping information set, the visual key value feature information set and the preset model selection hierarchical information. The second determining unit is configured to determine the target visual feature information set corresponding to the subsequent video frame and the feature similarity set of the target object tracking template, wherein the subsequent video frame is any video frame located after the target video frame where the preset object information to be tracked is located, and the target visual feature information set is obtained based on the visual key value feature information set corresponding to the subsequent video frame. The third determining unit is configured to determine the target precision location information of the preset object to be tracked in the subsequent video frames based on the feature similarity set.

13. An electronic device, characterized in that, include: One or more processors; Storage device, on which one or more programs are stored, When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1-11.

14. A computer-readable medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1-11.

Images