A spatiotemporal label merging method and device based on semantic guidance, equipment and medium

Abstract

The application discloses a space-time label merging method and device based on semantic guidance, equipment and medium, the method comprises the semantic correlation of acquiring visual label grid and video level semantic descriptor; the gradient vector of the semantic correlation relative to the visual label is obtained, and the semantic contribution of the visual label to the original video is determined based on the gradient vector; based on the semantic contribution and visual redundancy, merging is carried out in the spatial dimension; directional merging is carried out in the semantic stable window. The application determines the semantic contribution of the visual label to the original video by using the video level semantic descriptor, carries out spatial merging according to the semantic contribution and visual redundancy, carries out directional merging in the semantic stable window, so that the obtained compressed representation can not only significantly reduce the pre-filling overhead of long video, but also can retain the key semantic evidence valuable for multi-round question and answer, so that the compressed video prefix obtained by the first round of pre-filling can be directly reused by subsequent multi-round questions, and the repeated pre-filling calculation amount and multi-round interaction delay are reduced.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a method, apparatus, device and medium for merging spatiotemporal tags based on semantic guidance. Background Technology

[0002] Current video-based large language models perform well in video understanding and question answering tasks. These models typically use high-dimensional long sequence tags generated by visual encoding a large number of video frames as input data. However, inputting these high-dimensional long sequence tags into the language model incurs significant computational and storage overhead. This leads to limitations in practical deployment of video-based large language models, including high inference latency, high cost of constructing prefixes for long videos, and severe repetitive computation issues when asking multiple questions from the same video. These problems are particularly pronounced in long-video, multi-turn question answering scenarios, where not only is a complete video context needed for the first round of questions, but the need to handle repeated calls to subsequent rounds of questions also arises.

[0003] When using video large language models to perform video understanding and video question answering tasks, the visual tags output by the video encoder are typically compressed during the pre-filling stage to reduce computational and storage overhead. Currently, commonly used compression methods are mainly divided into two categories: query-aware compression methods oriented towards single questions, and purely visual, question-independent compression methods. Query-aware compression methods oriented towards single questions require recalculating retention or merging strategies for each question, leading to instability in the video prefix representation and hindering the reuse of key-value caches across rounds. While purely visual, question-independent compression methods produce more stable compression results and naturally support reuse, they rely solely on visual redundancy for compression decisions, easily resulting in the loss of semantic evidence crucial for downstream inference at high compression rates.

[0004] Therefore, existing technologies still need to be improved and enhanced. Summary of the Invention

[0005] The technical problem to be solved by this application is to provide a semantically guided spatiotemporal tag merging method, apparatus, device and medium to address the shortcomings of the existing technology.

[0006] To address the aforementioned technical problems, the first aspect of this application provides a semantically guided spatiotemporal tag merging method, wherein the semantically guided spatiotemporal tag merging method specifically includes: Obtain the visually labeled grid and video-level semantic descriptor of the original video; Obtain the semantic correlation between the visual marker grid and the video-level semantic descriptor; Obtain the gradient vector of the semantic relevance relative to each visual marker in the visual marker grid, and determine the semantic contribution of the visual marker to the original video based on the gradient vector; Based on the semantic contribution and visual redundancy, the visual marker grid is merged in the spatial dimension to obtain spatial visual markers; Obtain the semantically stable window of the original video, and perform directed merging of the spatial visual tags within the semantically stable window to determine the spatiotemporally merged visual tags.

[0007] The semantically guided spatiotemporal tagging merging method described above, wherein the process of obtaining the video-level semantic descriptor specifically includes: The original video is input into a video big language model, and the video big language model performs lightweight semantic generalization on the original video to obtain a video-level semantic descriptor, wherein the video-level semantic descriptor is used to reflect the video content overview of the original video.

[0008] The semantically guided spatiotemporal tagging merging method, wherein obtaining the semantic relevance between the visual tagging grid and the video-level semantic descriptor specifically includes: The video-level semantic descriptor is encoded into a semantic embedding, and video-level visual features are generated based on the visual marker grid. Obtain the semantic similarity between the semantic embedding and the video-level visual features to obtain semantic relevance.

[0009] The semantically guided spatiotemporal tag merging method, wherein determining the semantic contribution of visual tags to the original video based on the gradient vector specifically includes: The channel importance vector is determined based on the gradient vector of all visual markers; The visual marker is inner-producted with the channel importance vector to obtain the response value of the visual marker along the video channel direction; The semantic contribution of the visual markers to the original video is determined based on the response values.

[0010] The semantically guided spatiotemporal marker merging method, wherein merging the visual marker grid in the spatial dimension based on the semantic contribution and visual redundancy specifically includes: Using the root node in the quadtree as the parent node, the semantic relevance score of each child node corresponding to the parent node is obtained. The semantic relevance score is determined based on the semantic contribution of all visual markers within the spatial region corresponding to the child node. Obtain the visual redundancy between the parent node and each of its corresponding child nodes; Based on the semantic relevance score of each child node and the visual redundancy between each child node and its parent node, the joint judgment score of each child node is determined. The parent node is determined to meet the preset refinement conditions based on the joint judgment score of each child node. If the parent node does not meet the preset refinement conditions, the refinement of the parent node will be stopped, and the parent node will be retained as the visual marker of its corresponding spatial region. If the parent node meets the preset refinement conditions, the parent node is refined into its four corresponding child nodes, and each child node is taken as the parent node. The step of obtaining the semantic relevance score of each child node corresponding to the parent node is continued until all quadtree branches in the quadtree meet the preset refinement conditions or reach the finest level of the quadtree.

[0011] The semantically guided spatiotemporal tagging merging method, wherein determining the joint judgment score of each child node based on the semantic relevance score of each child node and the visual redundancy between each child node and its parent node specifically includes: Obtain the semantic weight coefficient corresponding to the semantic relevance score and the redundancy weight coefficient determined by the visual redundancy; A first judgment score is determined based on the semantic weight coefficient and the semantic relevance score, and a second judgment score is determined using the redundancy weight coefficient and the visual redundancy. The difference between the first judgment score and the second judgment score is obtained to obtain the joint judgment score.

[0012] The semantically guided spatiotemporal tag merging method, wherein determining whether the parent node meets the preset refinement conditions based on the joint judgment score of each child node specifically includes: The joint decision score of each child node is compared with the spatial refinement threshold; If the joint decision score of all child nodes is less than or equal to the spatial refinement threshold, the decision does not meet the preset refinement condition. If at least one child node has a joint decision score greater than the spatial refinement threshold, the preset refinement condition is satisfied.

[0013] The semantically guided spatiotemporal tagging merging method described above, wherein obtaining the semantically stable window of the original video specifically includes: Global visual embedding of each video frame in the original video; Obtain the cosine correlation score between the global visual embedding and the video-level semantic descriptor to form a time-ordered frame-level correlation sequence; Based on the frame-level correlation sequence, the semantic correlation score between adjacent video frames is determined, and the original video is divided into several consecutive semantically stable windows according to the semantic correlation score.

[0014] The semantically guided spatiotemporal marker merging method, wherein the directed merging of the spatial visual markers within the semantically stable window specifically includes: For two adjacent video frames in the semantically stable window, obtain spatial visual label pairs in which there are overlapping spatial regions in both adjacent video frames. Obtain the visual similarity of spatial visual tag pairs, and establish directed links between two spatial visual tags in the spatial visual tag pairs based on the visual similarity. Spatial visual tags formed through directed links are grouped into the same spatial visual tag group to obtain several spatial visual tag groups; The earliest spatial visual mark in each spatial visual mark group is taken as the representative mark, and the other spatial visual marks in each spatial visual mark group are merged into the representative mark of each spatial visual mark group.

[0015] A second aspect of this application provides a semantically guided spatiotemporal tag merging device, wherein the semantically guided spatiotemporal tag merging device specifically includes: Acquisition module. Used to acquire the visually labeled grid and video-level semantic descriptor of the raw video; A semantic relevance acquisition module is used to acquire the semantic relevance between the visual marker grid and the video-level semantic descriptor; A semantic contribution determination module is used to obtain the gradient vector of the semantic relevance relative to each visual marker in the visual marker grid, and determine the semantic contribution of the visual marker to the original video based on the gradient vector; A spatial merging module is used to merge the visual marker grid in the spatial dimension based on the semantic contribution and visual redundancy to obtain spatial visual markers; The temporal merging module is used to obtain the semantically stable window of the original video and perform directional merging of the spatial visual tags in the semantically stable window to determine the spatiotemporally merged visual tags.

[0016] A third aspect of this application provides a computer-readable storage medium storing one or more programs that can be executed by one or more processors to implement the steps in the semantically guided spatiotemporal tag merging method as described above.

[0017] A fourth aspect of this application provides a terminal device, which includes: a processor and a memory; The memory stores a computer-readable program that can be executed by the processor; When the processor executes the computer-readable program, it implements the steps in any of the semantically guided spatiotemporal tag merging methods described above.

[0018] Beneficial effects: 1. This application utilizes the video-level semantic descriptor of the original video to determine the semantic contribution of visual tags to the original video, and performs spatial merging based on semantic contribution and visual redundancy. Then, it performs cross-frame directed merging within the semantically stable window, so that the resulting compressed representation can significantly reduce the pre-filling overhead of long videos, retain key semantic evidence valuable for multi-round question answering, and ensure the stability and determinism of the compressed representation on the same video. This allows the compressed video prefix obtained in the first round of pre-filling to be directly reused in subsequent rounds of questions, reducing the computational cost of repeated pre-filling and the latency of multi-round interaction.

[0019] 2. The video-level semantic descriptor in this application is a holistic description of the original video. It can characterize the overall theme, main objects, key actions, event stages, significant textual information, and potential question distribution of the original video. While maintaining the stability of the same video compression result and supporting cache reuse, it introduces global semantic priors related to downstream question-answering evidence, avoiding the loss of semantic information caused by relying solely on visual redundancy judgment. It can still retain key semantic information valuable to downstream reasoning tasks under high compression rates, significantly improving the usability of compressed visual tags.

[0020] 3. This application uses a quadtree hierarchical spatial merging strategy to combine semantic contribution and visual redundancy to determine the merging of spatial visual labels. During the merging process, it can adaptively retain fine regions with rich semantic information and simplify regions with high visual redundancy and low semantic contribution, thereby compressing the input length while avoiding the loss of a large amount of key semantic information.

[0021] 4. This application performs directed merging in the time dimension using semantically stable windows as units. It merges overlapping spatial markers with high visual similarity only within continuous windows with stable semantic content. This not only further reduces the length of visual markers but also ensures that key markers at semantic stage transitions are fully preserved, thereby further improving the semantic integrity of the compression result. Attached Figure Description

[0022] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0023] Figure 1 A flowchart of a semantically guided spatiotemporal tag merging method provided in an embodiment of this application.

[0024] Figure 2 The overall architecture diagram of the semantically guided spatiotemporal tag merging method provided in the embodiments of this application is shown.

[0025] Figure 3 This is a schematic diagram of spatial dimension merging.

[0026] Figure 4 A schematic diagram of the visualization results for an example of spatial dimension merging.

[0027] Figure 5 A schematic diagram of the visualization results for another example of spatial dimension merging.

[0028] Figure 6 This is a schematic diagram of the visualization results of an example of semantically guided spatiotemporal tag merging.

[0029] Figure 7 A schematic diagram of the semantically guided spatiotemporal tag merging device provided in the embodiments of this application.

[0030] Figure 8 A schematic block diagram of the terminal device provided in the embodiments of this application. Detailed Implementation

[0031] This application provides a method, apparatus, device, and medium for merging spatiotemporal markers based on semantic guidance. To make the objectives, technical solutions, and effects of this application clearer and more explicit, the following detailed description is provided with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only for explaining this application and are not intended to limit this application.

[0032] Those skilled in the art will understand that, unless specifically stated otherwise, the singular forms “a,” “an,” “the,” and “the” used herein may also include the plural forms. It should be further understood that the term “comprising” as used in this application means the presence of the stated features, integers, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. It should be understood that when we say an element is “connected” or “coupled” to another element, it can be directly connected or coupled to the other element, or there may be intermediate elements. Furthermore, “connected” or “coupled” as used herein can include wireless connections or wireless coupling. The term “and / or” as used herein includes all or any units and all combinations of one or more associated listed items.

[0033] It will be understood by those skilled in the art that, unless otherwise defined, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains. It should also be understood that terms such as those defined in general dictionaries should be understood to have the same meaning as in the context of the prior art, and should not be interpreted in an idealized or overly formal sense unless specifically defined as herein.

[0034] It should be understood that the sequence number and size of each step in this embodiment do not imply the order of execution. The execution order of each process is determined by its function and internal logic, and should not constitute any limitation on the implementation process of this application embodiment.

[0035] Research has revealed that current video-based large language models perform exceptionally well in video understanding and question-answering tasks. These models typically use high-dimensional long sequence tags generated from visual encoding of a large number of video frames as input data. However, inputting these high-dimensional long sequence tags into the language model incurs significant computational and storage overhead. This leads to limitations in practical deployment of video-based large language models, including high inference latency, high costs of constructing prefixes for long videos, and severe repetitive computation issues when asking multiple questions from the same video. These problems are particularly pronounced in long-video, multi-turn question-answering scenarios, where not only is a complete video context needed for the first round of questions, but the need to accommodate repeated calls to subsequent rounds of questions also exacerbates these issues.

[0036] When using video large language models to perform video understanding and video question answering tasks, the visual tags output by the video encoder are typically compressed during the pre-filling stage to reduce computational and storage overhead. Currently, commonly used compression methods are mainly divided into two categories: query-aware compression methods oriented towards single questions, and purely visual, question-independent compression methods. Query-aware compression methods oriented towards single questions require recalculating retention or merging strategies for each question, leading to instability in the video prefix representation and hindering the reuse of key-value caches across rounds. While purely visual, question-independent compression methods produce more stable compression results and naturally support reuse, they rely solely on visual redundancy for compression decisions, easily resulting in the loss of semantic evidence crucial for downstream inference at high compression rates.

[0037] It is understandable that existing purely visual, question-independent compression methods can perform spatiotemporal compression on the visual tags output by the visual encoder, provided the model is retrained. However, these methods rely on local visual similarity to perform coarse-to-fine quadtree spatial merging for each frame, and then on directed pairwise merging in the temporal dimension based on visual tag similarity. They depend solely on visual redundancy for compression decisions. In other words, the merging basis of this method is visual feature similarity and spatiotemporal locality. Its "question-independent" nature is intended to support key-value cache reuse between different questions within the same video, ignoring the guidance of global semantic information in the merging process. It cannot distinguish the differences in semantic value of different local regions for downstream question-answering tasks. Therefore, at higher compression rates, it is prone to mistakenly merging regions with high semantic value but also high visual redundancy, ultimately resulting in the loss of crucial semantic evidence and affecting the accuracy of downstream video question-answering inference.

[0038] To this end, in this embodiment, a visual marker grid and a video-level semantic descriptor of the original video are obtained; the semantic correlation between the visual marker grid and the video-level semantic descriptor is obtained; the gradient vector of the semantic correlation relative to each visual marker in the visual marker grid is obtained, and the semantic contribution of the visual marker to the original video is determined based on the gradient vector; the visual marker grid is merged in the spatial dimension based on the semantic contribution and visual redundancy to obtain spatial visual markers; a semantic stability window of the original video is obtained, and the spatial visual markers are merging in the semantic stability window to determine the spatiotemporally merged visual markers. This embodiment utilizes the video-level semantic descriptor of the original video to determine the semantic contribution of visual markers to the original video, performs spatial merging based on semantic contribution and visual redundancy, and then performs cross-frame directed merging within the semantic stability window. This results in a compressed representation that significantly reduces the pre-filling overhead of long videos while retaining key semantic evidence valuable for multi-round question answering. Simultaneously, it ensures the stability and determinism of the compressed representation on the same video, allowing the compressed video prefix obtained in the first round of pre-filling to be directly reused in subsequent rounds of questions, reducing the computational load of repeated pre-filling and the latency of multi-round interactions.

[0039] It should be noted that the executing entity of each embodiment of this application can be a computing service system with data processing, network communication, and program execution functions, such as an electronic system capable of implementing the above functions, a semantically guided spatiotemporal tag merging system, etc. The following uses a semantically guided spatiotemporal tag merging system (hereinafter referred to as "the system") as an example to describe the following embodiments.

[0040] The application content will be further explained below with reference to the accompanying drawings and through the description of the embodiments.

[0041] This embodiment provides a semantically guided spatiotemporal tag merging method for execution during the pre-filling stage of a large video language model. For example... Figure 1 and Figure 2 As shown, the semantically guided spatiotemporal tag merging method specifically includes: S10. Obtain the visual marker grid and video-level semantic descriptor of the original video.

[0042] Specifically, the original video can be a long input video to be processed. Before processing, the original video needs to be preprocessed to extract the visual features of all video frames and arrange them according to spatial dimensions to generate a visual marker grid. The visual marker grid includes the spatial grid-like visual features of each video frame in the original video. Each spatial grid-like visual feature includes multiple spatial blocks of a preset size. Each spatial block corresponds to a visual marker, and each visual marker corresponds to a local spatial region in the video frame. For example, suppose the original video is... , the original video The input is a visual encoder, which extracts the spatial grid-like visual features of each video frame to obtain the visually labeled grid of the original video. ,in, Indicates the number of frames in the time dimension. Indicates the height of the visual marker grid in a single frame. This indicates the width of the visual marker grid in a single frame. This represents the dimension of the visual marker features.

[0043] The video-level semantic descriptor is a summary of the original video content, which stabilizes the spatiotemporal tagging process of visual markers. It can be understood that this video-level semantic descriptor does not answer a specific question, but rather represents a semantic summary that characterizes the overall theme, main objects, key actions, event stages, significant textual information, and potential question distribution of the video. In other words, the video-level semantic descriptor in this application differs from the question signal in query-aware compression, which involves "one round of question-driven filtering." Its generation process is an offline preprocessing step unrelated to subsequent question-answering tasks. It only needs to be executed once, and once the result is fixed, it can be repeatedly used in subsequent steps without changing with different questions.

[0044] In one embodiment, the process of obtaining the video-level semantic descriptor specifically includes: The original video is input into the video big language model, and the video big language model performs lightweight semantic generalization on the original video to obtain a video-level semantic descriptor.

[0045] Specifically, the video big language model performs lightweight semantic summarization on the input original video to obtain a semantic summary, which in turn produces a video-level semantic descriptor. The entire generation process is completed before the original video enters the pre-filling stage. The generated video-level semantic descriptor can be stored as a fixed global feature and does not need to be recalculated in subsequent rounds of question answering.

[0046] It should be noted that in practical applications, when the length of the original video is less than the preset length, the original video can be directly used for semantically guided spatiotemporal tagging merging; when the length of the original video is greater than or equal to the preset length, frames can be extracted at a fixed frame rate and a maximum frame count limit can be set (e.g., one frame per second, up to 128 frames, etc.) before semantically guided spatiotemporal tagging merging is performed.

[0047] S20. Obtain the semantic correlation between the visual marker grid and the video-level semantic descriptor.

[0048] Specifically, semantic relevance characterizes the degree of consistency between the overall content expressed by the visually labeled grid and the video-level semantic descriptor. In other words, semantic relevance is the degree of match between the video content expressed by all visual tags in the visually labeled grid and the overall semantic content of the original video. A higher semantic relevance value indicates a higher degree of consistency between the overall content expressed by the visually labeled grid and the video-level semantic descriptor; conversely, a lower semantic relevance value indicates a lower degree of consistency.

[0049] In one embodiment, obtaining the semantic relevance between the visual marker grid and the video-level semantic descriptor specifically includes: The video-level semantic descriptor is encoded into a semantic embedding, and video-level visual features are generated based on the visual marker grid. Obtain the semantic similarity between the semantic embedding and the video-level visual features to obtain semantic relevance.

[0050] Specifically, semantic embedding is the embedding representation obtained by encoding video-level semantic descriptors, and this semantic embedding can be represented as: , in, Represents a video-level semantic descriptor. This represents the semantic embedding corresponding to the video-level semantic descriptor. This represents a semantic encoder.

[0051] Video-level visual features are aggregated features obtained by performing an average pooling operation on all visual tags in the visual tagging mesh. These aggregated features can characterize the global visual content information contained in the entire visual tagging mesh. For example, for a visual tagging mesh... The visual marker grid is flattened along the time and spatial dimensions to obtain the visual marker feature sequence. , Indicates the number of visual markers. Indicates the first If a visual feature is identified by a visual marker, then the video-level visual feature can be represented as: , in, Represents video-level visual features. This represents the projection function, which projects the average pooled features into the same embedding space as the video-level semantic descriptor.

[0052] Next, after obtaining the semantic embedding and video-level visual features, the semantic similarity (e.g., cosine similarity) between the video-level visual features and the video-level semantic descriptor (i.e., semantic embedding) is calculated to obtain semantic relevance. This semantic relevance reflects the degree of matching between the global visual content and the global semantic generalization. This semantic relevance can be expressed as: , in, Indicates semantic relevance.

[0053] It should be noted that after obtaining the video-level semantic descriptor, it can be directly encoded into a semantic embedding. This allows the video-level semantic descriptor to participate in subsequent processing in the form of an embedding (i.e., the semantic embedding corresponding to the video-level semantic descriptor can be directly called when determining semantic relevance). This ensures consistency across all subsequent issues, thereby guaranteeing the stability of the merging result and meeting the requirements for key-value caching reuse. Furthermore, in practical applications, to ensure computational efficiency, the dimension of the video-level semantic descriptor can be kept consistent with the feature dimension of a single visual tag (i.e., after obtaining the average pooling features, no projection operation is needed). This avoids the additional alignment computation overhead caused by dimension mismatch and also facilitates the subsequent direct calculation of the semantic relevance between the visual tag and the video-level semantic descriptor.

[0054] In addition to the above methods, other methods can be used to obtain semantic relevance in practical applications. For example, the inner product of each visual tag in the visual tag grid with the video-level semantic descriptor can be calculated, and the mean of all inner product results can be used as the semantic relevance.

[0055] S30. Obtain the gradient vector of the semantic relevance relative to each visual marker in the visual marker grid, and determine the semantic contribution of the visual marker to the original video based on the gradient vector.

[0056] Specifically, the gradient vector is used to measure the influence of each feature channel of the visual tag on the semantic objective function. For a feature channel of the visual tag, if the absolute value of the gradient of the feature channel is large, it means that when the feature channel changes, the consistency between the video-level visual aggregate features and the semantic descriptor embedding will change significantly, that is, the feature channel is more important to the overall semantics of the current video. Conversely, if the absolute value of the gradient of the feature channel is small, it means that when the feature channel changes, the consistency between the video-level visual aggregate features and the semantic descriptor embedding will change less, that is, the feature channel is not important to the overall semantics of the current video.

[0057] The gradient vector can be represented as: , , in, This represents the gradient sequence of visual marker features formed by the semantic relevance of the visual marker grid. Represents a sequence of visual marker features. Indicates semantic relevance. Represents a video-level semantic descriptor. Indicates the first The gradient vector of each visual marker, Indicates the first The visual features of a visual marker.

[0058] It should be noted that this gradient vector can be calculated using the automatic differentiation mechanism or backpropagation mechanism of the native video large language model. Furthermore, during the calculation process, all parameters of the visual encoder, semantic encoding module, and video large language model remain unchanged; only those related to semantic relevance are retained. To visual marker feature sequence The computation graph is constructed, and a backpropagation is performed on the semantic relevance.

[0059] Furthermore, in step S30 above, the semantic contribution is used to characterize the positive contribution of visual tags to the overall semantics of the original video. The larger the semantic contribution, the higher the positive contribution of the visual tags to the overall semantics of the original video; conversely, the smaller the semantic contribution, the lower the positive contribution of the visual tags to the overall semantics of the original video. For example, for a video of "a person cutting vegetables in the kitchen and putting them into a pot," the video-level semantic descriptor can summarize high-level content such as the person, the kitchen, cutting vegetables, vegetables, knives, and pots. Visual tags representing the person's hands, knives, vegetables, and pots have a larger positive contribution to the above semantic objective function, while visual tags representing irrelevant backgrounds have a relatively lower positive contribution.

[0060] In one embodiment, determining the semantic contribution of the visual markers to the original video based on the gradient vector specifically includes: The channel importance vector is determined based on the gradient vector of all visual markers; The visual marker is inner-producted with the channel importance vector to obtain the response value of the visual marker along the video channel direction; The semantic contribution of the visual markers to the original video is determined based on the response values.

[0061] Specifically, the channel importance vector is the global channel weight obtained by aggregating all gradient vectors. It can reflect the importance of different visual feature channels to the overall semantic expression of the video. By aggregating the gradients of all visual markers, the influence of local fluctuations of individual markers can be reduced, and a more stable global semantic channel importance can be obtained. Each element of the channel importance vector corresponds to the weight of a visual feature channel. The larger the weight, the more critical the corresponding visual feature channel is to the overall semantic expression.

[0062] For example, by performing global average pooling on the gradient vectors corresponding to all visual markers along the feature dimension, the channel importance vector can be obtained, which can be represented as: , in, This represents the channel importance vector.

[0063] Next, after obtaining the channel importance vector, the visual features of each visual tag are compared with the channel importance vector. Performing the inner product yields the response value of the visual tag along the video channel direction. This response value corresponds to the activation level of the key semantic channels within the visual tag. This response value can be represented as... ,in, Indicates the first The response value of a visual marker, if Explanation of the first A visual marker has a positive effect on video-level semantic objectives, if Explanation of the first The first visual tag makes a positive contribution to the video-level semantic objective, or the second visual tag makes a positive contribution to the video-level semantic objective. A visual marker has a reverse effect on video-level semantic targets.

[0064] Since the semantic contribution in this application is used to characterize the positive contribution of the visual tag to the overall semantics of the original video, after obtaining the response value of the visual tag along the video channel direction, the response value can be gated to retain only the positive effect of the visual tag on the overall semantics of the video. For example, a linear rectified function can be used to process the response value of the visual tag along the video channel direction to obtain the semantic contribution of the visual tag, which can be expressed as: , in, Indicates the first The semantic contribution of each visual tag Represents the linear rectified function. This represents the maximum value function.

[0065] This application embeds video-level semantic descriptors into a representation space that can be compared with video-level visual features (used as a visual summary of the original video). It constructs a similarity target (i.e., semantic relevance) between the semantic embeddings corresponding to the video-level visual features and the video-level semantic descriptors, and calculates the gradient of this semantic relevance with respect to the visual markers. This yields the contribution of the spatial region corresponding to each visual marker to the overall semantics of the video. This allows the overall semantic information of the video to be introduced into the compression decision process, no longer relying solely on local visual similarity for merging decisions, thus solving the problem of existing methods ignoring global semantic value differences.

[0066] S40. Based on the semantic contribution and visual redundancy, the visual marker grid is merged in the spatial dimension to obtain spatial visual markers.

[0067] It is understandable that when merging the visual marker grid in the spatial dimension, both the visual redundancy between visual markers and the contribution of visual markers to the overall semantics of the video are considered. Visual marker merging is only performed when the visual redundancy requirement is met and the risk of semantic preservation is low. For visual markers with high semantic contribution (such as visual markers corresponding to high semantic value areas such as human faces, hand gestures, key objects, text, instruments, and event triggering areas), even if the visual markers are visually similar to the background or their adjacent areas (i.e., high visual redundancy), high semantic value visual markers are preferred to be retained. This reduces the risk of key semantic evidence being averaged out by the background or repetitive areas under high compression rates. Thus, while ensuring compression effect, it avoids the incorrect merging of high semantic value areas and retains key information useful for downstream question answering tasks to the greatest extent.

[0068] In one embodiment, when merging the visual marker grids in the spatial dimension, a quadtree merging method can be used, that is, for each video frame's marker grid in the original video... A multi-level quadtree is constructed. The finest level of this multi-level quadtree consists of visual tags of video frames; when moving up one level, adjacent frames are... The visual marker block is downsampled into a coarser-grained node; this process is repeated to obtain coarser spatial levels layer by layer. Each coarse-grained parent node is connected to its four child nodes at the next level. This multi-level quadtree enables the embodiments of this application to adaptively select the granularity of retention for each region in a coarse-to-fine manner.

[0069] Based on this, the merging of the visual marker grid in the spatial dimension based on the semantic contribution and visual redundancy specifically includes: S41. Using the root node in the quadtree as the parent node, obtain the semantic relevance score of each child node corresponding to the parent node, wherein the quadtree is constructed based on the labeled grid of the video frame; S42. Obtain the visual redundancy between the parent node and each child node corresponding to the parent node; S43. Determine the joint judgment score of each child node based on the semantic relevance score of each child node and the visual redundancy between each child node and the parent node. S44. Determine whether the parent node meets the preset refinement conditions based on the joint judgment score of each child node; S45. If the parent node does not meet the preset refinement conditions, then stop refining the parent node and retain the parent node as the visual marker of its corresponding spatial region. S46. If the parent node meets the preset refinement conditions, the parent node is refined into its four corresponding child nodes, and each child node is taken as the parent node. The step of obtaining the semantic relevance score of each child node corresponding to the parent node is continued until all quadtree branches in the quadtree meet the preset refinement conditions or reach the finest level of the quadtree.

[0070] In step S41, when merging spatial dimensions, such as Figure 3 As shown, the granularity of preservation for each region is adaptively selected from coarse to fine. The root node (the top-level node of the quadtree, corresponding to the spatial region of the entire video frame) is used as the starting node for granularity selection, and then the selection is performed layer by layer downwards until the finest individual visual marker is reached. Child nodes are spatial regions located one level below their parent nodes. Each parent node corresponds to four adjacent child nodes, and each child node corresponds to a sub-region formed by dividing the parent node's spatial region into four equal parts. The spatial region corresponding to a child region is one-quarter of the spatial region corresponding to its parent node.

[0071] In step S42, the semantic relevance score is determined based on the semantic contribution of all visual markers within the spatial region corresponding to the child node. For example, a summation or average aggregation operation is performed on the semantic contributions of all visual markers within the spatial region corresponding to the child node, and the resulting aggregation is the semantic relevance score of the child node. This semantic relevance score reflects the importance of the spatial region corresponding to the child node to the overall semantics of the video. A high semantic relevance score indicates that the child node is highly important to the overall semantics of the video, and the necessity of retaining the child node is higher. Conversely, a low semantic relevance score indicates that the child node is less important to the overall semantics of the video, and the necessity of retaining the child node is lower.

[0072] For example, for a parent-child node pair , Indicates the parent node, The child node represents the sub-visual label feature sequence corresponding to the spatial region of the child node. , Represents the first sub-visual label feature sequence The visual characteristics of a visual marker The semantic relevance score of a child node can be expressed as: (This represents the number of visual markers contained in the spatial region corresponding to the child node.) , in, Represents child nodes semantic relevance score, This represents the visual markers contained in the spatial region corresponding to the child node. The semantic contribution of each visual marker.

[0073] In step S43, visual redundancy reflects the degree of feature similarity between the global visual features of the spatial region corresponding to the parent node and the global visual features of the spatial region corresponding to the child node. If the visual redundancy between the child region and the parent node is high, it indicates a high degree of feature similarity between the parent and child nodes, and a high probability that the parent node can characterize the spatial region corresponding to the child node. Conversely, if the visual redundancy between the child region and the parent node is low, it indicates a low degree of feature similarity between the parent and child nodes, and a low probability that the parent node can characterize the spatial region corresponding to the child node. The visual redundancy can be calculated using the cosine similarity between the global visual features corresponding to the parent node and the global visual features corresponding to the child node.

[0074] For example, for the parent-child node pair mentioned above We can first determine the global visual features of the parent node by using the global visual features of each of its four child nodes (e.g., by performing average pooling on the global visual features of the four child nodes to obtain the global visual features of the parent node), and then calculate the global visual features of the parent node. With child nodes The cosine similarity between them is used to obtain visual redundancy, where the parent-child node pair The corresponding visual redundancy can be expressed as: , in, Represents parent-child node pairs Corresponding visual redundancy, Indicates the parent node Global visual features, Represents child nodes Global visual features.

[0075] Furthermore, in step S43 above, after obtaining the semantic relevance score and visual redundancy, a joint decision score can be calculated using the semantic relevance score of each child node and the visual redundancy between each child node and its parent node. This joint decision score is then used as the basis for granularity decisions, i.e., determining whether to refine the parent node based on the decision score. This embodiment uses the joint decision score determined based on the semantic relevance score and visual redundancy as the basis for refinement. It does not uniformly compress the entire frame, nor does it simply merge based on visual similarity. Instead, it allows local content that is truly closely related to the overall semantics of the video to achieve higher fidelity, enabling the subsequent language model to still access key evidence when answering questions.

[0076] In one embodiment, determining the joint judgment score of each child node based on the semantic relevance score of each child node and the visual redundancy between each child node and its parent node specifically includes: Obtain the semantic weight coefficient corresponding to the semantic relevance score and the redundancy weight coefficient determined by the visual redundancy; A first judgment score is determined based on the semantic weight coefficient and the semantic relevance score, and a second judgment score is determined using the redundancy weight coefficient and the visual redundancy. The difference between the first judgment score and the second judgment score is obtained to obtain the joint judgment score.

[0077] Specifically, the semantic weight coefficient and the redundancy weight coefficient are pre-set weight parameters used to adjust the balance between semantic information preservation and visual redundancy elimination. Increasing the semantic weight coefficient enhances the role of regional semantic relevance in joint determination, causing more semantically important regions to be further refined, thus retaining more fine-grained visual markers. Increasing the redundancy weight coefficient enhances the role of parent-child visual similarity in joint determination, causing more visually highly redundant regions to stop refining, thereby increasing the marker compression ratio.

[0078] In practical applications, semantic weight coefficients, redundancy weight coefficients, and subsequent spatial refinement thresholds can be determined based on the target label budget using a pre-selected set of calibration videos. Specifically, while keeping the parameters of the large language model in the video constant, different combinations of semantic weight coefficients, redundancy weight coefficients, and subsequent spatial refinement thresholds are tested. The number of compressed visual tags and video question-answering performance are statistically analyzed, and semantic weight coefficients, redundancy weight coefficients, and subsequent spatial refinement thresholds that meet the target label budget while maintaining high question-answering performance are selected. When selecting semantic weight coefficients, redundancy weight coefficients, and subsequent spatial refinement thresholds, one or more of the following factors can be considered: (1) Feature distribution of the visual encoder used; (2) The numerical range of semantic relevance score and visual redundancy score; (3) The proportion of visual markers retained; (4) Requirements for retaining key objects, key actions, and text areas; (5) The type of video large language model and the size of the visual marker grid.

[0079] For the same video large language model, the same visual encoder, the same target label budget, and the same compression configuration, the semantic weight coefficient, the redundancy weight coefficient, and the subsequent spatial refinement threshold remain unchanged in different question-answering rounds to ensure that the same video can generate a stable and reproducible compressed label sequence.

[0080] Furthermore, the first decision score represents the need to preserve fine-grained semantic information of the sub-region, while the second decision score represents the possibility of using the parent node to replace the child node and eliminate visual redundancy. When the semantic relevance of the sub-region is high, the first decision score increases, indicating that the sub-region needs further refinement, which can increase the joint decision score. When the visual similarity between the parent node and the child node is high, the second decision score increases, indicating that the sub-region is easier to represent by the parent node and does not need further refinement, which can decrease the joint decision score.

[0081] Therefore, after obtaining the first decision score and the second decision score, the difference between the first decision score and the second decision score can be used as the joint decision score, that is, the joint decision score can be expressed as: , in, Represents parent-child node pairs The corresponding joint judgment score, Represents semantic weight coefficients. This represents the redundancy weight coefficient.

[0082] The joint decision score in this application embodiment actually reflects two opposite factors simultaneously: one is whether the sub-region contains important semantic information that needs to be protected, and the other is whether the sub-region is highly similar to the parent region and has strong visual redundancy. This can better balance the relationship between semantic preservation and compression efficiency, and while meeting the label compression rate requirements, retain the key semantic content in the video that is valuable to downstream tasks to the greatest extent, and avoid high semantic value regions being merged and lost due to compression.

[0083] Furthermore, in steps S44, S45, and S46 above, the joint decision score of each child node is used to determine whether the parent node meets the preset refinement condition, that is, to determine whether the child node corresponding to the parent node is selected as a retained node. The preset refinement condition can be: If there is at least one child node satisfy: , Then determine the parent node If a node cannot adequately represent all its sub-regions (i.e., it cannot represent all its child nodes), then the spatial region corresponding to the parent node will no longer be uniformly represented by the parent node, and all child nodes corresponding to the parent node will be reserved as nodes.

[0084] Therefore, determining whether the parent node meets the preset refinement conditions based on the joint judgment score of each child node specifically includes: The joint decision score of each child node is compared with the spatial refinement threshold; If the joint decision score of all child nodes is less than or equal to the spatial refinement threshold, the decision does not meet the preset refinement condition. If at least one child node has a joint decision score greater than the spatial refinement threshold, the preset refinement condition is satisfied.

[0085] Specifically, the spatial refinement threshold is preset, for example, it can be set as described above. If the joint decision score of all child nodes is less than or equal to the spatial refinement threshold, the preset refinement condition is not met, and refinement of the parent node stops, retaining the parent node as the visual marker of its corresponding spatial region. If at least one child node's joint decision score is greater than the spatial refinement threshold, the preset refinement condition is met, and the parent node is refined into its four corresponding child nodes. Each child node is then treated as a parent node, and the step of obtaining the semantic relevance scores of each child node corresponding to the parent node continues until all quadtree branches in the quadtree meet the preset refinement condition or reach the finest level of the quadtree.

[0086] This application embodiment, through its coarse-to-fine adaptive quadtree decomposition and preservation process, can automatically select the most suitable visual marker granularity for different spatial regions of a single video frame. Decomposition is stopped early only for regions with low semantic value and high visual redundancy, achieving differentiated visual marker compression in the spatial dimension. This effectively reduces the total number of visual markers retained in a single frame, meeting the input length constraints of downstream tasks, while maximizing the preservation of fine-grained details in high-semantic-value regions such as people, key objects, text information, and action areas in the video. This avoids the erroneous merging or loss of key semantic information due to compression, improving the expressive power of the compressed visual markers for key video content. Furthermore, Figure 4 and Figure 5 The visualization of the spatial merging results also shows that semantically salient regions are preserved with finer granularity, while large areas of background are merged into coarser granularity.

[0087] S50. Obtain the semantically stable window of the original video, and perform directed merging of the spatial visual tags in the semantically stable window to determine the spatiotemporally merged visual tags.

[0088] Specifically, the semantically stable window is the execution window for cross-frame merging (i.e., temporal merging). It treats the video frames included in the semantically stable window as a processing unit and merges the spatial visual markers corresponding to the video frames within the semantically stable window in the temporal dimension. Since video content typically exhibits semantic continuity between consecutive frames, the semantic content of the same spatial region in adjacent frames usually does not undergo drastic changes. Therefore, by merging stable, repeating visual markers within the semantically stable window, the total number of visual markers can be further compressed in the temporal dimension without losing key semantic change information. The semantically stable window can be adaptively determined based on the degree of semantic change between adjacent frames or can be preset to a fixed length.

[0089] In one embodiment, a semantically stable window is adaptively determined based on the degree of semantic change between adjacent frames. Compared with a fixed-length sliding window, this can be closer to the semantically stable interval in the video, providing safer boundary conditions for cross-frame merging within the subsequent window and avoiding excessive merging across semantic mutation points.

[0090] For example, obtaining the semantically stable window of the original video specifically includes: Global visual embedding of each video frame in the original video; Obtain the cosine correlation score between the global visual embedding and the video-level semantic descriptor to form a time-ordered frame-level correlation sequence; Based on the frame-level correlation sequence, the semantic correlation score between adjacent video frames is determined, and the original video is divided into several consecutive semantically stable windows according to the semantic correlation score.

[0091] Specifically, for each video frame in the original video, the global visual embedding of the video frame can be extracted first. Then, the cosine correlation score between the global visual embedding and the video-level semantic descriptor can be calculated. The cosine correlation scores of each video frame are arranged according to their temporal order in the original video to form a frame-level correlation sequence. Then, the absolute difference between the frame-level correlation scores of two adjacent video frames can be calculated, and this absolute difference can be used as the semantic correlation score between the corresponding adjacent frames. When the semantic correlation score between adjacent frames is greater than a preset semantic change threshold, it indicates that the video semantics of the two adjacent frames have changed significantly. This position is used as the segmentation point of the semantic stability window, and all consecutive video frames between the two segmentation points constitute a semantic stability window. In this embodiment, the original video can be automatically divided into several semantic stability intervals of unequal length. Segments with relatively stable semantics will form longer windows, which are suitable for more fully merging redundant information. Positions with semantic transitions, event switching, or key action changes will form shorter windows to prevent different semantic stages from being incorrectly merged. This ensures that the video semantic change amplitude within each semantic stability window is small and there are no drastic semantic jumps, providing a reasonable processing unit for subsequent cross-frame merging.

[0092] Furthermore, after obtaining the semantically stable window, cross-frame marker merging can be performed within each semantically stable window. This involves comparing spatially aligned spatial visual markers between adjacent frames, determining mergeable spatial visual marker pairs based on the comparison results, establishing directed links between the mergeable pairs in chronological order, and finally merging them along the temporal dimension based on these directed links. Spatially aligned refers to overlapping spatial regions corresponding to two spatial visual markers, and mergeable means that two spatial visual markers meet preset merging conditions (such as visual similarity exceeding a cross-frame visual similarity threshold).

[0093] In one embodiment, the directed merging of the spatial visual markers within the semantically stable window specifically includes: For two adjacent video frames in the semantically stable window, obtain spatial visual label pairs in which there are overlapping spatial regions in both adjacent video frames. Obtain the visual similarity of spatial visual tag pairs, and establish directed links between two spatial visual tags in the spatial visual tag pairs based on the visual similarity. Spatial visual tags formed through directed links are grouped into the same spatial visual tag group to obtain several spatial visual tag groups; The earliest spatial visual mark in each spatial visual mark group is taken as the representative mark, and the other spatial visual marks in each spatial visual mark group are merged into the representative mark of each spatial visual mark group.

[0094] Specifically, for the semantically stable window, the first Frame and the Frame, number Spatial visual markers in frames , No. Spatial visual markers in the frame are Spatial visual markers The corresponding spatial region is Spatial visual markers The corresponding spatial region is ,like and Same, or, and If there are overlapping spatial regions, then spatial visual markings will be used. and spatial visual markers Identify candidate matching tags, i.e., spatial visual tags. and spatial visual markers These constitute pairs of spatial visual markers with overlapping spatial regions. The overlapping spatial regions are used only to determine whether the two spatial visual markers represent the same or partially the same spatial content.

[0095] Next, the visual similarity between the two spatial visual label pairs is calculated, that is, the spatial visual label is calculated. Corresponding global visual features and spatial visual markers The visual similarity between corresponding global visual features, rather than just calculating and Visual similarity of overlapping spatial regions. The visual similarity of this spatial visual label pair can be expressed as: , in, Represents the visual similarity of spatial visual label pairs. Spatial visual markers Corresponding global visual features Spatial visual markers The corresponding global visual features.

[0096] Then, the visual similarity of the spatial visual tag pairs is calculated. Cross-frame visual similarity threshold If a comparison is made, Then determine the spatial visual markers. and spatial visual markers In the time direction, it represents continuous and stable identical or similar content, and establishes a directed link from the later frame marker to the previous frame marker. This directed link is represented as: .

[0097] Therefore, for two spatial visual tags to be linked across frames, both of the following conditions must be met: There is an overlapping spatial region between the two spatial visual markers; The visual similarity between two spatial visual tags is higher than the cross-frame visual similarity threshold; The two spatial visual markers are located within the same semantically stable time window.

[0098] This application uses visual feature similarity as the merging criterion and utilizes a semantically stable time window to limit the merging range across frames. This not only avoids merging errors in marking different event stages, states before and after actions, before and after the appearance of subtitles, or before and after changes in key objects, but also preserves the position of the first appearance of content and key change evidence while compressing repetitive backgrounds and stable objects. This makes the compressed sequence more stable, more interpretable, and more suitable for subsequent deterministic sorting and cache reuse.

[0099] Furthermore, after establishing all directed links within the semantically stable window, the spatial visual tags formed through directed links are grouped into the same spatial visual tag group to obtain several spatial visual tag groups. Additionally, to avoid ambiguity in the meaning of the connected components of the directed graph, the direction of the directed edges can be ignored, and spatial visual tags belonging to the same weakly connected component can be grouped into the same spatial visual tag group. The spatial visual tag group can be represented as: , in, Indicates spatial visual marker group, This indicates the number of spatial visual markers in a spatial visual marker group.

[0100] Next, the earliest spatial visual markers in time will be... This representative marker serves as the representative marker for the spatial visual marker group, and other spatial visual markers within the group are merged into it. The visual characteristics of the merged representative marker are represented as follows: , in, The visual characteristics of the merged representative mark. Spatial visual markers Visual features Spatial visual markers The combined weights.

[0101] In one specific embodiment, an equal-weighted average can be used, i.e. Then, the visual features of the merged representative mark are represented as follows: .

[0102] This application embodiment utilizes a spatiotemporally compressed tag set obtained by merging the temporal dimensions within a semantically stable window. This significantly reduces temporal redundancy while preserving multi-granularity spatial coverage. Simultaneously, Figure 6 The blank areas represent spatial visual tags that have been merged into the spatial visual tags of earlier frames within the semantically stable window, which also shows that the embodiments of this application have removed a large amount of cross-frame duplicate content without destroying key semantic changes.

[0103] It should be noted that in practical applications, the merging weight can also be determined based on the spatial area covered by the marker, the number of original visual markers represented by the marker, the semantic relevance score of the region, or the temporal distance between the marker and the representative marker. No specific restrictions are imposed here.

[0104] Furthermore, after completing the semantically guided spatiotemporal tag merging, the compressed spatiotemporal tags are rearranged in a deterministic order and then input into subsequent layers of the language model after adding positional encoding. Since the spatial hierarchy selection, temporal window division, and cross-frame merging path are deterministic under the same video and the same video-level semantic descriptor conditions, the resulting compressed prefix representation is stable and reproducible. Thus, in multi-turn question-answering scenarios where multiple questions correspond to the same video, the system only needs to perform a complete pre-filling and key-value cache once in the first turn; subsequent turns can directly reuse this shared video prefix without recalculating the video compression strategy for each question. Multi-turn latency analysis shows that after the method incurs the one-time semantic modeling and preprocessing cost in the first turn, the average latency of the second and third turns decreases to 0.2313 seconds and 0.2509 seconds, respectively. In contrast, the query-aware baseline still maintains a continuous computational overhead of approximately 2.5 to 2.7 seconds in the latter three turns, indicating that this application can improve the timeliness of multi-turn interactive video understanding.

[0105] Furthermore, under different labeling budgets, the compression ratio requirements can be met by adjusting the spatial merging threshold, temporal similarity threshold, and temporal window boundary threshold. Experimental results show that the embodiment of this application performs stably under both 50% and 25% budgets. Especially under more aggressive compression conditions, it maintains performance better than comparison methods that rely solely on visual similarity, indicating that the embodiment of this application has good budget adjustability.

[0106] In summary, this embodiment provides a semantically guided spatiotemporal marker merging method. The method includes: acquiring a visual marker grid and a video-level semantic descriptor from the original video; acquiring the semantic relevance between the visual marker grid and the video-level semantic descriptor; acquiring the gradient vector of the semantic relevance relative to each visual marker in the visual marker grid, and determining the semantic contribution of the visual marker to the original video based on the gradient vector; merging the visual marker grid in the spatial dimension based on the semantic contribution and visual redundancy to obtain spatial visual markers; acquiring a semantically stable window of the original video, and performing directed merging of the spatial visual markers within the semantically stable window to determine the spatiotemporally merged visual markers. This application embodiment, without retraining the model, utilizes the video-level semantic descriptor of the original video to determine the semantic contribution of visual tags to the original video, and performs spatial merging based on semantic contribution and visual redundancy. Then, it performs cross-frame directed merging within the semantically stable window, so that the resulting compressed representation can significantly reduce the pre-filling overhead of long videos, retain key semantic evidence valuable for multi-round question answering, and ensure the stability and determinism of the compressed representation on the same video. This allows the compressed video prefix obtained in the first round of pre-filling to be directly reused in subsequent rounds of questions, reducing the computational cost of repeated pre-filling and the latency of multi-round interactions.

[0107] Based on the above-described semantically guided spatiotemporal tag merging method, this embodiment provides a semantically guided spatiotemporal tag merging apparatus, such as... Figure 7 As shown, the semantically guided spatiotemporal tag merging device specifically includes: The acquisition module 100 is used to acquire the visual marker grid and video-level semantic descriptor of the original video; Semantic relevance acquisition modulus 200 is used to acquire the semantic relevance between the visual marker grid and the video-level semantic descriptor; The semantic contribution determination module 300 is used to obtain the gradient vector of the semantic relevance relative to each visual marker in the visual marker grid, and determine the semantic contribution of the visual marker to the original video based on the gradient vector. The spatial merging module 400 is used to merge the visual marker grid in the spatial dimension based on the semantic contribution and visual redundancy to obtain spatial visual markers. The temporal merging module 500 is used to obtain the semantically stable window of the original video and perform directed merging of the spatial visual tags in the semantically stable window to determine the spatiotemporally merged visual tags.

[0108] Based on the above semantically guided spatiotemporal tag merging method, this embodiment provides a computer-readable storage medium storing one or more programs that can be executed by one or more processors to implement the steps in the semantically guided spatiotemporal tag merging method as described in the above embodiment.

[0109] Based on the aforementioned semantically guided spatiotemporal tag merging method, this application also provides a terminal device, such as... Figure 8 As shown, it includes at least one processor 20; a display screen 21; and a memory 22, and may also include a communications interface 23 and a bus 24. The processor 20, display screen 21, memory 22, and communications interface 23 can communicate with each other via the bus 24. The display screen 21 is configured to display a preset user guide interface in the initial setup mode. The communications interface 23 can transmit information. The processor 20 can invoke logical instructions in the memory 22 to execute the methods described in the above embodiments.

[0110] Furthermore, the logical instructions in the aforementioned memory 22 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium.

[0111] The memory 22, as a computer-readable storage medium, can be configured to store software programs, computer-executable programs, such as program instructions or modules corresponding to the methods in the embodiments of this disclosure. The processor 20 executes functional applications and data processing by running the software programs, instructions, or modules stored in the memory 22, thereby implementing the methods in the above embodiments.

[0112] The memory 22 may include a program storage area and a data storage area. The program storage area may store the operating system and application programs required for at least one function; the data storage area may store data created based on the use of the terminal device. Furthermore, the memory 22 may include high-speed random access memory (RAM) and non-volatile memory. Examples include various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks, as well as transient storage media.

[0113] Furthermore, the specific process of loading and executing multiple instruction processors in the aforementioned storage medium and terminal device has been described in detail in the above method, and will not be repeated here.

[0114] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A semantically guided spatiotemporal tag merging method, characterized in that, The semantically guided spatiotemporal tag merging method specifically includes: Obtain the visually labeled grid and video-level semantic descriptor of the original video; Obtain the semantic relevance between the visual marker grid and the video-level semantic descriptor; Obtain the gradient vector of the semantic relevance relative to each visual marker in the visual marker grid, and determine the semantic contribution of the visual marker to the original video based on the gradient vector; Based on the semantic contribution and visual redundancy, the visual marker grid is merged in the spatial dimension to obtain spatial visual markers; Obtain the semantically stable window of the original video, and perform directed merging of the spatial visual tags within the semantically stable window to determine the spatiotemporally merged visual tags.

2. The semantically guided spatiotemporal tag merging method according to claim 1, characterized in that, The process of obtaining the video-level semantic descriptor specifically includes: The original video is input into a video big language model, and the video big language model performs lightweight semantic generalization on the original video to obtain a video-level semantic descriptor, wherein the video-level semantic descriptor is used to reflect the video content overview of the original video.

3. The semantically guided spatiotemporal tag merging method according to claim 1, characterized in that, The specific steps of obtaining the semantic relevance between the visual marker grid and the video-level semantic descriptor include: The video-level semantic descriptor is encoded into a semantic embedding, and video-level visual features are generated based on the visual marker grid. Obtain the semantic similarity between the semantic embedding and the video-level visual features to obtain semantic relevance.

4. The semantically guided spatiotemporal tag merging method according to claim 1, characterized in that, The determination of the semantic contribution of visual markers to the original video based on the gradient vector specifically includes: The channel importance vector is determined based on the gradient vector of all visual markers; The visual marker is inner-producted with the channel importance vector to obtain the response value of the visual marker along the video channel direction; The semantic contribution of the visual markers to the original video is determined based on the response values.

5. The semantically guided spatiotemporal tag merging method according to claim 1, characterized in that, The merging of the visual marker grid in the spatial dimension based on the semantic contribution and visual redundancy specifically includes: Using the root node in the quadtree as the parent node, the semantic relevance score of each child node corresponding to the parent node is obtained. The semantic relevance score is determined based on the semantic contribution of all visual markers within the spatial region corresponding to the child node. Obtain the visual redundancy between the parent node and each of its corresponding child nodes; Based on the semantic relevance score of each child node and the visual redundancy between each child node and its parent node, the joint judgment score of each child node is determined. The parent node is determined to meet the preset refinement conditions based on the joint judgment score of each child node. If the parent node does not meet the preset refinement conditions, the refinement of the parent node will be stopped, and the parent node will be retained as the visual marker of its corresponding spatial region. If the parent node meets the preset refinement conditions, the parent node is refined into its four corresponding child nodes, and each child node is taken as the parent node. The step of obtaining the semantic relevance score of each child node corresponding to the parent node is continued until all quadtree branches in the quadtree meet the preset refinement conditions or reach the finest level of the quadtree.

6. The semantically guided spatiotemporal tag merging method according to claim 5, characterized in that, The determination of the joint judgment score of each child node based on the semantic relevance score of each child node and the visual redundancy between each child node and its parent node specifically includes: Obtain the semantic weight coefficient corresponding to the semantic relevance score and the redundancy weight coefficient determined by the visual redundancy; A first judgment score is determined based on the semantic weight coefficient and the semantic relevance score, and a second judgment score is determined using the redundancy weight coefficient and the visual redundancy. The difference between the first judgment score and the second judgment score is obtained to obtain the joint judgment score.

7. The semantically guided spatiotemporal tag merging method according to claim 5, characterized in that, The step of determining whether the parent node meets the preset refinement conditions based on the joint judgment scores of each child node specifically includes: The joint decision score of each child node is compared with the spatial refinement threshold; If the joint decision score of all child nodes is less than or equal to the spatial refinement threshold, the decision does not meet the preset refinement condition. If at least one child node has a joint decision score greater than the spatial refinement threshold, the preset refinement condition is satisfied.

8. The semantically guided spatiotemporal tag merging method according to claim 1, characterized in that, The specific steps of obtaining the semantically stable window of the original video include: Global visual embedding of each video frame in the original video; Obtain the cosine correlation score between the global visual embedding and the video-level semantic descriptor to form a time-ordered frame-level correlation sequence; Based on the frame-level correlation sequence, the semantic correlation score between adjacent video frames is determined, and the original video is divided into several consecutive semantically stable windows according to the semantic correlation score.

9. The semantically guided spatiotemporal tag merging method according to claim 1, characterized in that, The directed merging of the spatial visual markers within the semantically stable window specifically includes: For two adjacent video frames in the semantically stable window, obtain spatial visual label pairs in which there are overlapping spatial regions in both adjacent video frames. Obtain the visual similarity of spatial visual tag pairs, and establish directed links between two spatial visual tags in the spatial visual tag pairs based on the visual similarity. Spatial visual tags formed through directed links are grouped into the same spatial visual tag group to obtain several spatial visual tag groups; The earliest spatial visual mark in each spatial visual mark group is taken as the representative mark, and the other spatial visual marks in each spatial visual mark group are merged into the representative mark of each spatial visual mark group.

10. A semantically guided spatiotemporal tag merging device, characterized in that, The semantically guided spatiotemporal tag merging device specifically includes: The acquisition module is used to acquire the visual marker grid and video-level semantic descriptor of the original video; A semantic relevance acquisition module is used to acquire the semantic relevance between the visual marker grid and the video-level semantic descriptor; A semantic contribution determination module is used to obtain the gradient vector of the semantic relevance relative to each visual marker in the visual marker grid, and determine the semantic contribution of the visual marker to the original video based on the gradient vector; A spatial merging module is used to merge the visual marker grid in the spatial dimension based on the semantic contribution and visual redundancy to obtain spatial visual markers; The temporal merging module is used to obtain the semantically stable window of the original video and perform directed merging of the spatial visual tags within the semantically stable window to determine the spatiotemporally merged visual tags.

11. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores one or more programs, which can be executed by one or more processors to implement the steps in the semantically guided spatiotemporal tag merging method as described in any one of claims 1-9.

12. A terminal device, characterized in that, include: Processor and memory; The memory stores a computer-readable program that can be executed by the processor; When the processor executes the computer-readable program, it implements the steps in the semantically guided spatiotemporal tag merging method as described in any one of claims 1-9.

Images