A video clip positioning method and system based on fine-grained spatiotemporal correlation modeling

Abstract

The present invention belongs to the field of video retrieval technology, and provides a method and system for locating video clips based on fine-grained spatiotemporal association modeling. The technical solution is as follows: obtain video clips, use spatiotemporal query representation to implicitly mine all potential object information in the video clips; then, based on the spatiotemporal representation multidimensional interaction module, fully model the spatiotemporal association relationship between objects; then, by organically integrating local and global representations, comprehensively improve the representation ability of video clips; finally, determine the target video clip based on the similarity score between the video clip representation and the user query representation. The present invention overcomes the problems in the prior art that rely on offline object detection tools to extract object spatiotemporal information, insufficient modeling of fine-grained object interaction information, etc., resulting in poor video understanding.

Description

Technical Field

[0001] The present invention belongs to the technical field of video retrieval, and in particular to a video segment positioning and system based on fine-grained spatiotemporal correlation modeling. Background Art

[0002] The statements in this section merely provide background information related to the present invention and do not necessarily constitute prior art.

[0003] As the number of long videos in real application scenarios gradually increases, the number of action instances in videos continues to increase, and the complexity of video backgrounds continues to increase, simply retrieving the long videos corresponding to the query is no longer able to meet actual needs. Practical applications hope to accurately locate the target video clips that meet the needs of specific applications from long videos.

[0004] To achieve accurate positioning of target video clips, the understanding and representation of video content plays a vital role. However, video data in open scenes is usually lengthy and complex, especially when it involves multiple interactive behaviors, which brings significant challenges to video content understanding and positioning.

[0005] Existing video clip localization methods often use pre-trained visual feature extractors, such as C3D and I3D, to extract video clip representations. However, these visual feature extractors are generally pre-trained on cropped video clip datasets, so it is difficult to fully capture the temporal contextual relationship and temporal boundary information between different actions, resulting in the lack of clear recognition of temporal boundaries in video clip representations. Transformer technology has been extended to the field of computer vision due to its outstanding performance in natural language processing. Representative models such as ViT have performed well in a variety of visual tasks. In order to improve the efficiency and effect of image classification of ViT, a variety of improved methods based on ViT have been proposed. For example, Swin Transformer uses local attention windows and enhances the interaction between windows through displacement operations; DaViT introduces a dual self-attention mechanism to capture global context with linear computational complexity. Inspired by the significant performance of ViT, researchers began to apply it to video content understanding. Uniformer combines convolution with self-attention to solve the problems of local redundancy and global dependency. TimeSformer and ViViT explore strategies to separate temporal and spatial attention, and both achieve excellent performance. MViT introduces multi-scale features for video sequences, while Video SwinTransformer adapts the model to a 3D environment.

[0006] However, the above models still have shortcomings in modeling fine-grained spatiotemporal correlation relationships. On the one hand, they fail to fully capture the object information in the video; on the other hand, they fail to effectively analyze the complex object behavior interaction information in the video. Summary of the invention

[0007] In order to solve at least one technical problem existing in the above-mentioned background technology, the present invention provides a video clip positioning method and system based on fine-grained spatiotemporal correlation modeling, which effectively captures the object information in the video by introducing object query representation, and fully understands the spatiotemporal interaction behavior of the object by introducing an interactive modeling mechanism of three dimensions of time-space-representation, thereby overcoming the defects of incomplete object capture and insufficient spatiotemporal interaction in the prior art.

[0008] In order to achieve the above object, the present invention adopts the following technical solution:

[0009] A first aspect of the present invention provides a video segment positioning method based on fine-grained spatiotemporal association modeling, comprising the following steps:

[0010] Acquire video data, perform multi-scale time window sampling on the video to obtain video clips of different time lengths, and perform frame-level encoding on the video clips to obtain a frame representation sequence of the encoded video clips;

[0011] According to the spatiotemporal query representation, the spatiotemporal representation of the object is obtained from the sequence of frame representations of the video clip;

[0012] Based on the spatiotemporal representation of objects, the fine-grained interaction information of objects at the temporal, spatial and semantic levels is modeled by alternately performing temporal attention, spatial attention and channel attention, and the ability to understand spatiotemporal interaction information of videos is enhanced from multiple dimensions, thus obtaining a frame sequence representation of video clips jointly enhanced by time, space and channels.

[0013] Extracting a global representation of a video segment;

[0014] The global representation of the video is fused with the time-space-channel joint enhanced video clip frame sequence representation to obtain the final video clip representation;

[0015] Encode the text query statement input by the user to obtain a query representation;

[0016] According to the similarity between the video segment representation and the query representation, the target video segment is determined.

[0017] Furthermore, obtaining the spatiotemporal representation of the object from the video clip frame representation sequence according to the spatiotemporal query representation includes:

[0018] According to the temporal query representation and the spatial query representation, a first spatiotemporal query representation is obtained by copying the spatial query representation to each temporal query representation;

[0019] Perform self-attention calculation on the first spatiotemporal query representation to obtain a second spatiotemporal query representation that can capture the spatiotemporal correlation relationship;

[0020] According to the second spatiotemporal query representation, cross-attention is used to obtain the spatiotemporal representation of the object from the sequence of frame representations of the video clip.

[0021] Furthermore, performing self-attention calculation on the first spatiotemporal query representation to obtain a second spatiotemporal query representation capable of capturing the spatiotemporal correlation relationship includes:

[0022] Perform spatial self-attention calculation on the first spatiotemporal query representation to capture the spatial interaction information of objects and obtain a spatiotemporal query representation with enhanced spatial perception;

[0023] Perform temporal self-attention calculation on the spatial-perception enhanced spatiotemporal query representation to obtain the spatial-perception enhanced spatiotemporal query representation;

[0024] The spatiotemporal query representation enhanced with spatiotemporal awareness is fused with the first spatiotemporal query representation to obtain a second spatiotemporal query representation.

[0025] Furthermore, the obtaining the object spatiotemporal representation from the video clip frame representation sequence using cross attention according to the second spatiotemporal query representation includes:

[0026] Taking the second spatiotemporal query representation as the query and the video clip frame sequence representation as the key and value, calculate the cross attention mechanism to obtain the spatiotemporal representation of the object;

[0027] fusing the object spatiotemporal representation with the second spatiotemporal query representation to obtain an updated object spatiotemporal representation;

[0028] The updated spatiotemporal representation of the object is input into the multi-layer perceptron network and added to the updated spatiotemporal representation of the object to obtain the final spatiotemporal representation of the object.

[0029] Furthermore, the object-based spatiotemporal representation, by alternately performing temporal attention, spatial attention and channel attention, models the fine-grained interaction information of the object at the temporal, spatial and semantic levels, enhances the video spatiotemporal interaction information understanding ability from multiple dimensions, and obtains the video clip frame sequence representation with joint temporal-spatial-channel enhancement, including:

[0030] Temporal self-attention is calculated for each object in the spatiotemporal representation of the object to capture the temporal information and obtain a temporally enhanced frame sequence representation of the video clip;

[0031] For each frame in the temporally enhanced video clip frame sequence representation, spatial self-attention is calculated separately to capture the interaction information between objects and obtain the spatiotemporally enhanced video clip frame sequence representation;

[0032] The channel dimensions of the spatiotemporal enhanced video clip frame sequence representation are grouped, and self-attention calculation is performed within each group to further enhance the semantic level interaction and obtain the spatiotemporal-channel jointly enhanced video clip frame sequence representation.

[0033] Furthermore, when extracting the global representation of the video clip, the pre-trained video understanding large model InternVideo2 is used.

[0034] Furthermore, when the global representation of the video clip is fused with the time-space-channel jointly enhanced video clip frame sequence representation, the global representation of the video clip is copied, expanded to a dimension consistent with the time-space-channel jointly enhanced video clip frame sequence representation, and then added to the time-space-channel jointly enhanced video clip frame sequence representation to obtain the final fine-grained video clip representation.

[0035] Furthermore, the pre-trained natural language understanding large model BERT is used to extract user text query representation.

[0036] Furthermore, the target video segment is determined according to the maximum cosine similarity score between the video segment and the query representation.

[0037] A second aspect of the present invention provides a video segment positioning system based on fine-grained spatiotemporal association modeling, comprising:

[0038] A video encoding module is used to perform multi-scale time window sampling on the video to obtain video clips of different time lengths, and to perform frame-level encoding on the video clips to obtain a frame representation sequence of the encoded video clips;

[0039] A spatiotemporal representation acquisition module, which is used to obtain the spatiotemporal representation of the object from the video clip frame representation sequence according to the spatiotemporal query representation;

[0040] The local representation extraction module is used to model the fine-grained interaction information of objects at the temporal, spatial and semantic levels based on the spatiotemporal representation of objects by alternately performing temporal attention, spatial attention and channel attention, and enhance the ability to understand the spatiotemporal interaction information of videos from multiple dimensions, thus obtaining a frame sequence representation of video clips jointly enhanced by time, space and channels;

[0041] A global representation extraction module for extracting a global representation of a video clip;

[0042] A representation fusion module is used to fuse the global representation of the video segment with the time-space-channel joint enhanced video segment frame sequence representation to obtain the final video segment representation;

[0043] The target video segment determination module is used to encode the text query sentence input by the user to obtain the query representation; and determine the target video segment according to the similarity between the video segment representation and the query representation.

[0044] Compared with the prior art, the present invention has the following beneficial effects:

[0045] 1. The present invention considers multiple key factors, including temporal interaction behavior, spatial interaction behavior, and video segment representation enhanced by channel semantic perception through the object spatiotemporal representation extraction network and the spatiotemporal representation multidimensional interaction network, improves the behavior understanding ability of fine-grained objects in open scene videos, and effectively optimizes the accuracy of target video segment positioning;

[0046] 2. The object spatiotemporal representation extraction network of the present invention can effectively capture the object information and object spatiotemporal interaction information in the video without the help of external target detection tools, thereby improving the flexibility of video segment representation extraction and the completeness of video content understanding;

[0047] 3. The spatiotemporal representation multidimensional interaction network of the present invention can jointly mine the temporal interaction behavior and spatial interaction behavior between objects, while effectively perceiving the channel semantic information, and obtain a video clip representation with enhanced fine-grained content understanding.

[0048] Advantages of additional aspects of the present invention will be given in part in the following description, and in part will become obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] The accompanying drawings in the specification, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0050] Figure 1 It is a flow chart of a video clip positioning method based on fine-grained spatiotemporal association modeling provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0051] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0052] It should be noted that the following detailed descriptions are all illustrative and intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present invention belongs.

[0053] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit exemplary embodiments according to the present invention. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprising" and / or "including" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof.

[0054] For the practical applications mentioned in the background technology, it is more expected to accurately locate the target video clips that meet the needs of specific applications from long videos. Taking smart cities as an example, imagine such a scenario: when the city’s monitoring system detects abnormal behavior, people only need to use natural language to query, such as "there is a video clip of a fight at the entrance of a shopping mall", to quickly retrieve and identify the relevant target abnormal behavior, and then promptly issue an alarm to the relevant departments, which will greatly improve the efficiency of urban management and the speed of response.

[0055] Therefore, it is particularly important to study efficient video clip localization methods that can accurately and quickly locate video clips that meet natural language queries from unedited long videos containing multiple action instances and complex backgrounds.

[0056] The present invention provides a video segment positioning method based on fine-grained spatiotemporal correlation modeling, which performs sufficient spatiotemporal information modeling to enhance the video segment representation, such as Figure 1 As shown in the figure, firstly, the spatiotemporal query representation is used to implicitly mine all the potential object information in the video; then, based on the spatiotemporal representation multi-dimensional interaction module, the spatiotemporal correlation between objects is fully modeled; finally, the representation ability of video clips is comprehensively improved by organically integrating local and global representations. It effectively captures the object information in the video by introducing the object query representation, and fully understands the spatiotemporal interaction behavior of the object by introducing the interactive modeling mechanism of the three dimensions of time-space-representation, thus overcoming the defects of incomplete object capture and insufficient spatiotemporal interaction in the prior art.

[0057] Embodiment 1

[0058] This embodiment provides a video segment positioning method based on fine-grained spatiotemporal association modeling, including the following steps:

[0059] Step 1: Obtain video data, perform multi-scale time window sampling on the video, and obtain video clips of different time lengths { }, Indicates Video clips, Indicates the number of video clips.

[0060] Step 2: Encode the video clip at the frame level to obtain an encoded video clip frame representation sequence;

[0061] In this embodiment, a pre-trained visual representation model is used to perform frame-level encoding on the video clip, including:

[0062] Step 201, extracting video frames from the collected video clips. Assuming that the current video clip consists of T consecutive video frames, the video clip is formally represented as ,inH and W Represents the height and width of the video frame respectively.

[0063] Step 202: Use the pre-trained visual representation model ViViT (Video Vision Transformer) to perform frame-level encoding on each video segment and output the encoded video segment representation. ,in Represents the feature representation dimension.

[0064] Step 3: According to the spatiotemporal query representation, obtain the spatiotemporal representation of the object from the video clip frame representation sequence; specifically, the steps include:

[0065] Step 301: based on the temporal query representation and the spatial query representation, by copying the spatial query representation to each temporal query representation, a spatiotemporal query representation is obtained;

[0066] In order to accurately capture the dynamic changes and location information of objects in the video, the present invention decouples the spatiotemporal query representation into a temporal query and spatial query By copying the spatial query to each frame, we can get the spatiotemporal query, i.e. ,in and are temporal and spatial indexes respectively. The structural design of the query representation helps to stably track the same object in a video sequence, especially when the object moves slightly.

[0067] Step 302: Perform self-attention calculation on the spatiotemporal query representation to obtain a spatiotemporal query representation that can capture the spatiotemporal correlation relationship.

[0068] The present invention will interact with the video representation based on the spatiotemporal query representation to obtain the spatiotemporal representation information of the object. Specifically, the spatiotemporal query representation is first self-attention calculated. In order to save computing resource consumption, the spatial self-attention calculation is performed first, and then the temporal attention calculation is performed.

[0069] The specific steps include:

[0070] Step 3021: First, perform spatial self-attention calculation on the given spatiotemporal query representation to capture the spatial interaction information of objects and obtain a spatiotemporal query representation with enhanced spatial perception:

[0071] ,

[0072] in, is the self-attention mechanism function, Represents a given spatiotemporal query representation.

[0073] Step 3022: Perform temporal self-attention calculation on the spatial-perception enhanced spatiotemporal query representation to obtain the spatial-perception enhanced spatiotemporal query representation:

[0074] ,

[0075] in, is the self-attention mechanism function.

[0076] Step 3023: Fusing the spatiotemporal query representation enhanced with spatiotemporal awareness with the given spatiotemporal query representation to obtain an updated spatiotemporal query representation:

[0077] ,

[0078] Step 303: Based on the updated spatiotemporal query representation, the object spatiotemporal representation is obtained from the video frame representation sequence using cross attention. Specifically, it includes:

[0079] Step 3031: Use the updated spatiotemporal query representation as the query, the video frame sequence representation as the key and value, calculate the cross attention mechanism, and obtain the spatiotemporal representation of the object:

[0080] ,

[0081] in, represents the updated spatiotemporal query representation, The value range is , The value range is .

[0082] Step 3032: Fuse the object spatiotemporal representation with the updated spatiotemporal query representation to obtain an updated object spatiotemporal representation:

[0083] ,

[0084] in, The value range is , The value range is .

[0085] Step 3033: input the updated spatiotemporal representation of the object into the multilayer perceptron network and add it to the updated spatiotemporal representation of the object to obtain the final spatiotemporal representation of the object;

[0086] ,

[0087] in, The value range is , The value range is .

[0088] Step 4: Based on the final spatiotemporal representation of the object, by alternately performing temporal attention, spatial attention, and channel attention, model the fine-grained interaction information of the object at the temporal, spatial, and semantic levels, enhance the ability to understand the spatiotemporal interaction information of the video from multiple dimensions, and obtain the local video segment representation ;

[0089] In order to capture the spatiotemporal interaction relationship between objects and enhance the understanding of video content, a spatiotemporal representation multi-dimensional interaction module is designed, in which temporal attention, spatial attention, and channel attention are executed alternately.

[0090] The specific steps include:

[0091] Step 401: Calculate temporal self-attention for each object in the final spatiotemporal representation of the object, aiming to capture its temporal information and obtain a temporally enhanced video clip frame sequence representation:

[0092] ,

[0093] in, , is the normalization function.

[0094] Step 402: In the temporally enhanced video frame sequence representation, spatial self-attention is calculated separately for each frame to capture the interaction information between objects, and a spatiotemporally enhanced video clip frame sequence representation is obtained:

[0095] ,

[0096] in, , is the normalization function.

[0097] Step 403: Group the channel dimensions of the spatiotemporal enhanced video clip frame sequence representation and perform self-attention calculation within each group to further enhance the semantic level interaction and obtain a spatiotemporal-channel jointly enhanced video frame sequence representation.

[0098] In many cases, the data feature dimensionality values ​​will be high. To mitigate the inherent quadratic complexity of self-attention involving the channel dimension, this implementation divides the channels into multiple groups and performs self-attention calculations within each group.

[0099] Assumptions Indicates the number of groups, represents the feature dimension in each group, :

[0100] ,

[0101] in, , is the normalization function.

[0102] Step 5: Extract the global representation of the video segment;

[0103] In this embodiment, the global representation of the video clip is extracted by pre-training the video understanding model InternVideo2, which is denoted as .

[0104] Step 6: Fuse the global representation of the video clip with the temporal-spatial-channel jointly enhanced video clip frame sequence representation to obtain the final video clip representation.

[0105] In this embodiment, during fusion, Make a copy and expand it to the same dimension as consistent with; then, The final fine-grained video representation is obtained by adding .

[0106] Execute steps 2 to 6 for each video clip to obtain the representation of all video clips, denoted as { };

[0107] Step 7: Encode the text query statement input by the user to obtain a query representation;

[0108] In this embodiment, the query representation is extracted by pre-training the natural language understanding large model BERT, which is denoted as .

[0109] Step 8: Determine the target video segment based on the similarity between the video segment representation and the query representation.

[0110] In this embodiment, by calculating any video clip With query representation The cosine similarity of , the segment with the largest score value is taken as the target video segment.

[0111] The above scheme takes into account multiple key factors, including temporal interaction behavior, spatial interaction behavior, and channel semantic perception-co-enhanced video clip representation, through the object spatiotemporal representation extraction network and the spatiotemporal representation multidimensional interaction network.

[0112] Among them, the object spatiotemporal representation extraction network can effectively capture the object information and object spatiotemporal interaction information in the video without the help of external target detection tools, improving the flexibility of video clip representation extraction and the completeness of video content understanding. The spatiotemporal representation multidimensional interaction network can jointly mine the temporal interaction behavior and spatial interaction behavior between objects, while effectively perceiving channel semantic information, and obtain video clip representation with enhanced fine-grained content understanding.

[0113] Embodiment 2

[0114] This embodiment provides a video segment positioning system based on fine-grained spatiotemporal correlation modeling, including:

[0115] A video encoding module is used to perform multi-scale time window sampling on the video to obtain video clips of different time lengths, and to perform frame-level encoding on the video clips to obtain a frame representation sequence of the encoded video clips;

[0116] A spatiotemporal representation acquisition module, which is used to obtain the spatiotemporal representation of the object from the video clip frame representation sequence according to the spatiotemporal query representation;

[0117] The local representation extraction module is used to model the fine-grained interaction information of objects at the temporal, spatial and semantic levels based on the spatiotemporal representation of objects by alternately performing temporal attention, spatial attention and channel attention, and enhance the ability to understand the spatiotemporal interaction information of videos from multiple dimensions, thus obtaining a frame sequence representation of video clips jointly enhanced by time, space and channels;

[0118] A global representation extraction module for extracting a global representation of a video clip;

[0119] A representation fusion module is used to fuse the global representation of the video segment with the time-space-channel joint enhanced video segment frame sequence representation to obtain the final video segment representation;

[0120] The target video segment determination module is used to encode the text query sentence input by the user to obtain the query representation; and determine the target video segment according to the similarity between the video segment representation and the query representation.

[0121] Embodiment 3

[0122] This embodiment provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the steps in the video segment positioning method based on fine-grained spatiotemporal association modeling as described above are implemented.

[0123] Embodiment 4

[0124] This embodiment provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the steps in the method for locating video segments based on fine-grained spatiotemporal correlation modeling as described above are implemented.

[0125] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A video segment localization method based on fine-grained spatiotemporal correlation modeling, characterized in that: The steps include: Acquire video data, perform multi-scale time window sampling on the video to obtain video clips of different time lengths, and perform frame-level encoding on the video clips to obtain a frame representation sequence of the encoded video clips; According to the spatiotemporal query representation, the spatiotemporal representation of the object is obtained from the sequence of frame representations of the video clip; Based on the spatiotemporal representation of objects, the fine-grained interaction information of objects at the temporal, spatial and semantic levels is captured by alternately performing temporal attention, spatial attention and channel attention, and the ability to understand spatiotemporal interaction information of videos is enhanced from multiple dimensions. The frame sequence representation of video clips jointly enhanced by time, space and channels is obtained, including: Temporal self-attention is calculated for each object in the spatiotemporal representation of the object to capture the temporal information and obtain a temporally enhanced frame sequence representation of the video clip; For each frame in the temporally enhanced video clip frame sequence representation, spatial self-attention is calculated separately to capture the interaction information between objects and obtain the spatiotemporally enhanced video clip frame sequence representation; The channel dimension of the spatiotemporal enhanced video clip frame sequence representation is grouped, and self-attention calculation is performed within each group to further enhance the semantic level interaction, thereby obtaining the spatiotemporal-channel jointly enhanced video clip frame sequence representation; and the global representation of the video clip is extracted; The global representation of the video clip is fused with the frame sequence representation of the video clip jointly enhanced in time, space and channel to obtain the final video clip representation; Encode the text query statement input by the user to obtain a query representation; According to the similarity between the video segment representation and the query representation, the target video segment is determined.

2. The video segment positioning method based on fine-grained spatiotemporal correlation modeling as claimed in claim 1, characterized in that: The step of obtaining the object spatiotemporal representation from the video clip frame representation sequence according to the spatiotemporal query representation includes: According to the temporal query representation and the spatial query representation, a first spatiotemporal query representation is obtained by copying the spatial query representation to each temporal query representation; Perform self-attention calculation on the first spatiotemporal query representation to obtain a second spatiotemporal query representation that can capture the spatiotemporal correlation relationship; According to the second spatiotemporal query representation, cross-attention is used to obtain the spatiotemporal representation of the object from the sequence of frame representations of the video clip.

3. The video segment positioning method based on fine-grained spatiotemporal correlation modeling as claimed in claim 2, characterized in that: The performing self-attention calculation on the first spatiotemporal query representation to obtain a second spatiotemporal query representation capable of capturing the spatiotemporal correlation relationship includes: Perform spatial self-attention calculation on the first spatiotemporal query representation to capture the spatial interaction information of objects and obtain a spatiotemporal query representation with enhanced spatial perception; Perform temporal self-attention calculation on the spatial-perception enhanced spatiotemporal query representation to obtain the spatial-perception enhanced spatiotemporal query representation; The spatiotemporal query representation enhanced with spatiotemporal awareness is fused with the first spatiotemporal query representation to obtain a second spatiotemporal query representation.

4. The video segment positioning method based on fine-grained spatiotemporal correlation modeling as claimed in claim 2, characterized in that: The step of obtaining the object spatiotemporal representation from the video segment frame representation sequence using cross attention according to the second spatiotemporal query representation includes: Taking the second spatiotemporal query representation as the query and the video clip frame sequence representation as the key and value, calculate the cross attention mechanism to obtain the spatiotemporal representation of the object; fusing the object spatiotemporal representation with the second spatiotemporal query representation to obtain an updated object spatiotemporal representation; The updated spatiotemporal representation of the object is input into the multi-layer perceptron network and added to the updated spatiotemporal representation of the object to obtain the final spatiotemporal representation of the object.

5. The video segment positioning method based on fine-grained spatiotemporal correlation modeling as claimed in claim 1, characterized in that: When extracting the global representation of video clips, the pre-trained video understanding large model InternVideo2 is used.

6. The video segment positioning method based on fine-grained spatiotemporal correlation modeling as claimed in claim 1, characterized in that: When fusing the global representation of a video clip with the time-space-channel jointly enhanced video clip frame sequence representation, the global representation of the video clip is copied, expanded to a dimension consistent with the time-space-channel jointly enhanced video frame sequence representation, and then added to the time-space-channel jointly enhanced video frame sequence representation to obtain the final fine-grained video clip representation.

7. The video segment positioning method based on fine-grained spatiotemporal correlation modeling as claimed in claim 1, characterized in that: When extracting user text query representations, the pre-trained natural language understanding large model BERT is used.

8. The video segment positioning method based on fine-grained spatiotemporal correlation modeling as claimed in claim 1, characterized in that: The target video segment is determined based on the maximum cosine similarity score between the video segment and the query representation.

9. A video segment positioning system based on fine-grained spatiotemporal correlation modeling, characterized in that: include: A video encoding module is used to perform multi-scale time window sampling on the video to obtain video clips of different time lengths, and to perform frame-level encoding on the video clips to obtain a frame representation sequence of the encoded video clips; A spatiotemporal representation acquisition module, which is used to obtain the spatiotemporal representation of the object from the video clip frame representation sequence according to the spatiotemporal query representation; The local representation extraction module is used to capture the fine-grained interaction information of objects at the temporal, spatial and semantic levels based on the spatiotemporal representation of objects by alternately performing temporal attention, spatial attention and channel attention, and enhance the ability to understand the spatiotemporal interaction information of videos from multiple dimensions. The video clip frame sequence representation with joint enhancement of time, space and channel is obtained, which specifically includes: Temporal self-attention is calculated for each object in the spatiotemporal representation of the object to capture the temporal information and obtain a temporally enhanced frame sequence representation of the video clip; For each frame in the temporally enhanced video clip frame sequence representation, spatial self-attention is calculated separately to capture the interaction information between objects and obtain the spatiotemporally enhanced video clip frame sequence representation; The channel dimension of the spatiotemporal enhanced video clip frame sequence representation is grouped, and self-attention calculation is performed within each group to further enhance the semantic level interaction, thereby obtaining a spatiotemporal-channel jointly enhanced video clip frame sequence representation; a global representation extraction module is used to extract the global representation of the video clip; A representation fusion module is used to fuse the global representation of the video segment with the time-space-channel joint enhanced video segment frame sequence representation to obtain the final video segment representation; The target video segment determination module is used to encode the text query sentence input by the user to obtain the query representation; and determine the target video segment according to the similarity between the video segment representation and the query representation.

Images