Video retrieval method and system based on hyperbolic space representation, terminal equipment and medium

Abstract

The invention discloses a video retrieval method and system based on hyperbolic space representation, terminal equipment and a medium, and relates to the technical field of video retrieval, the method comprises the following steps: performing feature extraction and sparse sampling on a video to be retrieved, and processing through a full connection layer FC and HLFormer module to obtain hyperbolic space representation enhanced video features; the method comprises the following steps: performing feature extraction on a to-be-queried text, and obtaining text statement features through a full connection layer FC, a Transform module and an attention pooling module Attention; and performing cosine similarity calculation through the text statement features and the video features, taking the highest video clip cosine similarity as a retrieval score, and selecting the video with the highest retrieval score as a target video corresponding to the to-be-queried text. According to the method, video features are coded and dynamically fused into multi-scale features in a mixed space through the HLFormer module, the defect of Euclidean space in the aspect of video hierarchical modeling is overcome through hyperbolic space learning, and the video retrieval performance is improved.

Description

Technical Field

[0001] This invention relates to the field of video retrieval technology, and in particular to a video retrieval method, system, terminal device and medium based on hyperbolic space representation. Background Technology

[0002] With the continuous advancement of technology and the widespread application of video sharing platforms, video has become a core form of information dissemination in modern society. Therefore, T2VR (Text-to-Video Retrieval) has become an important research area in academia and industry. Compared to traditional T2VR, PRVR (Partially Relevant Video Retrieval) is closer to real-world application scenarios and presents a greater challenge. Existing PRVR methods approach the task from multiple angles, aiming to enhance retrieval performance. The MS-SL method defines the PRVR task as multi-instance learning and provides a strong benchmark by explicitly modeling video segment information through a multi-scale sliding window. GMMFormer (Gaussian-Mixture-Model based Transformer) and PEAN propose implicit video segment modeling strategies to improve retrieval efficiency. DLDKD has achieved significant results through dynamic distillation of pre-trained CLIP models. BGM-Net employs an instance-level matching scheme to pair queries with videos. LH enhances the quality of query representation by incorporating contextual descriptions. However, these methods all rely on encoding the video in Euclidean space. The flat geometry of Euclidean space often leads to the incorrect representation of the hierarchical structure information of the video, thus ignoring the hierarchical semantics within the video and ultimately producing suboptimal video temporal modeling results.

[0003] Therefore, existing technologies urgently need to fully exploit the prior information of video hierarchical structure in order to further improve retrieval performance. Summary of the Invention

[0004] The technical problem this invention aims to solve is the insufficient ability of existing video retrieval technologies in hierarchical feature modeling and spatial representation. Traditional Euclidean space struggles to effectively characterize the hierarchical semantic structure of video content, such as the hierarchical relationships between scenes, objects, and actions, resulting in limited ability to capture long-distance dependencies and multi-scale features. The flat geometric properties of Euclidean space often lead to the incorrect representation of the hierarchical structure information of videos, thus ignoring the hierarchical semantics within the video and ultimately producing suboptimal video temporal modeling results. Furthermore, existing methods lack dynamic adjustment mechanisms for feature interactions in multimodal feature fusion, such as cross-modal matching of video and text, affecting retrieval accuracy. Therefore, existing technologies urgently need to fully exploit the prior information of video hierarchical structure to further improve retrieval performance.

[0005] An effective solution is urgently needed.

[0006] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is as follows:

[0007] In a first aspect, the present invention provides a video retrieval method based on hyperbolic space representation, the method comprising:

[0008] The first feature network is used to extract features from the video to be retrieved to obtain preprocessed video features, and the preprocessed video features are sparsely sampled to obtain several video segments.

[0009] All video clips are input into a fully connected layer (FC) and an HLFormer module to obtain video features enhanced with hyperbolic space representation.

[0010] The second feature network is used to extract features from the query text to obtain preprocessed text features.

[0011] The preprocessed text features are input into a fully connected layer (FC), a Transformer module, and an attention pooling module (Attention) to obtain text sentence features.

[0012] Cosine similarity is calculated using the text sentence features and the video features, and the highest cosine similarity of the video segment is used as the retrieval score.

[0013] By calculating the search scores corresponding to the query text and several videos to be retrieved, the video with the highest search score is selected as the target video corresponding to the query text.

[0014] In one implementation, the retrieval score expression is:

[0015]

[0016] Where q represents the text statement feature, V c ∈Rm×d Let m represent the number of video segments and d represent the dimension of the video features. Let S represent the features of the m-th video segment, max represent selecting the maximum value, cos represent cosine similarity calculation, and S c This indicates the search score.

[0017] In one implementation, the first feature network is a pre-trained I3D or ResNet model, the second feature network is a pre-trained 1024-dimensional RoBERTa model, the fully connected layer (FC) consists of a layer normalization layer, a linear layer, and a ReLU activation function, and the Transformer module consists of a self-attention module and a feedforward neural network.

[0018] In one implementation, the attention pooling module pools the output of the Transformer module to obtain text sentence features q∈R. d ,Right now:

[0019]

[0020] Where w represents the trainable vector, Q represents the feature matrix output by the Transformer module, and N q Indicates the length of the statement. Indicates attention weights, Let q represent the attention weight at position i. i This represents the context feature vector at the i-th position output by the Transformer module.

[0021] In one implementation, the HLFormer module includes N L Lorenz attention modules and N E N Euclidean attention modules arranged in parallel form a single unit. O The system includes a Gaussian attention module and a mean-guided adaptive interaction module. The mean-guided adaptive interaction module includes a cross-attention module, which comprises a cross-attention layer (CA) and a fully connected layer (FC). The mean-guided adaptive interaction module obtains the output of the Gaussian attention module and calculates the aggregate weights through the cross-attention module.

[0022]

[0023]

[0024] Where, φ∈R 1×d Indicates a global query. Let w represent the output of the i-th Gaussian attention module, M represent the number of video segments, and w represent the output of the i-th Gaussian attention module. i ∈R M Let τ represent the aggregate weights of the i-th Gaussian attention module, and let τ represent the temperature factor. This represents the aggregation feature of video segment j.

[0025] In one implementation, the Euclidean attention module is constructed by replacing the self-attention in the Transformer module with Euclidean Gaussian attention, and the expression for the Euclidean Gaussian attention is:

[0026]

[0027] Where, x∈R M×d This represents the input feature embedding, where M represents the number of video segments and d represents the feature dimension. Represents a Gaussian matrix σ 2 Represents variance.

[0028] W q W k W v Denotes the linear projection matrix, used for query, key, and value mapping respectively, d h ⊙ represents the potential attention dimension, and ⊙ represents element-wise multiplication.

[0029] In one implementation, the Lorentz attention module is constructed by replacing the self-attention in the Transformer module with Lorentz Gaussian attention. The Lorentz attention module maps video features from Euclidean space to hyperbolic space using a hyperbolic exponential map, enhances features through a Lorentz linear layer, and captures video hierarchical information in hyperbolic space using Lorentz self-attention. The expression for the hyperbolic exponential map is:

[0030]

[0031] Where x represents the input feature, W represents the trainable weight matrix, p represents the trainable vector parameters, and b and b ′ λ represents the bias term, λ>0 is used to adjust the scaling range, and h represents the activation function;

[0032] The expression for the Lorentz linear layer is:

[0033]

[0034] The Lorentz-Gaussian attention expression is:

[0035]

[0036] The expression for the squared Lorentz distance is: S ij μ represents the attention weight between positions i and j. i q represents the feature representation of position i after attention-weighted aggregation. i Let k represent the i-th query feature vector. j Let v represent the feature vector of the j-th key. j Let K represent the feature vector of the j-th value, and K represent the key feature set.

[0037] Secondly, embodiments of the present invention also provide a video retrieval system based on hyperbolic space representation, the system comprising:

[0038] The video segment acquisition module is used to extract features from the video to be retrieved through a first feature network to obtain preprocessed video features, and to perform sparse sampling on the preprocessed video features to obtain several video segments.

[0039] The video feature acquisition module is used to input all video clips into the fully connected layer FC and HLFormer module to obtain video features enhanced with hyperbolic space representation;

[0040] The preprocessed text feature acquisition module is used to extract features from the query text through the second feature network to obtain preprocessed text features.

[0041] The text sentence feature acquisition module is used to input the preprocessed text features into the fully connected layer (FC), the Transformer module, and the attention pooling module (Attention) to obtain the text sentence features.

[0042] The retrieval score calculation module is used to calculate the cosine similarity using the text sentence features and the video features, and take the highest video segment cosine similarity as the retrieval score.

[0043] The target video acquisition module is used to calculate the search scores corresponding to the query text and several searchable videos, and select the video with the highest search score as the target video corresponding to the query text.

[0044] Thirdly, embodiments of the present invention also provide a terminal device, the terminal device including a memory, a processor, and a video retrieval program based on hyperbolic space representation stored in the memory and executable on the processor, wherein when the processor executes the video retrieval program based on hyperbolic space representation, it implements the steps of the video retrieval method based on hyperbolic space representation as described in any of the above schemes.

[0045] Fourthly, embodiments of the present invention also provide a computer-readable storage medium storing a video retrieval program based on hyperbolic space representation, wherein when the video retrieval program based on hyperbolic space representation is executed by a processor, it implements the steps of the video retrieval method based on hyperbolic space representation as described in any of the above schemes.

[0046] Beneficial Effects: This invention discloses a video retrieval method, system, terminal device, and medium based on hyperbolic space representation, relating to the field of video retrieval technology. The method includes firstly extracting features from the video to be retrieved, performing sparse sampling, and then processing these features through a fully connected layer (FC) and an HLFormer module to obtain hyperbolic space-enhanced video features. Next, feature extraction is performed on the query text, and these features are processed through a fully connected layer (FC), a Transformer module, and an Attention module to obtain text sentence features. Finally, cosine similarity is calculated between the text sentence features and the video features. The highest cosine similarity score is used as the retrieval score, and the video with the highest retrieval score is selected as the target video corresponding to the query text. This invention uses the HLFormer module to encode video features in a hybrid space and dynamically fuse multi-scale features, utilizing hyperbolic space learning to compensate for the shortcomings of Euclidean space in video hierarchical modeling, thereby improving video retrieval performance. Attached Figure Description

[0047] Figure 1 A flowchart illustrating a specific implementation of the video retrieval method based on hyperbolic space representation provided in this invention.

[0048] Figure 2 This is a branch structure diagram of the video retrieval method based on hyperbolic space representation provided in the embodiments of the present invention.

[0049] Figure 3 This is a structural diagram of the fully connected layer (FC) in the video retrieval method based on hyperbolic space representation provided in the embodiments of the present invention.

[0050] Figure 4 This is a structural diagram of the HLFormer module in the video retrieval method based on hyperbolic space representation provided in this embodiment of the invention.

[0051] Figure 5 This is a block diagram of the video retrieval system based on hyperbolic space representation provided in an embodiment of the present invention.

[0052] Figure 6 This is a block diagram illustrating the internal structure of the terminal device provided in an embodiment of the present invention. Detailed Implementation

[0053] To make the objectives, technical solutions, and effects of this invention clearer and more explicit, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0054] The flowchart shown in the attached diagram is for illustrative purposes only and does not necessarily include all content, operations, or steps, nor does it require execution in the described order. For example, some operations or steps can be broken down, combined, or partially merged, so the actual execution order may change depending on the actual situation.

[0055] It should be understood that the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the invention. As used in this specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.

[0056] It should be understood that, in order to clearly describe the technical solutions of the embodiments of the present invention, the terms "first" and "second" are used in the embodiments of the present invention to distinguish identical or similar items with essentially the same function and effect. For example, "first control information" and "second control information" are only used to distinguish different control information and do not limit their order.

[0057] Those skilled in the art will understand that the words "first" and "second" do not limit the quantity or the order of execution, and that the words "first" and "second" do not necessarily imply that they are different.

[0058] It should also be understood that the term "and / or" as used in this specification and the appended claims refers to any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0059] With the continuous advancement of technology and the widespread application of video sharing platforms, video has become a core form of information dissemination in modern society. Therefore, T2VR (Text-to-Video Retrieval) has become an important research area in academia and industry. Compared to traditional T2VR, PRVR (Partially Relevant Video Retrieval) is closer to real-world application scenarios and presents a greater challenge. Existing PRVR methods approach the task from multiple angles, aiming to enhance retrieval performance. The MS-SL method defines the PRVR task as multi-instance learning and provides a strong benchmark by explicitly modeling video segment information through a multi-scale sliding window. GMMFormer (Gaussian-Mixture-Model based Transformer) and PEAN propose implicit video segment modeling strategies to improve retrieval efficiency. DLDKD has achieved significant results through dynamic distillation of pre-trained CLIP models. BGM-Net employs an instance-level matching scheme to pair queries with videos. LH enhances the quality of query representation by incorporating contextual descriptions. However, these methods all rely on encoding the video in Euclidean space. The flat geometry of Euclidean space often leads to the incorrect representation of the hierarchical structure information of the video, thus ignoring the hierarchical semantics within the video and ultimately producing suboptimal video temporal modeling results.

[0060] This embodiment provides a video retrieval method based on hyperbolic space representation, such as... Figure 1 As shown, the specific steps include the following:

[0061] Step S100: Extract features from the video to be retrieved using the first feature network to obtain preprocessed video features, and perform sparse sampling on the preprocessed video features to obtain several video segments.

[0062] Step S200: Input all video clips into the fully connected layer FC and HLFormer module to obtain video features enhanced with hyperbolic space representation.

[0063] Step S300: Extract features from the query text using the second feature network to obtain preprocessed text features.

[0064] Step S400: Input the preprocessed text features into the fully connected layer FC, the Transformer module and the attention pooling module Attention to obtain text sentence features.

[0065] Step S500: Calculate the cosine similarity using the text sentence features and the video features, and use the highest video segment cosine similarity as the retrieval score.

[0066] Step S600: By calculating the search scores corresponding to the text to be queried and several videos to be retrieved, select the video with the highest search score as the target video corresponding to the text to be queried.

[0067] In this embodiment, from Figure 2 As shown in the branch structure diagram, the video retrieval method includes two branches: a video branch and a text branch.

[0068] For the videos to be retrieved, video branching is used. Specifically, a preprocessed feature vector is first obtained through a feature network. Next, video segment information is obtained through sparse sampling. This sparse sampling includes average pooling of adjacent video frames along the temporal dimension. Finally, the video features are enhanced with hyperbolic space representation through a fully connected layer (FC) and an HLFormer block.

[0069] For the provided text query, a text branch is used for processing. Specifically, preprocessed text features are first obtained through a feature network, then passed through a fully connected layer (FC) and a Transformer module, and finally the final text sentence features are obtained through an attention pooling module.

[0070] When performing similarity retrieval, cosine similarity is calculated using text sentence features and video features. The highest score is taken as the retrieval score, and finally, the videos corresponding to the text are obtained by sorting them.

[0071] In one implementation, the retrieval score expression is:

[0072]

[0073] Where q represents the text statement feature, V c ∈R m×d Let m represent the number of video segments and d represent the dimension of the video features. Let S represent the features of the m-th video segment, max represent selecting the maximum value, cos represent cosine similarity calculation, and S c This indicates the search score.

[0074] In this embodiment, specific parameters are processed and calculated tailored to the specific video retrieval scenario. The video feature dimension is set to a reasonable value that balances computational efficiency and accuracy. Features for each video segment are extracted meticulously, and cosine similarity is calculated to measure the similarity between text sentence features and video segment features. Then, a maximum value function is used to select the maximum value as the retrieval score. This process ensures that the retrieval score accurately reflects the matching degree between the text and video segments, providing a reliable basis for selecting the target video that best matches the query text, effectively improving the accuracy and effectiveness of video retrieval.

[0075] In one implementation, the first feature network is a pre-trained I3D or ResNet model, the second feature network is a pre-trained 1024-dimensional RoBERTa model, the fully connected layer (FC) consists of a layer normalization layer, a linear layer, and a ReLU activation function, and the Transformer module consists of a self-attention module and a feedforward neural network.

[0076] In this embodiment, the first feature network is a pre-trained I3D or ResNet model, extracting video features using pre-trained I3D or ResNet features provided by MS-SL. The second feature network is a pre-trained 1024-dimensional RoBERTa model, extracting text features using pre-trained 1024-dimensional RoBERTa features provided by MS-SL. Figure 3 As shown, the fully connected (FC) layer consists of simple layer normalization, linear layers, and ReLU activation functions. The Transformer module is a common Transformer module, composed of a self-attention module and a feedforward neural network.

[0077] In one implementation, the attention pooling module pools the output of the Transformer module to obtain text sentence features q∈R. d ,Right now:

[0078] a q =softmax(wQ) T )

[0079] Where w represents the trainable vector, Q represents the feature matrix output by the Transformer module, and N q Indicates the length of the statement. Indicates attention weights. Let q represent the attention weight at position i. i This represents the context feature vector at the i-th position output by the Transformer module.

[0080] In this embodiment, the trainable vector w is initialized with an appropriate dimension to ensure effective computation with the feature matrix Q output by the Transformer module. This is achieved by calculating wQ. T The attention score is obtained, and then normalized using the softmax function to determine the attention weight at each position. For example, for statement length N q For each position i within the range, calculate the corresponding... Then, it is compared with the context feature vector q at the i-th position output by the Transformer module. i The products are multiplied, and finally the results of the multiplications at all positions are summed to obtain the sentence features q that accurately represent the semantics of the text. This process effectively focuses on key semantic information by dynamically adjusting the weights of features at each position, thereby improving the expressive power of the text sentence features q for the semantics of the text, which can then be used for subsequent cross-modal matching of video and text.

[0081] In one implementation, the HLFormer module includes N L Lorenz attention modules and N E N Euclidean attention modules arranged in parallel form a single unit. O The system includes a Gaussian attention module and a mean-guided adaptive interaction module. The mean-guided adaptive interaction module includes a cross-attention module, which comprises a cross-attention layer (CA) and a fully connected layer (FC). The mean-guided adaptive interaction module obtains the output of the Gaussian attention module and calculates the aggregate weights through the cross-attention module.

[0082]

[0083]

[0084] Where, φ∈R 1×d Indicates a global query. Let w represent the output of the i-th Gaussian attention module, M represent the number of video segments, and w represent the output of the i-th Gaussian attention module. i ∈R M Let τ represent the aggregate weights of the i-th Gaussian attention module, and let τ represent the temperature factor. This represents the aggregation feature of video segment j.

[0085] In this embodiment, HLFormer forms the core of our method. For example... Figure 4As shown, this module comprises three key components: the Euclidean Attention Block (EAB), which captures fine visual features in Euclidean space; the Lorentz Attention Block (LAB), which embeds the video into a hyperbolic Lorentz space, effectively capturing the video's hierarchical structure while alleviating the limitations of Euclidean space hierarchical modeling; and the Mean-Guided Adaptive Interaction Module (MAIM), which dynamically fuses features from the hybrid space.

[0086] N L Lorenz attention modules and N E N Euclidean attention modules are arranged in parallel to generate N O A Gaussian attention module is used for video embedding in a multi-scale mixed space. To effectively integrate these features, a Mean-Guided Adaptive Interaction Module is proposed. This module utilizes global mean pooling features to learn adaptive aggregation weights for dynamic fusion. Specifically, a global query is first obtained, and aggregation weights are calculated through a Cross Attention Block, which consists of a Cross Attention layer (CA) and a fully connected layer (FC).

[0087] In one implementation, the Euclidean attention module is constructed by replacing the self-attention in the Transformer module with Euclidean Gaussian attention, and the expression for the Euclidean Gaussian attention is:

[0088]

[0089] Where, x∈R M×d This represents the input feature embedding, where M represents the number of video segments and d represents the feature dimension. Represents a Gaussian matrix σ 2 Represents variance.

[0090] W q W k W v Denotes the linear projection matrix, used for query, key, and value mapping respectively, d h ⊙ represents the potential attention dimension, and ⊙ represents element-wise multiplication.

[0091] In this embodiment, by changing σ, feature interactions at different scales can be modeled, thereby generating video features with multiple receptive fields. Specifically, several different σ values ​​are set, such as a smaller σ1, a medium σ2, and a larger σ3, to achieve refined modeling of multi-scale feature interactions. For a smaller σ1, the Gaussian matrix... The distribution of σ² is more concentrated, allowing attention to focus on the interaction of local features, capturing subtle details such as texture and motion within video clips. A moderate σ² provides adequate coverage, effectively modeling semantic relationships between adjacent clips, such as the smooth transition of actions. A larger σ³ broadens the scope of the Gaussian matrix, capturing the overall video scene and long-distance dependencies. In this way, video features generated under different σ values ​​possess local, mesoscopic, and global receptive fields, respectively. Fusing these multi-receptive-field video features enables the model to comprehensively capture multi-level information from details to the whole in the video, significantly improving its ability to represent complex video content and providing richer, more discriminative feature representations for subsequent video retrieval.

[0092] In one implementation, the Lorentz attention module is constructed by replacing the self-attention in the Transformer module with Lorentz Gaussian attention. The Lorentz attention module maps video features from Euclidean space to hyperbolic space using a hyperbolic exponential map, enhances features through a Lorentz linear layer, and captures video hierarchical information in hyperbolic space using Lorentz self-attention. The expression for the hyperbolic exponential map is:

[0093]

[0094] Where x represents the input feature, W represents the trainable weight matrix, p represents the trainable vector parameters, and b and b ′ λ represents the bias term, λ>0 is used to adjust the scaling range, and h represents the activation function;

[0095] The expression for the Lorentz linear layer is:

[0096]

[0097] The Lorentz-Gaussian attention expression is:

[0098]

[0099] The expression for the squared Lorentz distance is: S ij μ represents the attention weight between positions i and j. i q represents the feature representation of position i after attention-weighted aggregation. i Let k represent the i-th query feature vector. jLet v represent the feature vector of the j-th key. j Let K represent the feature vector of the j-th value, and K represent the key feature set.

[0100] In this embodiment, as Figure 4 As shown, the Lorentz attention mechanism also uses a Gaussian gating mechanism to obtain multi-scale features. Specifically, in Lorentz attention, the Gaussian matrix is ​​adjusted...

[0101] The variance σ 2 This generates multi-scale gated signals. When σ 2 When σ is small, the Gaussian matrix is ​​concentrated, focusing on local feature interactions and capturing detailed information such as subtle movements of objects; when σ is small... 2 When the value is large, the distribution is broad, focusing on the global hierarchical structure, such as the relationship between the scene and objects. These gated signals are related to the squared Lorentz distance. Element-wise multiplication, dynamically adjusting attention weight S ij The calculation. For example, when processing videos with multiple scene transitions, the small σ... 2 Pay attention to the details of the actions within the scene, large σ 2 Focusing on the hierarchical transitions between scenes, making μ i It can integrate multi-scale information from local to global levels, enhance the representation ability of complex hierarchical structures in videos, and thus improve the accuracy of video retrieval.

[0102] This paper models the hierarchical information of videos using hyperbolic space and proposes a Lorentz attention module to fully mine and utilize the prior information of the hierarchical videos within the model. Furthermore, a Gaussian gating mechanism is introduced to control the role of the Lorentz attention module in multi-scale hyperbolic video information fusion, thereby improving the model's expressive power. Finally, a mean-guided adaptive interaction module is employed to achieve dynamic adaptive fusion of mixed video information in Euclidean and hyperbolic spaces, resulting in fewer parameters and higher computational efficiency.

[0103] In summary, under the technical solution of the above embodiments, the HLFormer hybrid spatial modeling framework is constructed, and hyperbolic space representation is innovatively introduced to achieve accurate modeling of the hierarchical semantic structure of videos. When constructing the HLFormer module, a Gaussian attention module group with parallel Lorentz and Euclidean attention modules is used, combined with a mean-guided adaptive interaction module to dynamically calculate aggregation weights, ensuring the integrity of multi-spatial features and the logicality of interaction. The introduction of hyperbolic space breaks through the limitations of flat geometry in Euclidean space. Through hyperbolic exponential mapping and Lorentz linear layers, it effectively captures the hierarchical relationships and long-distance dependencies of scenes, objects, and actions in videos, solving the problem of insufficient representation of hierarchical semantic structures in traditional methods. The dynamic feature interaction mechanism achieves fine-grained alignment of text and video segment semantics through global query-guided attention weight calculation, significantly improving cross-modal matching accuracy. The video retrieval model constructed by this method not only enhances the hierarchical features of hyperbolic space representation in the feature extraction stage but also achieves accurate matching of video segments and text through cosine similarity calculation, significantly improving the reliability of retrieval scores. In practical applications, it can effectively improve the retrieval performance of scenarios such as video sharing platforms and educational resource retrieval, provide a more scientific technical solution for cross-modal information retrieval, and promote the development of video retrieval technology towards higher efficiency and intelligence.

[0104] like Figure 5 As shown in the figure, this embodiment of the invention provides a video retrieval system based on hyperbolic space representation. The system includes: a video segment acquisition module 10, a video feature acquisition module 20, a preprocessed text feature acquisition module 30, a text sentence feature acquisition module 40, a retrieval score calculation module 50, and a target video acquisition module 60.

[0105] Specifically, the video segment acquisition module 10 is used to extract features from the video to be retrieved through a first feature network to obtain preprocessed video features, and to perform sparse sampling on the preprocessed video features to obtain several video segments; the video feature acquisition module 20 is used to input all video segments into a fully connected layer (FC) and an HLFormer module to obtain hyperbolic space representation-enhanced video features; the preprocessed text feature acquisition module 30 is used to extract features from the query text through a second feature network to obtain preprocessed text features; the text sentence feature acquisition module 40 is used to input the preprocessed text features into a fully connected layer (FC), a Transformer module, and an Attention pooling module to obtain text sentence features; the retrieval score calculation module 50 is used to calculate the cosine similarity between the text sentence features and the video features, and to take the highest video segment cosine similarity as the retrieval score; the target video acquisition module 60 is used to select the video with the highest retrieval score as the target video corresponding to the query text by calculating the retrieval scores corresponding to the query text and several videos to be retrieved.

[0106] Based on the above embodiments, the present invention also provides a terminal device, the principle block diagram of which can be as follows: Figure 6 As shown, the terminal device includes a processor, memory, network interface, display screen, and temperature sensor connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides the environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The network interface is used to communicate with external terminals via a network connection. When the computer program is executed by the processor, it implements a video retrieval method based on hyperbolic space representation. The display screen can be an LCD screen or an e-ink screen. The temperature sensor is pre-installed inside the terminal device to detect the operating temperature of the internal components.

[0107] Those skilled in the art will understand that Figure 6 The schematic diagram shown is only a partial structural diagram related to the present invention and does not constitute a limitation on the terminal device to which the present invention is applied. The specific terminal device may include more or fewer components than shown in the figure, or combine certain components, or have different component arrangements.

[0108] In one embodiment, a terminal device is provided, including a memory and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by one or more processors. The one or more programs include instructions for performing the following operations:

[0109] The first feature network is used to extract features from the video to be retrieved to obtain preprocessed video features, and the preprocessed video features are sparsely sampled to obtain several video segments.

[0110] All video clips are input into a fully connected layer (FC) and an HLFormer module to obtain video features enhanced with hyperbolic space representation.

[0111] The second feature network is used to extract features from the query text to obtain preprocessed text features.

[0112] The preprocessed text features are input into a fully connected layer (FC), a Transformer module, and an attention pooling module (Attention) to obtain text sentence features.

[0113] Cosine similarity is calculated using the text sentence features and the video features, and the highest cosine similarity of the video segment is used as the retrieval score.

[0114] By calculating the search scores corresponding to the query text and several videos to be retrieved, the video with the highest search score is selected as the target video corresponding to the query text.

[0115] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided by this invention can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and RAMbus dynamic RAM (RDRAM), etc.

[0116] In summary, this invention discloses a video retrieval method, system, terminal device, and medium based on hyperbolic space representation, relating to the field of video retrieval technology. The method includes: extracting features from the video to be retrieved, performing sparse sampling, and processing them through a fully connected layer (FC) and an HLFormer module to obtain hyperbolic space-enhanced video features; extracting features from the query text and processing them through a fully connected layer (FC), a Transformer module, and an attention pooling module to obtain text sentence features; calculating cosine similarity between the text sentence features and the video features, using the highest cosine similarity of the video segment as the retrieval score, and selecting the video with the highest retrieval score as the target video corresponding to the query text. This invention uses the HLFormer module to encode video features in a hybrid space and dynamically fuse multi-scale features, utilizing hyperbolic space learning to compensate for the shortcomings of Euclidean space in video hierarchical modeling, thereby improving video retrieval performance.

[0117] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0118] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A video retrieval method based on hyperbolic space representation, characterized in that, The method includes: The first feature network is used to extract features from the video to be retrieved to obtain preprocessed video features, and the preprocessed video features are sparsely sampled to obtain several video segments. All video clips are input into a fully connected layer (FC) and an HLFormer module to obtain video features enhanced with hyperbolic space representation. The second feature network is used to extract features from the query text to obtain preprocessed text features. The preprocessed text features are input into a fully connected layer (FC), a Transformer module, and an attention pooling module (Attention) to obtain text sentence features. Cosine similarity is calculated using the text sentence features and the video features, and the highest cosine similarity of the video segment is used as the retrieval score. By calculating the search scores between the query text and several videos, the video with the highest search score is selected as the target video corresponding to the query text.

2. The video retrieval method based on hyperbolic space representation according to claim 1, characterized in that, The retrieval score expression is: Where q represents the text statement feature, V c ∈R m×d Let m represent the number of video segments, d represent the dimension of the video features, and V represent the video features. cm ∈R 1×d Let S represent the features of the m-th video segment, max represent selecting the maximum value, cos represent cosine similarity calculation, and S c This indicates the search score.

3. The video retrieval method based on hyperbolic space representation according to claim 1, characterized in that, The first feature network is a pre-trained I3D or ResNet model, the second feature network is a pre-trained 1024-dimensional RoBERTa model, the fully connected layer (FC) consists of a layer normalization layer, a linear layer, and a ReLU activation function, and the Transformer module consists of a self-attention module and a feedforward neural network.

4. The video retrieval method based on hyperbolic space representation according to claim 1, characterized in that, The attention pooling module performs pooling processing on the output of the Transformer module to obtain the text sentence features q∈R. d ,Right now: Where w represents the trainable vector, Q represents the feature matrix output by the Transformer module, and N q Indicates the length of the statement. Indicates attention weights, Let q represent the attention weight at position i. i This represents the context feature vector at the i-th position output by the Transformer module.

5. The video retrieval method based on hyperbolic space representation according to claim 1, characterized in that, The HLFormer module includes N L Lorenz attention modules and N E N Euclidean attention modules arranged in parallel form a single unit. O The system includes a Gaussian attention module and a mean-guided adaptive interaction module. The mean-guided adaptive interaction module includes a cross-attention module, which comprises a cross-attention layer (CA) and a fully connected layer (FC). The mean-guided adaptive interaction module obtains the output of the Gaussian attention module and calculates the aggregate weights through the cross-attention module. Where, φ∈R 1×d Indicates a global query. Let w represent the output of the i-th Gaussian attention module, M represent the number of video segments, and w represent the output of the i-th Gaussian attention module. i ∈R M Let τ represent the aggregate weights of the i-th Gaussian attention module, and let τ represent the temperature factor. This represents the aggregation feature of video segment j.

6. The video retrieval method based on hyperbolic space representation according to claim 5, characterized in that, The Euclidean attention module is constructed by replacing the self-attention in the Transformer module with Euclidean Gaussian attention. The expression for the Euclidean Gaussian attention is: Where, x∈R M×d This represents the input feature embedding, where M represents the number of video segments and d represents the feature dimension. Represents a Gaussian matrix σ 2 W represents variance. q W k W v Denotes the linear projection matrix, used for query, key, and value mapping respectively, d h ⊙ represents the potential attention dimension, and ⊙ represents element-wise multiplication.

7. The video retrieval method based on hyperbolic space representation according to claim 6, characterized in that, The Lorentz attention module is constructed by replacing the self-attention in the Transformer module with Lorentz Gaussian attention. This Lorentz attention module maps video features from Euclidean space to hyperbolic space using a hyperbolic exponential map, enhances features through a Lorentz linear layer, and captures video hierarchical information in hyperbolic space using Lorentz self-attention. The expression for the hyperbolic exponential map is: Where x represents the input feature, W represents the trainable weight matrix, p represents the trainable vector parameters, and b and b ′ λ represents the bias term, λ>0 is used to adjust the scaling range, and h represents the activation function; The expression for the Lorentz linear layer is: The Lorentz-Gaussian attention expression is: The expression for the squared Lorentz distance is: S ij μ represents the attention weight between positions i and j. i q represents the feature representation of position i after attention-weighted aggregation. i Let k represent the i-th query feature vector. j Let v represent the feature vector of the j-th key. j Let K represent the feature vector of the j-th value, and K represent the key feature set.

8. A video retrieval system based on hyperbolic space representation, characterized in that, The system includes: The video segment acquisition module is used to extract features from the video to be retrieved through a first feature network to obtain preprocessed video features, and to perform sparse sampling on the preprocessed video features to obtain several video segments. The video feature acquisition module is used to input all video clips into the fully connected layer FC and HLFormer module to obtain video features enhanced with hyperbolic space representation; The preprocessed text feature acquisition module is used to extract features from the query text through the second feature network to obtain preprocessed text features. The text sentence feature acquisition module is used to input the preprocessed text features into the fully connected layer (FC), the Transformer module, and the attention pooling module (Attention) to obtain the text sentence features. The retrieval score calculation module is used to calculate the cosine similarity using the text sentence features and the video features, and take the highest video segment cosine similarity as the retrieval score. The target video acquisition module is used to calculate the search scores corresponding to the query text and several searchable videos, and select the video with the highest search score as the target video corresponding to the query text.

9. A terminal device, characterized in that, The terminal device includes a memory, a processor, and a video retrieval program based on hyperbolic space representation stored in the memory and executable on the processor. When the processor executes the video retrieval program based on hyperbolic space representation, it implements the steps of the video retrieval method based on hyperbolic space representation as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a video retrieval program based on hyperbolic space representation, which, when executed by a processor, implements the steps of the video retrieval method based on hyperbolic space representation as described in any one of claims 1-7.

Images