Fine-grained video text localization method and system based on cross-modal semantic hierarchy graph

The cross-modal semantic hierarchy graph method addresses the limitations of existing video text localization techniques by enhancing semantic alignment and interaction, leading to improved accuracy and generalization in video text localization.

CN118711112BActive Publication Date: 2024-12-03SICHUAN DAILY NETWORK MEDIA DEV CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411187880.8
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-08-28
Publication Date
2024-12-03
Estimated Expiration
2044-08-28

AI Technical Summary

Technical Problem

Existing video text localization techniques face challenges in capturing fine-grained semantic details and handling complex topological hierarchies due to selective bias and lack of multi-granularity modeling, limiting their generalization and accuracy in practical applications.

Method used

A fine-grained video text localization method using a cross-modal semantic hierarchy graph that performs hierarchical parsing and encoding of query and video features, followed by hierarchical semantic alignment and cross-modal temporal prediction to achieve refined query-video clip alignment.

Benefits of technology

Enhances the accuracy of video positioning by integrating global and local semantic interactions, improving generalization and reducing selective bias, thereby achieving more precise video text localization.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118711112B_ABST
    Figure CN118711112B_ABST
Patent Text Reader

Abstract

The present invention belongs to the technical field of video text positioning, and specifically discloses a fine-grained video text positioning method and system based on a cross-modal semantic hierarchy graph. By constructing a cross-modal semantic hierarchy graph from global to local, a more refined query-video clip alignment is achieved. This hierarchical semantic alignment can maximize mutual information, thereby improving the accuracy of video positioning. The present invention realizes fine-grained semantic interaction by fusing text and video features from coarse to fine through hierarchical graph interaction, which is used for temporal positioning score learning. It can establish multi-granular intrinsic connections between the two modalities, avoid the low generalization problem caused by content selectivity bias, and enrich temporal clues at different levels to promote accurate video positioning. The present invention can integrate the advantages of proposal-based and proposal-free methods, improve the performance of video text positioning tasks, enhance the generalization ability of the model, and provide a new perspective for video understanding in practical applications.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the technical field of video text positioning, and in particular relates to a fine-grained video text positioning method and system based on a cross-modal semantic hierarchy graph. Background Art

[0002] The task of video text spotting involves identifying segments that semantically match a specific natural language query in a large number of unedited videos. This technology has a wide range of uses in real-world applications, such as helping users quickly find content of interest in a large number of videos. This task requires in-depth multimodal semantic analysis between queries and videos, and accurate extraction of contextual information to match complex semantic expressions. Therefore, it is a crucial and challenging task in the field of video understanding.

[0003] Existing video text localization techniques are mainly divided into two methods: proposal-based methods and proposal-free methods. Proposal-based methods maximize the mutual information between queries and video clips through a proposal ranking framework to learn feature representations to improve localization performance. This method usually relies on global information for feature extraction. Proposal-free methods directly predict the start and end timestamps of the video through the segment-word level semantic interaction between the two modalities, achieving semantic alignment without proposals.

[0004] In the field of video text localization, proposal-based methods and proposal-free methods have each demonstrated their unique advantages and made progress. However, both methods still face a series of challenges and limitations in practical applications. Proposal-based methods usually rely on pre-generated video segment proposals, and then locate relevant content in the video by comparing the similarity between these proposals and query statements. A major advantage of this method is that it can use global information to guide feature extraction, thereby improving the accuracy of localization to a certain extent. However, this method also has obvious disadvantages, such as the tendency to capture selective bias in content, which may cause the model to overemphasize typical actions or nouns in the video and ignore other key visual and language information, such as time and direction. This selective bias may limit the model's understanding of fine-grained visual and language concepts, thereby affecting the model's generalization ability and the effect of practical applications. Proposal-free methods try to get rid of the reliance on pre-generated proposals by directly locating the content referred to by the query statement in the video. This method usually involves segment-word-level semantic interaction between video and text to achieve more refined semantic matching. The advantage of this method is that it can handle the diversity of video content more flexibly, while also helping to reduce the consumption of computing resources. However, proposal-free methods also face their own challenges, such as difficulty in handling complex topological hierarchies in videos and texts. Due to the lack of multi-granularity modeling, such methods may have difficulty in capturing semantic details and understanding the complex relationship between global and local content. Summary of the invention

[0005] The purpose of the present invention is to provide a fine-grained video text localization method and system based on a cross-modal semantic hierarchy graph to solve the above-mentioned problems existing in the prior art.

[0006] In order to achieve the above object, the present invention adopts the following technical solutions:

[0007] First, a fine-grained video text localization method based on a cross-modal semantic hierarchy graph is provided, including:

[0008] Get the input query text and target video;

[0009] Performing video encoding on the target video to obtain the video features of the target video, segmenting the target video into a number of candidate segments, extracting the video features of each candidate segment, and performing text encoding on the query text to obtain the query features of the query text;

[0010] Perform hierarchical parsing on the query features to obtain a hierarchical query embedding representation of the query text, and perform hierarchical encoding on the video features of each candidate segment to obtain a hierarchical video embedding representation of each candidate segment;

[0011] Performing hierarchical semantic alignment analysis on the query hierarchical embedding representation and the video hierarchical embedding representation of the corresponding candidate segment, determining the hierarchical similarity between the query hierarchical embedding representation and the corresponding video hierarchical embedding representation, and associating the hierarchical similarity with the corresponding candidate segment;

[0012] The video features of the target video are hierarchically encoded to obtain a hierarchical video embedding representation of the target video, and cross-modal temporal prediction is performed based on the query hierarchical embedding representation and the hierarchical video embedding representation of the target video to obtain a complete positioning score;

[0013] Determine the segmented positioning score of each candidate segment according to the complete positioning score, the number of candidate segments, and the start time and end time of each candidate segment;

[0014] Obtaining a priori parameters determined by positioning regression analysis, wherein the priori parameters include an IoU value;

[0015] Multiply the hierarchical similarity, segment positioning score and IoU value to obtain the final positioning score of the corresponding candidate segment;

[0016] The final positioning score of all candidate segments is the highest as the target positioning segment of the query text, and the target positioning segment is output.

[0017] In one possible design, the video feature of the target video is expressed as ,in, The video feature component representing the target video, T represents the number of candidate segments of the video, t represents the candidate segment number, R represents the real number set, and D represents the feature dimension; the video feature of the candidate segment Expressed as ,in, The video feature components that characterize the candidate clips, Characterize the number of video feature components, Characterizes the video feature component number; the query feature of the query text is expressed as ,in, The query feature component that characterizes the query text, Characterizes the number of query feature components, Characterizes the query feature component number.

[0018] In a possible design, the hierarchical parsing of the query features to obtain a hierarchical query embedding representation of the query text includes:

[0019] Based on the BERT model, the query feature Q is hierarchically parsed to obtain the global node representation = , the action node represents , and object node representation ,in, is the number of action nodes, is the number of object nodes;

[0020] The preset query embedding transformation is used to transform each hierarchical node representation to obtain the query hierarchical embedding representation of the query text. , the query embedding transformation is ,in, is the set graph node transformation matrix, , To query the global embedding representation, Embedding representation for query action, Embedding representation for query objects, = .

[0021] In a possible design, the hierarchical encoding of the video features of each candidate segment to obtain a hierarchical video embedding representation of each candidate segment includes:

[0022] The video features of the candidate segments are hierarchically encoded using the set segment embedding transformation formula to obtain a video hierarchical embedding representation of the candidate segments, wherein the video hierarchical embedding representation includes a video global embedding representation. , video action embedding representation , and video object embedding representation , the fragment embedding transformation formula is

[0023]

[0024]

[0025] Among them, the video global embedding representation , video action embedding representation = = , video object embedding representation = = , To set the first mapping matrix, To set the second mapping matrix, To set the third mapping matrix, To set the fourth mapping matrix.

[0026] In a possible design, performing hierarchical semantic alignment analysis on the query hierarchical embedding representation and the video hierarchical embedding representation of the corresponding candidate segment to determine the hierarchical similarity between the query hierarchical embedding representation and the corresponding video hierarchical embedding representation includes:

[0027] Compute query global embedding representation Represented with global video embedding The cosine similarity of , ,in, Characterize the cosine similarity operation;

[0028] Compute query action embedding representation and video action embedding representation The cosine similarity of ,

[0029]

[0030]

[0031]

[0032]

[0033] in, Characterization Function operation, is the setting coefficient, , max represents the maximum value operator;

[0034] Compute query object embedding representation Embedded representation with video objects The cosine similarity of ,

[0035]

[0036]

[0037]

[0038] ;

[0039] Calculate the hierarchical similarity between the query hierarchical embedding representation and the corresponding video hierarchical embedding representation , .

[0040] In a possible design, the cross-modal temporal prediction based on the query hierarchical embedding representation and the video hierarchical embedding representation of the target video is performed to obtain a complete positioning score, including:

[0041] A preset cross-modal graph convolutional network is used to hierarchically aggregate the query hierarchical embedding representation and the target video hierarchical embedding representation to obtain a comprehensive feature F.

[0042] Based on the comprehensive feature F, the set cross-modal positioning score formula is used to perform positioning prediction to obtain the complete positioning score , the cross-modal positioning score formula is ,in, Represents a multilayer perceptron.

[0043] In a possible design, determining the segmented positioning score of each candidate segment according to the complete positioning score, the number of candidate segments, and the start time and end time of each candidate segment includes:

[0044] According to the complete positioning score, the number of candidate segments, and the start and end time of each candidate segment, the segment positioning score of each candidate segment is calculated using the set segment positioning score formula. , the fragment positioning score formula is ,in, , T represents the number of candidate fragments, and Respectively represent the start time and end time of the candidate segment;

[0045] The segment location score for each candidate segment Perform normalization to obtain the segmentation positioning score of each candidate segment .

[0046] In a possible design, the method is implemented by a preset semantic hierarchical positioning model.

[0047] Secondly, a fine-grained video text localization system based on a cross-modal semantic hierarchy graph is provided, including:

[0048] A memory for storing instructions;

[0049] A processor is used to read the instructions stored in the memory and execute any one of the methods described in the first aspect according to the instructions.

[0050] In a third aspect, a fine-grained video text localization system based on a cross-modal semantic hierarchy graph is provided, wherein the system is installed with a computer program product, and when the computer program product runs on the system, the system executes any one of the methods described in the first aspect.

[0051] Beneficial effects: The present invention can achieve fine-grained semantic alignment and interaction based on a cross-modal semantic hierarchy graph. Compared with traditional proposal-based and proposal-free methods, the present invention achieves a more refined query-video clip alignment by constructing a cross-modal semantic hierarchy graph from global to local. This hierarchical semantic alignment can maximize mutual information, thereby improving the accuracy of video positioning. The present invention achieves fine-grained semantic interaction by fusing text and video features from coarse to fine through hierarchical graph interaction for temporal positioning score learning. It can establish multi-granular intrinsic connections between the two modalities, avoid the low generalization problem caused by content selectivity bias, and enrich temporal clues at different levels to promote accurate video positioning. The present invention can integrate the advantages of proposal-based and proposal-free methods, improve the performance of video text positioning tasks, enhance the generalization ability of the model, and provide a new perspective for video understanding in practical applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0053] Figure 1 This is a schematic diagram of the method steps of Example 1;

[0054] Figure 2 This is a schematic diagram of the method flow of Example 1;

[0055] Figure 3 This is a schematic diagram of the cross-modal graph convolutional network structure in Example 1;

[0056] Figure 4 This is a chart showing the first experimental results in Example 1;

[0057] Figure 5 This is a chart showing the results of the second experiment in Example 1;

[0058] Figure 6 This is a schematic diagram of the system configuration of Example 2. DETAILED DESCRIPTION

[0059] It should be noted that the description of these embodiments is used to help understand the present invention, but does not constitute a limitation of the present invention. The specific structures and functional details disclosed herein are only used to describe the exemplary embodiments of the present invention. However, the present invention can be embodied in many alternative forms, and it should not be understood that the present invention is limited to the embodiments set forth herein.

[0060] It should be understood that, unless otherwise clearly specified and limited, the term "connection" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the embodiments can be understood according to specific circumstances.

[0061] In the following description, certain details are provided to facilitate a complete understanding of the example embodiments. However, it will be appreciated by those of ordinary skill in the art that the example embodiments may be implemented without these certain details. For example, the system may be shown in a block diagram to avoid obscuring the example with unnecessary details. In other embodiments, well-known processes, structures, and techniques may not be shown in unnecessary detail to avoid obscuring the embodiments.

[0062] Embodiment 1:

[0063] This embodiment provides a fine-grained video text localization method based on a cross-modal semantic hierarchy graph, which can be applied to a semantic hierarchy localization model. The semantic hierarchy localization model can use a cross-modal semantic hierarchy graph to achieve fine-grained semantic alignment and interaction. Compared with traditional proposal-based and proposal-free methods, the semantic hierarchy localization model can achieve more refined query-video clip alignment by constructing a cross-modal semantic hierarchy graph from global to local. Specifically, the design of the semantic hierarchy localization model allows it to capture multi-level semantic content in the video query text, such as global background, specific actions, and objects involved. This hierarchical graph structure helps the semantic hierarchy localization model to fully understand the fine-grained semantic content.

[0064] Specifically, Figure 1 to Figure 2 As shown, the method for implementing fine-grained video text positioning using the semantic hierarchical positioning model includes the following steps:

[0065] S1. Get the input query text and target video.

[0066] S2. Perform video encoding on the target video to obtain the video features of the target video, and divide the target video into several candidate segments, extract the video features of each candidate segment, perform text encoding on the query text, and obtain the query features of the query text.

[0067] In specific implementation, the target video can be encoded based on a three-dimensional convolutional neural network, such as a C3D convolutional neural network, to extract the video features of the target video. At the same time, the position encoding is incorporated into it, and the self-attention mechanism is used to extract the context. The obtained video features of the target video are expressed as ,in, The video feature component that represents the target video, T represents the number of candidate segments of the video, t represents the candidate segment number, R represents the real number set, and D represents the feature dimension. Through sparse sampling, several candidate segments of the target video can be extracted, and the video features of the candidate segments Expressed as ,in, The video feature components that characterize the candidate clips, Characterize the number of video feature components, Represents the video feature component number. For the query text, a tokenizer can be used to generate word tokens (representations), and these word tokens are input into a pre-trained BERT model for text encoding to obtain the query feature representation as ,in, The query feature component that characterizes the query text, Characterizes the number of query feature components, Characterizes the query feature component number.

[0068] S3. Perform hierarchical analysis on the query features to obtain a hierarchical query embedding representation of the query text, and perform hierarchical encoding on the video features of each candidate segment to obtain a hierarchical video embedding representation of each candidate segment.

[0069] In specific implementation, in order to obtain the global graph structure, actions and objects of corresponding actions, semantic role parsing tools can be used to obtain various semantic relationships between query nodes. Specifically, role parsing will identify verbs, noun phrases, etc. in the query and their corresponding semantic relationships (i.e., roles in actions). The entire sentence query is taken as a global node, and the verb is regarded as an action node, which is connected to the global node through direct edges to implicitly learn the order of actions in subsequent reasoning. Other parts such as noun phrases are connected through corresponding action nodes to form object nodes, and the type of edge connecting it to the action is determined by its semantic role in the action, such as subject, object, direction, etc.

[0070] In order to generate node representation, the query features can be hierarchically parsed, and the [CLS] tag output by the BERT model is used as the global node representation of the entire query. = , directly use the features of the position marked as a verb in the query representation as the action node representation , for object nodes, extract features of object phrases in the query representation and perform average pooling to obtain object node representation ,in, is the number of action nodes, is the number of object nodes.

[0071] The graph structure enhances the understanding of complex query semantics. It not only decomposes the query from global to local, but also describes the semantics more accurately. For example, without additional information, the action "puts" is ambiguous in the graph and can be interpreted as different actions. However, "he" and "another tire" provide context to make this action clearer. Therefore, graph reasoning is needed to further mine the semantic information in the hierarchical text graph. A dynamic graph attention network is used for graph reasoning, and all graph node embeddings use a common graph node transformation matrix Perform query embedding transformation. The query embedding transformation formula is ,in, is the set graph node transformation matrix, , To query the global embedding representation, Embedding representation for query action, Embedding representation for query objects, = .

[0072] Using graph node transformation matrices Can be initialized to make Then the node Its neighbor nodes Connect, select relevant context from neighbor nodes, and embed them through the relational matrix Derive the corresponding attention coefficients for various edges ,

[0073]

[0074]

[0075] in, is a one-hot vector representing the node and neighbor nodes The edge type between them comes from the semantic hierarchical analysis results. is the number of edge types, Represents the intermediate parameters, LeakyReLU represents the LeakyReLU activation function, represents the feature connection operation, Representation Node The set of neighbor nodes ( ). Then the contextual information of participating neighbor nodes is propagated to the node through residual connections. , so the output embedding is expressed as:

[0076]

[0077] in, represents the ELU activation function. Reassign them to the final graph nodes according to their positions.

[0078] The video contains time segments corresponding to the global, action, and object-level semantics in the query. The time segments in the video can be independently constructed into three levels of embedding. First, after obtaining the entire video feature V through the video encoder, the feature extraction is performed for each candidate segment in the video. Then, three randomly initialized learnable mapping matrices ( ), is the first mapping matrix, is the second mapping matrix, For the third mapping matrix, to obtain the fragment embedding of the three spaces:

[0079]

[0080] Clip-level features can be directly used for action and object embedding, that is, video action embedding is represented as , and the video object embedding is represented as , for the global embedding, attention-aware aggregation can be used to obtain the global embedding:

[0081]

[0082] in, is a randomly initialized learnable mapping matrix, namely the fourth mapping matrix.

[0083] S4. Perform hierarchical semantic alignment analysis on the query hierarchical embedding representation and the video hierarchical embedding representation of the corresponding candidate segment, determine the hierarchical similarity between the query hierarchical embedding representation and the corresponding video hierarchical embedding representation, and associate the hierarchical similarity with the corresponding candidate segment.

[0084] In the specific implementation, in order to extract the correlation between the query text and the video period at multiple levels, multi-granularity similarity can be calculated from three levels. At the global level, cosine similarity is used to measure the similarity score between the query and the period, that is, the query global embedding representation is calculated. With the global video embedding representation Cosine similarity of , ,in, Represents the cosine similarity operation. At the action level, the query action embedding representation can be calculated and video action embedding representation Cosine similarity of ,

[0085]

[0086]

[0087]

[0088]

[0089] in, Characterization Function operation, is the setting coefficient, , max represents the maximum value operator. At the object level, the query object embedding representation can be calculated Embedded representation with video objects Cosine similarity of ,

[0090]

[0091]

[0092]

[0093] ;

[0094] Finally, the average similarity of the three levels is taken as the final query-segment similarity , .

[0095] S5. Hierarchically encode the video features of the target video to obtain a hierarchical video embedding representation of the target video, and perform cross-modal temporal prediction based on the query hierarchical embedding representation and the hierarchical video embedding representation of the target video to obtain a complete positioning score.

[0096] In practice, in order to fully explore the interactive information of the cross-modal semantic hierarchy and achieve enhanced flexibility in localization, the query hierarchical embedding representation can be integrated with the hierarchical representation of the entire video to make temporal prediction possible. Figure 3 The cross-modal graph convolutional network shown in Figure 1 performs hierarchical aggregation to obtain a comprehensive feature F. The hierarchical representation of the entire video comes from inputting the entire video into the video hierarchical feature generation network to obtain the hierarchical features of the entire video, including the global hierarchical , action level , and the object hierarchy At the global level, the cross-modal graph convolutional network concatenates the global embeddings of the two modalities and then obtains the interactive features through mapping:

[0097]

[0098] in, is the learnable mapping matrix, The learnable bias vector, Represents the splicing operation. Then, it is distributed to the respective action levels, through splicing and mapping and Get the action space of two modes and .

[0099] We use a cross-modal graph convolutional network to interact between the video and query action levels. We first measure the cross-modal similarity between the query action and the video action and construct an adjacency matrix:

[0100]

[0101] Among them, the value , and It is a linear mapping of two specific modalities, projecting the features of one modality into the same latent space as the features of the other modality. and is the normalized adjacency matrix. Therefore, the aligned video representation and query representation can be obtained as follows:

[0102]

[0103] in, and is a learnable weight matrix. Then use the obtained We further interact with object-level embeddings by firstly adding and Fusion to obtain , and embed and Similarly, a cross-modal graph convolutional network can be used to obtain the final And use it as a fusion feature Output.

[0104] Based on the comprehensive feature F, the cross-modal positioning score formula can be used Perform positioning prediction and obtain a complete positioning score ,in, Represents a multilayer perceptron.

[0105] S6. Determine the segmented positioning score of each candidate segment according to the complete positioning score, the number of candidate segments, and the start time and end time of each candidate segment.

[0106] In specific implementation, the corresponding fragment positioning score formula can be used to calculate the fragment positioning score of each candidate fragment. , the fragment positioning score formula is ,in, , T represents the number of candidate fragments, and Respectively represent the start time and end time of the candidate segment. Then the segment positioning score of each candidate segment Perform normalization to obtain the segmentation positioning score of each candidate segment .

[0107] S7. Obtain a priori parameters determined by positioning regression analysis, where the priori parameters include an IoU value.

[0108] In specific implementation, the IoU value between the model's predicted segment and the target segment can be determined through positioning regression analysis during the training and positioning prediction and reasoning stages of the semantic level positioning model. , that is, the intersection-and-union ratio, the IoU value as the prior parameters of the model.

[0109] S8. Multiply the hierarchical similarity, the segment positioning score and the IoU value to obtain the final positioning score of the corresponding candidate segment.

[0110] In the specific implementation, the hierarchical similarity of the corresponding candidate fragments is and segment positioning score And IoU value Multiply them together to get the final positioning score of the corresponding candidate segment, that is, the final positioning score of the corresponding candidate segment = .

[0111] S9. The final positioning score of all candidate segments is the highest as the target positioning segment of the query text, and the target positioning segment is output.

[0112] In the specific implementation, after obtaining the final positioning score of each candidate segment, the model selects the segment with the highest final positioning score from all candidate segments as the target positioning segment of the query text, and outputs the target positioning segment to achieve fine-grained video text positioning based on the cross-modal semantic hierarchy graph.

[0113] For the semantic level positioning model, the training process can use the total loss function , ,in:

[0114]

[0115]

[0116]

[0117] in, and is a batch of training queries and training video clips, and Respectively represent and and Matching video clips and queries, , , is a hyperparameter that controls the smoothness of the distribution.

[0118]

[0119] in, If it is within the target time interval, it is 1, otherwise it is 0. If it is within the target interval, it indicates whether a frame in the video belongs to the segment to be predicted, that is, within the start and end time of the predicted segment;

[0120]

[0121] in, Indicates the IoU value of the corresponding candidate segment to the target segment.

[0122] To illustrate the effect of the method of this embodiment, the "R@n, IoU=m" indicator is used to measure the performance of this method. This indicator measures the proportion of the first n retrieved video clips whose intersection over union (IoU) value with the target clip exceeds the threshold "m", which is a widely used evaluation criterion. ActivityNet-Captions and Charades-STA datasets are also used to evaluate this method. ActivityNet-Captions contains 20,000 unedited videos and 100,000 related descriptions, covering various complex human activity scenes. On this dataset, according to the official segmentation, 37,417, 17,505, and 17,031 sentence-video pairs are used as training, validation, and test sets, respectively. Charades-STA is built on the Charades dataset and contains indoor activity videos with text queries. Its training set and test set contain 12,408 and 3,720 video query pairs, respectively. To ensure a fair comparison, pre-trained CNN networks were used on both datasets to extract video embedding features (ActivityNet-Captions uses C3D networks, and Charades-STA uses VGG networks). In the comparison of ActivityNet-Captions dataset using C3D features, our method outperformed other competing algorithms in all evaluation indicators, such as Figure 4The first experimental result chart shown summarizes the corresponding results. Experiments show that the semantic hierarchical localization model (i.e., SHG) adopted by this method outperforms all baselines in all indicators. For example, compared with the previous state-of-the-art method MGSL, SHG achieved an absolute improvement of 1.17% and 1.07% on "R@1 IoU=0.5" and "R@1 IoU=0.7", respectively, proving the superiority of the semantic hierarchical localization model. This is mainly due to encouraging the model to achieve hierarchical alignment and interaction, and to establish multi-granular intrinsic connections, thereby alleviating selective bias in the content. In the comparison of positioning performance using VGG features on the Charades-STA dataset, this method also outperforms other competing algorithms in all evaluation indicators, such as Figure 5 The second experimental result chart shown summarizes the corresponding results. The experiments show that the semantic hierarchical localization model (i.e., SHG) adopted by this method still achieves state-of-the-art performance on all indicators. Although MMN, which combines metric learning with basic regression, has achieved advanced results, it fails to consider fine-grained semantic relationships. Compared with MMN, this method improves the "R@5 IoU=0.5" and "R@5 IoU=0.7" indicators by 1.09% and 1.02%, respectively. This further verifies the effectiveness of this method.

[0123] Embodiment 2:

[0124] This embodiment provides a fine-grained video text localization system based on a cross-modal semantic hierarchy graph, such as Figure 6 As shown, at the hardware level, it includes:

[0125] Data interface, used to establish data connection between the processor and the external data terminal;

[0126] A memory for storing instructions;

[0127] The processor is used to read the instructions stored in the memory and execute the fine-grained video text localization method based on the cross-modal semantic hierarchy graph in Example 1 according to the instructions.

[0128] Optionally, the device also includes an internal bus. The processor, the memory and the data interface can be interconnected through the internal bus, and the internal bus can be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc.

[0129] The memory may include, but is not limited to, random access memory (RAM), read only memory (ROM), flash memory, first input first output (FIFO) and / or first in last out (FILO), etc. The processor may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it may also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components.

[0130] Embodiment 3:

[0131] This embodiment provides a computer-readable storage medium, on which instructions are stored. When the instructions are executed on a computer, the computer executes the fine-grained video text localization method based on a cross-modal semantic hierarchy graph in Embodiment 1. The computer-readable storage medium refers to a carrier for storing data, which may include but is not limited to a floppy disk, an optical disk, a hard disk, a flash memory, a USB flash drive, and / or a memory stick, etc. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable systems.

[0132] This embodiment also provides a computer program product including instructions, which, when executed on a computer, enables the computer to execute the fine-grained video text localization method based on a cross-modal semantic hierarchy graph in Embodiment 1. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable systems.

[0133] Finally, it should be noted that the above description is only a preferred embodiment of the present invention and is not intended to limit the protection scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A fine-grained video text localization method based on a cross-modal semantic hierarchy graph, characterized in that: include: Get the input query text and target video; Performing video encoding on the target video to obtain the video features of the target video, segmenting the target video into a number of candidate segments, extracting the video features of each candidate segment, and performing text encoding on the query text to obtain the query features of the query text; Perform hierarchical parsing on the query features to obtain a hierarchical query embedding representation of the query text, and perform hierarchical encoding on the video features of each candidate segment to obtain a hierarchical video embedding representation of each candidate segment; Performing hierarchical semantic alignment analysis on the query hierarchical embedding representation and the video hierarchical embedding representation of the corresponding candidate segment, determining the hierarchical similarity between the query hierarchical embedding representation and the corresponding video hierarchical embedding representation, and associating the hierarchical similarity with the corresponding candidate segment; The video features of the target video are hierarchically encoded to obtain a hierarchical video embedding representation of the target video, and cross-modal temporal prediction is performed based on the query hierarchical embedding representation and the hierarchical video embedding representation of the target video to obtain a complete positioning score; Determine the segmented positioning score of each candidate segment according to the complete positioning score, the number of candidate segments, and the start time and end time of each candidate segment; Obtaining a priori parameters determined by positioning regression analysis, wherein the priori parameters include an IoU value; Multiply the hierarchical similarity, segment positioning score and IoU value to obtain the final positioning score of the corresponding candidate segment; The final positioning score of all candidate segments is the highest as the target positioning segment of the query text, and the target positioning segment is output; The video feature of the target video is expressed as ,in, The video feature component representing the target video, T represents the number of candidate segments of the video, t represents the candidate segment number, R represents the real number set, and D represents the feature dimension; the video feature of the candidate segment Expressed as ,in, The video feature components that characterize the candidate clips, Characterize the number of video feature components, Characterizes the video feature component number; the query feature of the query text is expressed as ,in, The query feature component that characterizes the query text, Characterizes the number of query feature components, Characterizes the query feature component number; The step of performing hierarchical parsing on the query features to obtain a hierarchical query embedding representation of the query text includes: Based on the BERT model, the query feature Q is hierarchically parsed to obtain the global node representation = , the action node represents , and object node representation ,in, is the number of action nodes, is the number of object nodes; The preset query embedding transformation is used to transform each hierarchical node representation to obtain the query hierarchical embedding representation of the query text. , the query embedding transformation is ,in, is the set graph node transformation matrix, , To query the global embedding representation, Embedding representation for query action, Embedding representation for query objects, = ; The step of hierarchically encoding the video features of each candidate segment to obtain a hierarchical video embedding representation of each candidate segment includes: The video features of the candidate segments are hierarchically encoded using the set segment embedding transformation formula to obtain a video hierarchical embedding representation of the candidate segments, wherein the video hierarchical embedding representation includes a video global embedding representation. , video action embedding representation , and video object embedding representation , the fragment embedding transformation formula is Among them, the video global embedding representation , video action embedding representation = = , video object embedding representation = = , To set the first mapping matrix, To set the second mapping matrix, To set the third mapping matrix, To set the fourth mapping matrix; The step of performing hierarchical semantic alignment analysis on the query hierarchical embedding representation and the video hierarchical embedding representation of the corresponding candidate segment to determine the hierarchical similarity between the query hierarchical embedding representation and the corresponding video hierarchical embedding representation includes: Compute query global embedding representation With the global video embedding representation Cosine similarity of , ,in, Characterize the cosine similarity operation; Compute query action embedding representation and video action embedding representation Cosine similarity of , in, Characterization Function operation, is the setting coefficient, , max represents the maximum value operator; Compute query object embedding representation Embedded representation with video objects Cosine similarity of , ; Calculate the hierarchical similarity between the query hierarchical embedding representation and the corresponding video hierarchical embedding representation , ; The cross-modal time prediction is performed based on the query hierarchical embedding representation and the video hierarchical embedding representation of the target video to obtain a complete positioning score, including: A preset cross-modal graph convolutional network is used to hierarchically aggregate the query hierarchical embedding representation and the target video hierarchical embedding representation to obtain a comprehensive feature F. Based on the comprehensive feature F, the set cross-modal positioning score formula is used to perform positioning prediction to obtain the complete positioning score , the cross-modal positioning score formula is ,in, represents a multi-layer perceptron; The step of determining the segmented positioning score of each candidate segment according to the complete positioning score, the number of candidate segments, and the start time and end time of each candidate segment includes: According to the complete positioning score, the number of candidate segments, and the start and end time of each candidate segment, the segment positioning score of each candidate segment is calculated using the set segment positioning score formula. , the fragment positioning score formula is ,in, , T represents the number of candidate fragments, and Respectively represent the start time and end time of the candidate segment; The segment location score for each candidate segment Perform normalization to obtain the segmentation positioning score of each candidate segment .

2. According to claim 1, the fine-grained video text localization method based on cross-modal semantic hierarchy graph is characterized in that: The method is implemented by a preset semantic hierarchical positioning model.

3. A fine-grained video text localization system based on a cross-modal semantic hierarchy graph, characterized by: include: A memory for storing instructions; A processor is used to read the instructions stored in the memory and execute the fine-grained video text localization method based on a cross-modal semantic hierarchy graph according to claim 1 or 2 according to the instructions.

4. A fine-grained video text localization system based on a cross-modal semantic hierarchy graph, characterized by: The system is installed with a computer program product, and when the computer program product runs on the system, the system executes the fine-grained video text localization method based on a cross-modal semantic hierarchy graph as described in claim 1 or 2.