Video localization model training method
By extracting and fusing video and text features, combining cross entropy loss and semantic perceptual learning, the problem that video moment positioning methods in the prior art cannot pay attention to key information and time information, significantly improving positioning accuracy and generalization capabilities.
Patent Information
- Application Number
- CN202411054298.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-02
- Publication Date
- 2024-11-19
- Estimated Expiration
- 2044-08-02
AI Technical Summary
Existing video moment positioning methods cannot effectively pay attention to key information and time information in the video, and it is difficult to understand the fine-grained semantics in the video and text, resulting in limited positioning accuracy and generalization capabilities.
By extracting word-level text features and frame-level video features in the sample data, the sentence-level text features and visual features are fused to determine the stage-specific representation of the video segment, and through training of candidate video segments and real video segments, weighted binary is adopted Cross-entropy loss and L1 loss are trained in model, combining regional semantic perception and main semantic perception learning to improve the positioning accuracy and generalization ability of the model.
The accuracy of video moment positioning and combined generalization capabilities of models are significantly improved, especially when processing query texts containing novel components.
Smart Images

Figure CN118587409B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of computer vision technology, and in particular to a video positioning model training method. The present application also relates to a video positioning model training device, a computing device, and a computer-readable storage medium. Background Art
[0002] Video is one of the media that contains a large amount of information. With the rapid development of the Internet and social mobile networks, the number of videos on the Internet has exploded. How to quickly and accurately find the video content that users need in a large number of Internet videos is a problem with practical application value that is related to improving user experience and improving the production efficiency of video creators. It is also one of the key problems that traditional video retrieval needs to solve. Text-based video moment positioning is an emerging retrieval method. In this way, users can locate the required segment in a long video by entering text content.
[0003] Most existing moment localization methods can be divided into two categories: candidate-based methods and non-candidate-based methods. Candidate-based methods usually pre-generate video clips that may match the query sentence. Then, they calculate the matching degree between all candidate clips and the query through a cross-modal fusion module and select the candidate clip with the highest matching degree as the output; non-candidate-based methods use video frames as the basic unit of the video and directly predict the start and end boundaries of the target clip on the video sequence. Most of these methods are designed with various specific cross-modal fusion modules, or use richer supervisory signals to achieve accurate moment localization by aligning videos and natural language sentences.
[0004] Although the above methods have achieved many promising performances, there are still two key unresolved issues: (1) Most candidate-based methods adopt temporal convolution design, which constructs video representation by performing convolution or pooling operations on frame-level video features. However, the pooling operation treats each element in the video equally and fails to focus on key information, and cannot model the rich temporal information in the video. The convolution operation is limited by the fixed structure of the convolution kernel and cannot represent different time stages in the video. (2) Existing methods mainly align unstructured global representations in videos and texts, and pay less attention to common fine-grained representations in vision or text as semantic backbones. This coarse-grained alignment operation between unstructured global representations makes it difficult for the model to fully understand the semantics contained in the video and text, but can only understand specific query patterns. At the same time, this unstructured global representation is also difficult to represent the fine-grained correspondence between video and text, so it cannot understand the novel components appearing in the query text, and the combination generalization ability is limited. Summary of the invention
[0005] In view of this, the embodiment of the present application provides a video positioning model training method to solve the technical defects existing in the prior art. The embodiment of the present application also provides a video positioning model training device, a computing device, and a computer-readable storage medium.
[0006] According to a first aspect of an embodiment of the present application, a video positioning model training method is provided, comprising:
[0007] Extracting word-level text features of sample texts in the sample data set and frame-level video features of sample videos in the sample data set through an initial video localization model;
[0008] Fusion of the word-level text features and the frame-level video features to obtain sentence-level text features and visual features containing contextual semantic information;
[0009] determining a stage-specific representation of a clipped video segment corresponding to the visual feature, and selecting a candidate video segment from the clipped video segment according to the stage-specific representation;
[0010] The initial video positioning model is trained by using the candidate video segments and the real video segments in the sample data set until a training stop condition is met, thereby obtaining a target video positioning model.
[0011] Optionally, the fusing of the word-level text features with the frame-level video features to obtain sentence-level text features and visual features containing contextual semantic information includes:
[0012] Dividing the video frames in the sample video to obtain a plurality of video segments of equal length;
[0013] Perform mean pooling on the frame-level video features corresponding to each video segment to obtain the clip-level video features;
[0014] The clip-level video features and the word-level text features are connected in series, and global semantic modeling is performed in conjunction with position encoding to obtain the sentence-level text features and the visual features containing contextual semantic information.
[0015] Optionally, determining the stage-specific representation of the clipped video segment corresponding to the visual feature comprises:
[0016] Converting the visual features into a start representation, an intermediate representation, and an end representation of a corresponding video end;
[0017] Based on an adjustable weight matrix, the start representation, the intermediate representation and the end representation are nonlinearly combined to obtain the stage-specific representation.
[0018] Optionally, the training of the initial video positioning model by using the candidate video segments and real video segments in the sample data set includes:
[0019] Based on the candidate video segment and the real video segment, calculating a matching score and an offset distance of the candidate video segment;
[0020] Calculating a weighted binary cross entropy loss and an L1 loss according to the matching score and the offset distance;
[0021] The initial video localization model is trained based on the weighted binary cross entropy loss and the L1 loss.
[0022] Optionally, the training the initial video positioning model based on the weighted binary cross entropy loss and the L1 loss includes:
[0023] Calculate the sum of the weighted binary cross entropy loss and the L1 loss to obtain the moment positioning loss;
[0024] Determine a video segment corresponding to the real video segment as a positive sample, and select negative samples from other video segments and other videos outside the positive sample;
[0025] Calculating contrastive learning loss based on the positive sample and the negative sample;
[0026] The initial video positioning model is trained based on the moment positioning loss and the contrastive learning loss.
[0027] Optionally, the training of the initial video positioning model based on the moment positioning loss and the contrastive learning loss includes:
[0028] By using a preset masking strategy, the basic components of the sample text and the sample video are masked to obtain positive and negative video sentence sample pairs, wherein the basic components of the sample text are words, and the basic components of the sample video are video frames;
[0029] Processing the positive and negative video sentence sample pairs through the initial video localization model, and calculating the semantic contrastive learning loss based on the processing results;
[0030] The initial video positioning model is trained based on the moment positioning loss, the contrastive learning loss and the semantic contrastive learning loss.
[0031] Optionally, the method of masking the basic components of the sample text and the sample video by a preset masking strategy to obtain positive and negative video sentence sample pairs includes:
[0032] Assigning different weights to the video frames in the sample video corresponding to the real video segment and the video frames not corresponding to the real video segment, respectively, and masking the assignment results based on a preset masking strategy to obtain positive video samples and negative video samples;
[0033] According to the word classes of the words in the sample text, different weights are assigned respectively, and based on a preset shielding strategy, the assignment results are shielded to obtain positive sentence samples and negative sentence samples;
[0034] A positive video sentence sample pair is constructed based on the positive video sample and the positive sentence sample, and a negative video sentence sample pair is constructed based on the negative video sample and the negative sentence sample, wherein the positive and negative video sentence sample pairs include the positive video sentence sample pair and the negative video sentence sample pair.
[0035] Optionally, the processing the positive and negative video sentence sample pairs by the initial video localization model and calculating the semantic contrastive learning loss based on the processing results includes:
[0036] Processing the positive video sentence sample pair through the initial video positioning model to obtain a first predicted matching score;
[0037] Processing the negative video sentence sample pair through the initial video localization model to obtain a second predicted matching score;
[0038] The semantic contrastive learning loss is calculated according to the first predicted matching score, the second predicted matching score, and the matching score.
[0039] Optionally, the training of the initial video positioning model based on the moment positioning loss, the contrastive learning loss and the semantic contrastive learning loss includes:
[0040] Calculating a first binary cross entropy loss based on the first predicted matching score and the matching score;
[0041] Summing the first binary cross entropy loss and the semantic contrastive learning loss to obtain a main semantic perception loss;
[0042] The initial video localization model is trained based on the moment localization loss, the contrastive learning loss and the main semantic perception loss.
[0043] Optionally, the training of the initial video localization model based on the moment localization loss, the contrastive learning loss and the main semantic perception loss includes:
[0044] Randomly masking words in the sample text, and reconstructing the masked sample text using the initial video positioning model;
[0045] Calculate the cross entropy loss based on the reconstruction result and the sample text;
[0046] The moment positioning loss, the contrastive learning loss, the main semantic perception loss and the cross entropy loss are weighted and summed based on a preset weighting coefficient to obtain a target loss function;
[0047] The initial video positioning model is trained using the target loss function.
[0048] According to a second aspect of an embodiment of the present application, a video positioning model training device is provided, comprising:
[0049] An extraction module is configured to extract word-level text features of sample texts in the sample data set and frame-level video features of sample videos in the sample data set through an initial video localization model;
[0050] A fusion module is configured to fuse the word-level text features with the frame-level video features to obtain sentence-level text features and visual features containing contextual semantic information;
[0051] a selection module configured to determine a stage-specific representation of a clipped video segment corresponding to the visual feature, and select a candidate video segment from the clipped video segment according to the stage-specific representation;
[0052] The training module is configured to train the initial video positioning model through the candidate video segments and the real video segments in the sample data set until a training stop condition is met to obtain a target video positioning model.
[0053] According to a third aspect of an embodiment of the present application, a computing device is provided, including:
[0054] Memory and processor;
[0055] The memory is used to store computer-executable instructions, and the processor implements the steps of the video positioning model training method when executing the computer-executable instructions.
[0056] According to a fourth aspect of an embodiment of the present application, a computer-readable storage medium is provided, which stores computer-executable instructions, and when the instructions are executed by a processor, the steps of the video positioning model training method are implemented.
[0057] According to a fifth aspect of an embodiment of the present application, a chip is provided, which stores a computer program, and when the computer program is executed by the chip, the steps of the video positioning model training method are implemented.
[0058] The video localization model training method provided in the present application extracts word-level text features of sample text in a sample data set and frame-level video features of sample video in the sample data set through an initial video localization model; fuses the word-level text features with the frame-level video features to obtain sentence-level text features and visual features containing contextual semantic information; determines the stage-specific representation of the clipped video segment corresponding to the visual features, and selects candidate video segments from the clipped video segment according to the stage-specific representation; trains the initial video localization model through the candidate video segments and the real video segments in the sample data set until the training stop condition is met, thereby obtaining a target video localization model. The method achieves the acquisition of visual features that contain both global contextual information and local information, significantly improves the localization accuracy of the model, and promotes fine-grained cross-modal alignment of the model through regional semantic perception learning and primary semantic perception learning, thereby improving its combined generalization ability. BRIEF DESCRIPTION OF THE DRAWINGS
[0059] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.
[0060] Figure 1 is a flow chart of a video positioning model training method provided by an embodiment of the present application;
[0061] Figure 2 This is a flow chart of an implementation of a video positioning model training method provided in an embodiment of the present application;
[0062] Figure 3 It is a schematic diagram of a framework of a video positioning model training method provided in an embodiment of the present application;
[0063] Figure 4 This is a diagram showing the effect of a video positioning model training method provided by an embodiment of the present application;
[0064] Figure 5 It is a structural schematic diagram of a video positioning model training device provided by an embodiment of the present application;
[0065] Figure 6 It is a structural block diagram of a computing device provided in one embodiment of the present application. DETAILED DESCRIPTION
[0066] Many specific details are described in the following description to facilitate a full understanding of the present application. However, the present application can be implemented in many other ways than those described herein, and those skilled in the art can make similar generalizations without violating the connotation of the present application, so the present application is not limited by the specific implementation disclosed below.
[0067] The terms used in one or more embodiments of the present application are only for the purpose of describing specific embodiments, and are not intended to limit one or more embodiments of the present application. The singular forms of "a", "said" and "the" used in one or more embodiments of the present application and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used in one or more embodiments of the present application refers to and includes any or all possible combinations of one or more associated listed items.
[0068] It should be understood that, although the terms first, second, etc. may be used to describe various information in one or more embodiments of the present application, such information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of one or more embodiments of the present application, the first may also be referred to as the second, and similarly, the second may also be referred to as the first.
[0069] First, the terms involved in one or more embodiments of the present invention are explained.
[0070] GloVe model: is a model for generating word vectors, which captures the global semantic relationship between words by training on a large-scale corpus.
[0071] C3D network: (Convolutional 3D Networks) is a three-dimensional convolutional network based on deep learning, which aims to use three-dimensional convolutional neural networks to extract features in videos to achieve tasks such as video classification and behavior recognition.
[0072] I3D network: It is a new type of deep learning network that realizes the analysis of stereo images by combining a three-dimensional convolutional layer for image analysis and a bidirectional recurrent layer for time series analysis. It is mainly used for tasks such as video classification, action segmentation, action recognition, and 3D posture estimation.
[0073] In the present application, a video positioning model training method is provided. The present application also relates to a video positioning model training device, a computing device, and a computer-readable storage medium, which are described in detail one by one in the following embodiments.
[0074] Figure 1A flowchart of a video positioning model training method provided according to an embodiment of the present application is shown, which specifically includes the following steps:
[0075] Step S102: extracting word-level text features of sample texts in the sample data set and frame-level video features of sample videos in the sample data set through the initial video localization model;
[0076] Step S104: fusing the word-level text features with the frame-level video features to obtain sentence-level text features and visual features containing contextual semantic information;
[0077] Step S106: determining a stage-specific representation of the edited video segment corresponding to the visual feature, and selecting a candidate video segment from the edited video segment according to the stage-specific representation;
[0078] Step S108: The initial video positioning model is trained by using the candidate video segments and the real video segments in the sample data set until a training stop condition is met, thereby obtaining a target video positioning model.
[0079] Among them, a training data set is selected, that is, a sample data set, which contains an uncropped long video and a corresponding text description, that is, a sample video and a sample text, and then the word-level text features and frame-level video features are extracted through the initial video positioning model. The process of extracting the word-level text features of the sample text can be completed by the word vector extraction submodel contained in the initial video positioning model, such as the GloVe model, etc. The specific word vector extraction submodel type used is determined by the actual usage scenario, and this embodiment does not limit it. The process of extracting the frame-level video features of the sample video can be completed by the visual feature extraction submodel contained in the initial video positioning model. The visual feature extraction submodel can use an I3D or C3D neural network. The specific type of neural network used is determined by the actual usage scenario, and this embodiment does not limit it.
[0080] Based on this, the sample video is divided into clipped video segments. According to the division results, the word-level text features and frame-level video features are fused to obtain sentence-level text features corresponding to each clipped video segment and visual features containing contextual semantic information. Then, the stage-specific representation of each clipped video segment is calculated by constructing a main time tag reorganization unit; the candidate video segments are obtained by traversing the clipped video segments as the start and end moments respectively; the visual features of the candidate video segments are obtained by combining the stage-specific representations of the candidate video segments, and the matching score of each candidate video segment is predicted using the linear prediction layer of the initial video localization model. Finally, the offset distance between each candidate video segment and the real video segment is predicted by the initial video localization model, and the localization loss function is defined to constrain the model to realize the training of the initial video localization model to obtain the target video localization model.
[0081] It should be noted that the output of the target video positioning model is the target video segment, that is, the one with the highest matching score among the candidate video segments. It can be understood as the video segment predicted by the target video positioning model, in which the content expressed in the sample video is most consistent with the content expressed in the sample text. For example, in a 60-second sample video, the first 30 seconds show a cat resting on the sofa in the room, and in the last 30 seconds, the cat leaves the video screen, and the sample text related to the video is "a cat resting on the sofa", then the target video segment can be identified as the first 30 seconds of the video.
[0082] Then, after dividing the sample video, there are multiple clipped video segments whose contents are consistent with the contents expressed in the sample text. Using the above example, if the 60-second sample video is divided into 10 clipped video segments of 6 seconds, then the target video segment determined at this time should be the 5 clipped video segments corresponding to the first 30 seconds. Therefore, the number of clipped video segments contained in the target video segment is determined by the actual usage scenario and is not limited in this embodiment.
[0083] In addition, after obtaining the target video positioning model, when positioning a new video, the video to be positioned and the text to be queried are respectively input into the trained model, the matching score of each candidate segment is predicted, and the segment with the highest matching score is selected as the optimal segment. The start and end times of the segments are summed with the predicted offset distance to obtain the final predicted segment, thereby realizing the positioning of the video to be positioned.
[0084] Furthermore, in step S104, the word-level text features and the frame-level video features are fused to obtain the sentence-level text features and the visual features containing the contextual semantic information. In this embodiment, the specific implementation is as follows:
[0085] The video frames in the sample video are divided to obtain a plurality of video segments of equal length; the frame-level video features corresponding to each video segment are mean-pooled to obtain clip-level video features; the clip-level video features and the word-level text features are concatenated, and global semantic modeling is performed in conjunction with position encoding to obtain the sentence-level text features and the visual features containing contextual semantic information.
[0086] Specifically, if all frames in the sample video are divided into N clips of equal length, i.e., clip video segments, the frame-level video features in each clip are average-pooled to obtain the clip-level features, expressed as ,in, represents the i-th clip-level feature, Indicates video relevance; concatenates the video clip-level features and word-level text features, and inputs the combined position encoding into the visual-language converter (Transformer) module of the initial video localization model for global semantic modeling to obtain sentence-level text features And visual features , Represents the vector dimension of the corresponding vector.
[0087] Furthermore, in step S106, the process of determining the stage-specific representation of the edited video segment corresponding to the visual feature is specifically implemented as follows in this embodiment:
[0088] The visual features are converted into a start representation, an intermediate representation and an end representation of the corresponding video end; based on an adjustable weight matrix, the start representation, the intermediate representation and the end representation are nonlinearly combined to obtain the stage-specific representation.
[0089] Specifically, the process of converting the visual features into the start representation, middle representation and end representation of the corresponding video end can be completed by the feature projection layer included in the initial video positioning model, so as to realize the conversion of the visual features into the start, middle and end representations of each clip arranged in chronological order, which is specifically expressed as , MLP (·) represents the processing result of multi-layer perceptron. It should be noted that are the feature representations corresponding to the beginning, middle, and end stages of each clip, respectively. Represents an activation function, such as a ReLU activation function. For the feature representations of all the clipped video segments at different stages, traverse each clipped video segment as the start and end time to obtain candidate segments, and calculate the feature representation of each candidate segment to construct a two-dimensional time segment feature map, which is expressed as ,in, It also represents the vector dimension of the corresponding vector.
[0090] In addition, for each candidate video segment, the start representation of the start clip, the middle representation of the middle clip, and the end representation of the end clip are respectively called the main features of the candidate video segment, denoted as , these three main features are nonlinearly combined to obtain the final feature representation of the candidate video segment. This process is expressed as ,in , and represents the trainable weight matrix, Represents the bias vector.
[0091] Furthermore, in step S108, the initial video positioning model is trained by using the candidate video segments and the real video segments in the sample data set. In this embodiment, the specific implementation is as follows:
[0092] Based on the candidate video segment and the real video segment, a matching score and an offset distance of the candidate video segment are calculated; based on the matching score and the offset distance, a weighted binary cross entropy loss and an L1 loss are calculated; based on the weighted binary cross entropy loss and the L1 loss, the initial video positioning model is trained.
[0093] Specifically, the matching score of the candidate video segment compared to the real video segment is predicted by the linear prediction layer included in the initial video localization model. and offset distance . For the predicted matching score and offset distance , respectively calculate the weighted binary cross entropy loss of the true value corresponding to the real video segment and L1 loss ,as follows:
[0094] ,
[0095] ,
[0096] in, represents the number of valid candidate segments, Q represents the number of valid candidate segments that pass the threshold The number of candidate segments after suppression; and Respectively represent the predefined moment position of the candidate segment and the actual moment position, represents the predicted matching score of the i-th candidate video segment, represents the actual matching score of the i-th candidate video segment, It represents the offset distance between the predicted position of the i-th candidate video segment and the starting position of the real video segment. It represents the offset distance between the predicted end position of the i-th candidate video segment and the real video segment.
[0097] Furthermore, based on the weighted binary cross entropy loss and L1 loss, the initial video positioning model is trained. In this embodiment, the specific implementation process is as follows:
[0098] The sum of the weighted binary cross entropy loss and the L1 loss is calculated to obtain the moment positioning loss; the video segment corresponding to the real video segment is determined as a positive sample, and negative samples are selected from other video segments and other videos outside the positive sample; the contrastive learning loss is calculated according to the positive sample and the negative sample; and the initial video positioning model is trained based on the moment positioning loss and the contrastive learning loss.
[0099] Among them, and Sum and get the loss for time positioning In order to ensure the accuracy of the video localization model, negative sample segments within and between videos can be mined during the regional semantic perception learning process, and cross-modal contrastive learning can be performed in the joint embedding based on the sentence-level text features and the visual features obtained above, and the contrastive learning loss within and between videos can be calculated.
[0100] Specifically, the video segments in the sample video corresponding to the real video segments are taken as positive samples, negative samples are mined from the same video and between different videos, and the contrastive learning loss is calculated in the joint embedding space. The process is:
[0101] ,
[0102] in, and denote the text and visual features in the joint embedding space, respectively. represents a positive sample, represents the set of negative samples, represents the temperature coefficient, express The elements in .
[0103] Furthermore, based on the moment positioning loss and the contrastive learning loss, the process of training the initial video positioning model is specifically implemented as follows in this embodiment:
[0104] Through a preset masking strategy, the basic components of the sample text and the sample video are masked to obtain positive and negative video sentence sample pairs, wherein the basic components of the sample text are words, and the basic components of the sample video are video frames; the positive and negative video sentence sample pairs are processed through the initial video positioning model, and the semantic contrastive learning loss is calculated based on the processing results; the initial video positioning model is trained based on the moment positioning loss, the contrastive learning loss and the semantic contrastive learning loss.
[0105] Furthermore, the basic components of the sample text and the sample video are masked by a preset masking strategy to obtain the positive and negative video sentence sample pairs. In this embodiment, the specific implementation method is as follows:
[0106] Different weights are assigned to the video frames in the sample video corresponding to the real video segment and the video frames not corresponding to the real video segment, and the assignment results are shielded based on a preset shielding strategy to obtain positive video samples and negative video samples; different weights are assigned according to the word classes of words in the sample text, and the assignment results are shielded based on a preset shielding strategy to obtain positive sentence samples and negative sentence samples; positive video sentence sample pairs are constructed based on the positive video samples and the positive sentence samples, and negative video sentence sample pairs are constructed based on the negative video samples and the negative sentence samples, wherein the positive and negative video sentence sample pairs include the positive video sentence sample pairs and the negative video sentence sample pairs.
[0107] Furthermore, the positive and negative video sentence sample pairs are processed by the initial video localization model, and the semantic contrast learning loss is calculated based on the processing results. In this embodiment, the specific implementation method is as follows:
[0108] The positive video sentence sample pair is processed by the initial video positioning model to obtain a first predicted matching score; the negative video sentence sample pair is processed by the initial video positioning model to obtain a second predicted matching score; and the semantic contrastive learning loss is calculated based on the first predicted matching score, the second predicted matching score and the matching score.
[0109] The word classes of words can be divided into nouns, verbs, adjectives, adverbs, etc. The specific word classes are determined by the words contained in the sample text, and this embodiment does not limit them.
[0110] It should be noted that the weights assigned to the video frames corresponding to the real video segments and the weights assigned to the video frames not corresponding to the real video segments can be adjusted according to the actual usage scenario, and the specific weight settings are not limited in this embodiment. In addition, in the process of shielding the above assignment results, shielding can be performed in proportion according to the weight values after the assignment, and the specific proportion value setting is also determined by the actual usage scenario and is not limited in this embodiment.
[0111] For example, the frames in the sample video corresponding to the real video segment are assigned a weight value of 1, and the other frames are assigned a random weight between (0, 1). The positive video sample is obtained by randomly masking half of the 50% frames with smaller weights, which is expressed as , randomly mask 40% of the frames with weight 1 to obtain negative video samples, expressed as At the same time, the nouns and verbs in the query sentences in the sample text are given a weight of 1, the adjectives and adverbs are given a weight of 0.6, and the other words are given a weight of 0. Randomly mask half of the words in the bottom 30% and top 70% of the weight ranking to obtain positive sentence samples and negative sentence samples, which are represented as and For the positive video sentence sample pairs and the negative video sentence sample pairs, they are used as the input of the initial video localization model to predict the target video segment, and the semantic contrastive learning loss is constructed based on the matching scores of the candidate segments corresponding to the above inputs and the prediction results of the real video segments. as follows:
[0112] .
[0113] Among them, P(·) represents the IoU value between all candidate video segments predicted by the model and the real video segments, and S represents the actual input query sentence.
[0114] Furthermore, based on the moment positioning loss, contrastive learning loss and semantic contrastive learning loss, the process of training the initial video positioning model is specifically implemented as follows in this embodiment:
[0115] According to the first predicted matching score and the matching score, a first binary cross entropy loss is calculated; the first binary cross entropy loss is summed with the semantic contrastive learning loss to obtain a main semantic perception loss; based on the moment positioning loss, the contrastive learning loss and the main semantic perception loss, the initial video positioning model is trained.
[0116] Among them, after the positive and negative video sentence sample pairs are input into the initial video localization model, the candidate video segments are determined, as in the process of calculating the matching score above, and the corresponding first predicted matching score is calculated, and the weighted binary cross entropy loss is calculated as above based on the first predicted matching score and the matching score. The process of calculating the first binary cross entropy loss , and finally and Sum and get the final main semantic perception loss, recorded as .
[0117] Furthermore, based on the moment positioning loss, contrastive learning loss and main semantic perception loss, the process of training the initial video positioning model is specifically implemented as follows in this embodiment:
[0118] Randomly block words in the sample text, and reconstruct the blocked sample text through the initial video positioning model; calculate the cross entropy loss based on the reconstruction result and the sample text; weightedly sum the moment positioning loss, the contrastive learning loss, the main semantic perception loss and the cross entropy loss based on a preset weighting coefficient to obtain a target loss function; train the initial video positioning model through the target loss function.
[0119] Among them, in order to further improve the prediction accuracy of the model, the amount of training materials is increased by means of shielding reconstruction. It should be noted that in the process of randomly shielding words in the sample text, the shielding ratio is determined by the actual usage scenario, which is not limited in this embodiment.
[0120] For example, during model training, use <mask>The markers randomly replace 15% of the words in the query sentence. Given the visual information, the model needs to reconstruct the masked words based on the unmasked words. The reconstructed results are used to calculate the cross entropy loss between the original words to assist model training, denoted as .
[0121] In addition, in the process of weighted summing the above loss functions with the preset weighting coefficients, the weighting coefficients are adjustable parameters, so the target loss function is as follows:
[0122] ,
[0123] in, , and is a hyperparameter for adjusting the proportion of each loss.
[0124] In actual usage scenarios, the above video localization model training method was tested. During the test, the R@1 indicator with an IoU (intersection over union) threshold was used. The IoU thresholds include 0.5 and 0.7, which means that in all query texts, the IoU ratio of the predicted segment to the true target segment is greater than the threshold. At the same time, the mIoU (mean intersection over union) indicator can also be used, which represents the average value of the intersection over union ratios of all query prediction segments and true target segments.
[0125] Specifically, Figure 2 As shown in the implementation flow chart of a video positioning model training method provided, by selecting a data set, a combined moment positioning model is constructed, that is, an initial video positioning model. After training the model using a loss function, a target video positioning model is obtained, and then parameters are loaded for moment positioning.
[0126] In addition, according to Figure 3 As shown in the framework diagram of a video localization model training method provided, the core idea of the video localization model is to fuse features of different modalities through the backbone network of the visual-language Transformer, and at the same time, to obtain feature representations containing sufficient contextual information and local area information by combining stage-specific representations of video clips. These feature representations are further used for contrastive learning in the joint visual-text embedding space, thereby promoting cross-modal semantic alignment, and thus significantly improving the accuracy of the model in moment-to-moment localization. In addition, a main semantic perception learning process is proposed, which can enable the model to learn the fine-grained correspondence between video and sentence by perceiving the semantic changes of video and text, thereby further improving the combined generalization ability of the model.
[0127] The following are the measured results in three specific application scenarios:
[0128] The first scenario case: Model training and testing are performed on the Charades-CG dataset, and the results are compared with previous methods. The Charades-CG dataset contains a training set for model training, a Test-Trivial test set, a Novel-Composition test set, and a Novel-Word test set, which contain 8281, 3096, 3442, and 703 video-text pairs, respectively. The Novel-Composition test set contains word combinations that have not been seen in the training set and is used to test the combinatorial generalization performance of the model. The Novel-Word test set contains words that have not been seen in the training set and is used to test the model's combinatorial generalization performance for unseen words. The Test-Trivial test set contains the same data distribution as the training set. The test results on Charades-CG are shown in Table 1:
[0129] Table 1 Comparison of moment positioning effects on the Charades-CG dataset
[0130]
[0131] Second scenario case: Model training and testing are performed on the ActivityNet-CG dataset, and the results are compared with previous methods. The ActivityNet-CG dataset includes a training set for model training, a Test-Trivial test set, a Novel-Composition test set, and a Novel-Word test set, which contain 36724, 15712, 12028, and 3944 video-text pairs, respectively. The Novel-Composition test set contains word combinations not seen in the training set, which is used to test the combinatorial generalization performance of the model. The Novel-Word test set contains words not seen in the training set, which is used to test the model's combinatorial generalization performance for unseen words. The Test-Trivial test set contains the same data distribution as the training set. The test results on ActivityNet-CG are shown in Table 2:
[0132] Table 2 Comparison of moment positioning effects on the Activitynet-Captions dataset
[0133]
[0134] The third scenario example: on the Charades-CG dataset, the actual result of using the method proposed in this invention to locate the video moment for a given query text. Figure 4 As shown in the effect display diagram of a video positioning model training method provided, the video positioning model training method provided in this embodiment successfully locates the corresponding segments in the video.
[0135] Corresponding to the above method embodiment, the present application also provides a video positioning model training device embodiment, Figure 5 FIG. 1 is a schematic diagram showing the structure of a video positioning model training device provided by an embodiment of the present application. Figure 5 As shown, the device comprises:
[0136] An extraction module 502 is configured to extract word-level text features of sample texts in the sample data set and frame-level video features of sample videos in the sample data set through an initial video localization model;
[0137] A fusion module 504 is configured to fuse the word-level text features with the frame-level video features to obtain sentence-level text features and visual features containing contextual semantic information;
[0138] A selection module 506 is configured to determine a stage-specific representation of the edited video segment corresponding to the visual feature, and select a candidate video segment from the edited video segment according to the stage-specific representation;
[0139] The training module 508 is configured to train the initial video positioning model through the candidate video segments and the real video segments in the sample data set until a training stop condition is met to obtain a target video positioning model.
[0140] In an optional embodiment, the fusion module 504 is further configured to:
[0141] The video frames in the sample video are divided to obtain a plurality of video segments of equal length; the frame-level video features corresponding to each video segment are mean-pooled to obtain clip-level video features; the clip-level video features and the word-level text features are concatenated, and global semantic modeling is performed in conjunction with position encoding to obtain the sentence-level text features and the visual features containing contextual semantic information.
[0142] In an optional embodiment, the selection module 506 is further configured to:
[0143] The visual features are converted into a start representation, an intermediate representation and an end representation of the corresponding video end; based on an adjustable weight matrix, the start representation, the intermediate representation and the end representation are nonlinearly combined to obtain the stage-specific representation.
[0144] In an optional embodiment, the training module 508 is further configured to:
[0145] Based on the candidate video segment and the real video segment, a matching score and an offset distance of the candidate video segment are calculated; based on the matching score and the offset distance, a weighted binary cross entropy loss and an L1 loss are calculated; based on the weighted binary cross entropy loss and the L1 loss, the initial video positioning model is trained.
[0146] In an optional embodiment, the training module 508 is further configured to:
[0147] The sum of the weighted binary cross entropy loss and the L1 loss is calculated to obtain the moment positioning loss; the video segment corresponding to the real video segment is determined as a positive sample, and negative samples are selected from other video segments and other videos outside the positive sample; the contrastive learning loss is calculated according to the positive sample and the negative sample; and the initial video positioning model is trained based on the moment positioning loss and the contrastive learning loss.
[0148] In an optional embodiment, the training module 508 is further configured to:
[0149] Through a preset masking strategy, the basic components of the sample text and the sample video are masked to obtain positive and negative video sentence sample pairs, wherein the basic components of the sample text are words, and the basic components of the sample video are video frames; the positive and negative video sentence sample pairs are processed through the initial video positioning model, and the semantic contrastive learning loss is calculated based on the processing results; the initial video positioning model is trained based on the moment positioning loss, the contrastive learning loss and the semantic contrastive learning loss.
[0150] In an optional embodiment, the training module 508 is further configured to:
[0151] Different weights are assigned to the video frames in the sample video corresponding to the real video segment and the video frames not corresponding to the real video segment, and the assignment results are shielded based on a preset shielding strategy to obtain positive video samples and negative video samples; different weights are assigned according to the word classes of words in the sample text, and the assignment results are shielded based on a preset shielding strategy to obtain positive sentence samples and negative sentence samples; positive video sentence sample pairs are constructed based on the positive video samples and the positive sentence samples, and negative video sentence sample pairs are constructed based on the negative video samples and the negative sentence samples, wherein the positive and negative video sentence sample pairs include the positive video sentence sample pairs and the negative video sentence sample pairs.
[0152] In an optional embodiment, the training module 508 is further configured to:
[0153] The positive video sentence sample pair is processed by the initial video positioning model to obtain a first predicted matching score; the negative video sentence sample pair is processed by the initial video positioning model to obtain a second predicted matching score; and the semantic contrastive learning loss is calculated based on the first predicted matching score, the second predicted matching score and the matching score.
[0154] In an optional embodiment, the training module 508 is further configured to:
[0155] According to the first predicted matching score and the matching score, a first binary cross entropy loss is calculated; the first binary cross entropy loss is summed with the semantic contrastive learning loss to obtain a main semantic perception loss; based on the moment positioning loss, the contrastive learning loss and the main semantic perception loss, the initial video positioning model is trained.
[0156] In an optional embodiment, the training module 508 is further configured to:
[0157] Randomly block words in the sample text, and reconstruct the blocked sample text through the initial video positioning model; calculate the cross entropy loss based on the reconstruction result and the sample text; weightedly sum the moment positioning loss, the contrastive learning loss, the main semantic perception loss and the cross entropy loss based on a preset weighting coefficient to obtain a target loss function; train the initial video positioning model through the target loss function.
[0158] The video positioning model training device provided by the present application extracts word-level text features of sample texts in a sample data set and frame-level video features of sample videos in the sample data set through an initial video positioning model; fuses the word-level text features with the frame-level video features to obtain sentence-level text features and visual features containing contextual semantic information; determines the stage-specific representation of the clipped video segment corresponding to the visual features, and selects candidate video segments from the clipped video segment according to the stage-specific representation; trains the initial video positioning model through the candidate video segments and the real video segments in the sample data set until the training stop condition is met, thereby obtaining a target video positioning model. Visual features containing both global contextual information and local information are obtained through the main time tag reorganization module, which significantly improves the positioning accuracy of the model; and proposes two semantic perception learning processes, namely regional semantic perception learning and main semantic perception learning, to promote fine-grained cross-modal alignment of the model, thereby improving its combined generalization ability; in addition, it has been experimentally verified that the method has surpassed the previous methods in the time positioning task, especially when the query sentence contains novel words, the method has significantly improved the effect compared with the previous methods.
[0159] The above is a schematic scheme of a video positioning model training device of the present embodiment. It should be noted that the technical scheme of the video positioning model training device and the technical scheme of the above-mentioned video positioning model training method belong to the same concept. For the details not described in detail in the technical scheme of the video positioning model training device, please refer to the description of the technical scheme of the above-mentioned video positioning model training method. In addition, each component in the device embodiment should be understood as a functional module that must be established to implement each step of the program flow or each step of the method, and each functional module is not an actual functional division or separation definition. The device claim defined by such a group of functional modules should be understood as a functional module architecture that mainly implements the solution through the computer program recorded in the specification, and should not be understood as a physical device that mainly implements the solution through hardware.
[0160] Figure 6 The block diagram of a computing device 600 according to an embodiment of the present application is shown. The components of the computing device 600 include but are not limited to a memory 610 and a processor 620. The processor 620 is connected to the memory 610 via a bus 630, and the database 650 is used to store data.
[0161] The computing device 600 also includes an access device 640 that enables the computing device 600 to communicate via one or more networks 660. Examples of these networks include a public switched telephone network (PSTN), a local area network (LAN), a wide area network (WAN), a personal area network (PAN), or a combination of communication networks such as the Internet. The access device 640 may include one or more of any type of network interface (e.g., a network interface card (NIC)) that is wired or wireless, such as an IEEE 802.11 wireless local area network (WLAN) wireless interface, a World Wide Interoperability for Microwave Access (Wi-MAX) interface, an Ethernet interface, a universal serial bus (USB) interface, a cellular network interface, a Bluetooth interface, a near field communication (NFC) interface, and the like.
[0162] In one embodiment of the present application, the above components of the computing device 600 and Figure 6 Other components not shown in the figure may also be connected to each other, for example, via a bus. It should be understood that Figure 6 The computing device structure block diagram shown is only for the purpose of illustration, and is not intended to limit the scope of the present application. Those skilled in the art may add or replace other components as needed.
[0163] The computing device 600 may be any type of stationary or mobile computing device, including a mobile computer or mobile computing device (e.g., a tablet computer, a personal digital assistant, a laptop computer, a notebook computer, a netbook, etc.), a mobile phone (e.g., a smart phone), a wearable computing device (e.g., a smart watch, smart glasses, etc.), or other types of mobile devices, or a stationary computing device such as a desktop computer or PC. The computing device 600 may also be a mobile or stationary server.
[0164] Among them, the processor 620 is used to execute computer executable instructions of each step of the video positioning model training method.
[0165] The above is a schematic scheme of a computing device of this embodiment. It should be noted that the technical scheme of the computing device and the technical scheme of the above-mentioned video positioning model training method belong to the same concept, and the details not described in detail in the technical scheme of the computing device can be referred to the description of the technical scheme of the above-mentioned video positioning model training method.
[0166] An embodiment of the present application also provides a computer-readable storage medium storing computer instructions, which, when executed by a processor, are used to execute the steps of the video positioning model training method.
[0167] The above is a schematic scheme of a computer-readable storage medium of this embodiment. It should be noted that the technical scheme of the storage medium and the technical scheme of the above-mentioned video positioning model training method belong to the same concept, and the details not described in detail in the technical scheme of the storage medium can be referred to the description of the technical scheme of the above-mentioned video positioning model training method.
[0168] An embodiment of the present application further provides a chip storing a computer program, which implements the steps of the video positioning model training method when executed by the chip.
[0169] The above describes specific embodiments of the present application. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.
[0170] The computer instructions include computer program codes, which may be in source code form, object code form, executable files or some intermediate forms, etc. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium, etc. It should be noted that the content contained in the computer-readable medium may be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electric carrier signals and telecommunication signals.
[0171] It should be noted that, for the above-mentioned method embodiments, for the sake of simplicity of description, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the described action sequence, because according to the present application, certain steps can be performed in other sequences or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present application.
[0172] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.
[0173] The preferred embodiments of the present application disclosed above are only used to help explain the present application. The optional embodiments do not describe all the details in detail, nor do they limit the invention to the specific implementation methods described. Obviously, many modifications and changes can be made according to the content of the present application. The present application selects and specifically describes these embodiments in order to better explain the principles and practical applications of the present application, so that those skilled in the art can understand and use the present application well. The present application is only limited by the claims and their full scope and equivalents.< / mask>
Claims
1. A video positioning model training method, characterized in that: include: Extracting word-level text features of sample texts in the sample data set and frame-level video features of sample videos in the sample data set through an initial video localization model; Fusion of the word-level text features and the frame-level video features to obtain sentence-level text features and visual features containing contextual semantic information; Determine a stage-specific representation of the edited video segment corresponding to the visual feature, and select a candidate video segment from the edited video segment according to the stage-specific representation, wherein the process of determining the stage-specific representation is to convert the visual feature into a start representation, an intermediate representation, and an end representation of the corresponding video end, and perform a nonlinear combination of the start representation, the intermediate representation, and the end representation based on an adjustable weight matrix to obtain the stage-specific representation; The initial video positioning model is trained by the candidate video segment and the real video segment in the sample data set until the training stop condition is met to obtain the target video positioning model, wherein the training process of the initial video positioning model is to calculate the matching score and the offset distance of the candidate video segment based on the candidate video segment and the real video segment, calculate the weighted binary cross entropy loss and the L1 loss according to the matching score and the offset distance, calculate the sum of the weighted binary cross entropy loss and the L1 loss, obtain the moment positioning loss, determine the video segment corresponding to the real video segment as a positive sample, and select the video segment outside the positive sample. Negative samples are selected from other video segments and other videos, and a contrastive learning loss is calculated based on the positive samples and the negative samples. The basic components of the sample text and the sample video are masked through a preset masking strategy to obtain positive and negative video sentence sample pairs, wherein the basic components of the sample text are words, and the basic components of the sample video are video frames. The positive and negative video sentence sample pairs are processed through the initial video positioning model, and the semantic contrastive learning loss is calculated based on the processing results. The initial video positioning model is trained based on the moment positioning loss, the contrastive learning loss and the semantic contrastive learning loss.
2. The method according to claim 1, characterized in that The fusing of the word-level text features and the frame-level video features to obtain sentence-level text features and visual features containing contextual semantic information includes: Dividing the video frames in the sample video to obtain a plurality of video segments of equal length; Perform mean pooling on the frame-level video features corresponding to each video segment to obtain the clip-level video features; The clip-level video features and the word-level text features are connected in series, and global semantic modeling is performed in conjunction with position encoding to obtain the sentence-level text features and the visual features containing contextual semantic information.
3. The method according to claim 1, characterized in that: The method of masking the basic components of the sample text and the sample video by using a preset masking strategy to obtain positive and negative video sentence sample pairs includes: Assigning different weights to the video frames in the sample video corresponding to the real video segment and the video frames not corresponding to the real video segment, respectively, and masking the assignment results based on a preset masking strategy to obtain positive video samples and negative video samples; According to the word classes of the words in the sample text, different weights are assigned respectively, and based on a preset shielding strategy, the assignment results are shielded to obtain positive sentence samples and negative sentence samples; A positive video sentence sample pair is constructed based on the positive video sample and the positive sentence sample, and a negative video sentence sample pair is constructed based on the negative video sample and the negative sentence sample, wherein the positive and negative video sentence sample pairs include the positive video sentence sample pair and the negative video sentence sample pair.
4. The method according to claim 3, characterized in that The processing of the positive and negative video sentence sample pairs by the initial video localization model and calculating the semantic contrast learning loss based on the processing results include: Processing the positive video sentence sample pair through the initial video positioning model to obtain a first predicted matching score; Processing the negative video sentence sample pair through the initial video localization model to obtain a second predicted matching score; The semantic contrastive learning loss is calculated according to the first predicted matching score, the second predicted matching score, and the matching score.
5. The method according to claim 4, characterized in that The training of the initial video positioning model based on the moment positioning loss, the contrastive learning loss and the semantic contrastive learning loss includes: Calculating a first binary cross entropy loss based on the first predicted matching score and the matching score; Summing the first binary cross entropy loss and the semantic contrastive learning loss to obtain a main semantic perception loss; The initial video localization model is trained based on the moment localization loss, the contrastive learning loss and the main semantic perception loss.
6. The method according to claim 5, characterized in that The training of the initial video positioning model based on the moment positioning loss, the contrastive learning loss and the main semantic perception loss includes: Randomly masking words in the sample text, and reconstructing the masked sample text using the initial video positioning model; Calculate the cross entropy loss based on the reconstruction result and the sample text; The moment positioning loss, the contrastive learning loss, the main semantic perception loss and the cross entropy loss are weighted and summed based on a preset weighting coefficient to obtain a target loss function; The initial video positioning model is trained using the target loss function.
