Video segment positioning method, system, medium and device for real-time tasks

Abstract

The present invention belongs to the field of multimedia retrieval technology, and provides a video clip positioning method, system, medium and device for real-time tasks. The technical scheme is as follows: after obtaining the video and query statement, the corresponding feature representation is obtained through the image feature processing model and the natural language feature processing model; the feature representations of two different modalities, video and query statement, are mapped to the same Hamming space through cross-modal dynamic hash coding, and the dynamic hash can adjust the hash coding according to the data input; the knowledge distillation technology is used to extract knowledge from a teacher model with large parameter volume and high performance, and migrate it to a student model with small parameter volume and low complexity, so as to construct a lightweight video clip positioning model. The present invention constructs a lightweight model by integrating dynamic hash coding technology and distillation learning technology, which realizes efficient capture of the deep semantics of the video content, while ensuring the rapid response and accuracy of real-time tasks, and solves the difficulty of video clip positioning.

Description

Technical Field

[0001] The present invention belongs to the field of multimedia retrieval technology, and in particular relates to a video segment positioning method, system, medium and equipment for real-time tasks. Background Art

[0002] The statements in this section merely provide background information related to the present invention and do not necessarily constitute prior art.

[0003] Existing methods for locating video clips mainly involve two core areas: computer vision (CV) and natural language processing (NLP). In computer vision, it is necessary to process and analyze the frames in the video to extract effective visual features such as object shape, color, and motion trajectory, providing strong support for subsequent video clip positioning; in natural language processing, this technology needs to parse and process the query language input by the user, understand the user's query intention and needs, and ensure the accuracy of video clip positioning. In order to achieve accurate positioning of video clips, cross-modal hashing technology is applied, which maps the visual features in the video and the text features in the query language into the same hash space to achieve effective association and matching between vision and text. The development and application of this technology will play an important role in video surveillance, video retrieval and other fields, and bring more convenience and benefits to people's lives and work.

[0004] However, there are certain difficulties in the current video clip positioning, including:

[0005] 1) Existing video clip positioning technologies mostly focus on real-time positioning of shorter video content. For real-time application scenarios, video sequences of several hours or even longer are not fast enough to process. The complex scene changes, diverse event flows, and potentially large amounts of irrelevant information contained in long video sequences make it difficult for existing technologies to fully meet the needs of fast processing of long video sequences in real-time environments.

[0006] 2) The training process of cross-modal hashing technology often faces significant instability problems. This is mainly due to the inherent differences and complexity between different modal data. During the training process, these differences may make it difficult to establish a stable mapping relationship of the hash function, which in turn makes the model face huge challenges in capturing the correlation between modalities and extracting effective features. In addition, the diversity and uneven distribution of training samples may also exacerbate this instability, requiring the model to have more complex optimization strategies and more iterations to achieve the desired effect, thereby greatly increasing the complexity and computational cost of model training;

[0007] 3) Real-time video segment localization faces the dual challenges of performance and efficiency. In practical applications, although the attention-based model has shown significant advantages in performance, it has encountered dual bottlenecks in processing speed and resource efficiency. Its high computational complexity and huge model parameters have become a stumbling block that hinders its adaptation to real-time processing needs, limiting the widespread application and deployment of these advanced models in complex and changing environments. Summary of the invention

[0008] In order to solve at least one technical problem existing in the above-mentioned background technology, the present invention provides a video segment positioning method, system, medium and device for real-time tasks, which integrates dynamic hash coding technology and distillation learning technology to construct a lightweight model, thereby achieving efficient capture of the deep semantics of video content while ensuring rapid response and accuracy of real-time tasks.

[0009] In order to achieve the above object, the present invention adopts the following technical solution:

[0010] A first aspect of the present invention provides a video segment positioning method for real-time tasks, comprising the following steps:

[0011] Get historical video data and query statements;

[0012] A matching score between the first video segment and the query is obtained based on historical video data, the query statement and the trained teacher model; when the teacher model is trained, the feature representations of different video segments and the query statement are mapped to a binary hash space, and the hash coding loss of the hash coding is dynamically adjusted according to different feature representation changes;

[0013] The knowledge of the teacher model is transferred to the lightweight student model through knowledge distillation, and the matching score between the second video clip and the query is obtained through the student model;

[0014] A distillation loss is calculated based on a matching score between the first video clip and the query and a matching score between the second video clip and the query;

[0015] The total loss function is obtained by combining the hash coding loss and the distillation loss. The video clip features located by minimizing the total loss function are used as the video clip corresponding to the query statement.

[0016] Furthermore, when mapping the feature representations of different video clips and query statements to a binary hash space, the features of different video clips and query statements are mapped to the binary hash space according to the characteristics of the features of different video clips and query statements to obtain hash codes of the video clips and hash codes of the query statements;

[0017] A dynamic hash function is introduced to dynamically adjust the hash function according to the error between the current feature and the hash code to make the hash code adapt to the features of different modes. The hash function parameters corresponding to the minimum error between the current feature and the hash code are used as the optimal hash function parameters to obtain the optimal hash function.

[0018] Furthermore, the feature representation extraction process of different video clips includes:

[0019] Split the video into multiple continuous video segments in chronological order, and the length of the segments is set according to the requirements of the C3D model;

[0020] The pre-trained C3D model is used as the backbone network to extract features from each video clip. The C3D model is used to simultaneously capture the spatial and temporal information in the video. The video clips are converted into high-dimensional feature vectors through convolutional layers, pooling layers, and fully connected layers, and the features of each video clip are obtained in the output layer.

[0021] Furthermore, the process of extracting the feature representation of the query sentence includes:

[0022] Load the pre-trained GloVe vector model and map each word in the query text to its corresponding GloVe word vector;

[0023] The obtained Glove word vector matrix is ​​used as the input of the BiGRU model, and a hidden state is output at each time step. For the hidden state of each time step output by the BiGRU model, the hidden state of the last time step is selected as the vector representation of the entire sentence.

[0024] Furthermore, when the teacher model is trained, the interaction between the two modalities of video and query statement is learned, and the matching score of the query statement corresponding to each video clip is predicted based on the learned interaction results. The video clip with the highest matching score is taken as the video clip with the highest query matching degree.

[0025] Furthermore, the expression of the total loss function is:

[0026] ,

[0027] ,

[0028] in, is the total loss function, is the weight coefficient, is the hash encoding loss, is the distillation loss, Output the matching score between the video clip and the query to the teacher model, Outputs the matching score between the video clip and the query for the student model.

[0029] Furthermore, after obtaining the historical video data and query statements, the video data and query statements are preprocessed, and the video data is preprocessed, including sampling the video frames, performing video denoising and video normalization operations on the video; irrelevant auxiliary words and modal particles in the query statements are removed, and then a segmentation operation is performed to obtain each word block.

[0030] A second aspect of the present invention provides a video segment positioning system for real-time tasks, comprising:

[0031] A data acquisition module, which is used to acquire historical video data and query statements;

[0032] A teacher model training module, which is used to obtain a matching score between a first video segment and a query based on historical video data, a query statement and a trained teacher model; when the teacher model is trained, the hash coding loss of the hash code is dynamically adjusted according to different feature representation changes when calculating the feature representations of different video segments and query statements mapped to a binary hash space;

[0033] A knowledge distillation module, which is used to transfer the knowledge of the teacher model to the lightweight student model through knowledge distillation, and obtain a matching score between the second video clip and the query through the student model;

[0034] A video positioning module is used to calculate the distillation loss based on the matching scores of the first video clip and the query and the matching scores of the second video clip and the query; a total loss function is obtained by combining the hash coding loss and the distillation loss, and the video clip features are positioned with the strategy of minimizing the total loss function as the video clip corresponding to the query statement.

[0035] A third aspect of the present invention provides a computer-readable storage medium.

[0036] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps in the above-mentioned method for locating video segments oriented to real-time tasks.

[0037] A fourth aspect of the present invention provides a computer device.

[0038] A computer device comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps in the above-mentioned method for locating video segments for real-time tasks when executing the program.

[0039] Compared with the prior art, the present invention has the following beneficial effects:

[0040] 1. The present invention improves computational efficiency through cross-modal hashing technology. The features of the video clip and the query statement are mapped to a low-dimensional binary hash space, which significantly reduces the amount of computation while effectively maintaining the expressiveness of the features, providing an efficient solution for the video clip positioning task;

[0041] 2. The present invention further improves the accuracy and robustness of hash coding through dynamic hash optimization. Traditional hash coding is static and cannot adapt to changes in different input data, while dynamic hash optimization can dynamically adjust hash coding according to changes in input features, so that it can better adapt to different input data;

[0042] 3. The present invention uses distillation learning technology to effectively retain the knowledge in the large teacher model in the lightweight student model; while maintaining lightweight, the performance of the student model can be as close as possible to the teacher model. This model design not only significantly improves the response speed of video clip positioning, but also maintains a deep semantic understanding of the video and query content, and is suitable for real-time tasks with high efficiency requirements.

[0043] Advantages of additional aspects of the present invention will be given in part in the following description, and in part will become obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] The accompanying drawings in the specification, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0045] Figure 1 It is a flowchart of a video segment positioning method for real-time tasks provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0046] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0047] It should be noted that the following detailed descriptions are all illustrative and intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present invention belongs.

[0048] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit exemplary embodiments according to the present invention. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprising" and / or "including" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof.

[0049] Video Moment Localization is an advanced multimedia retrieval technology that uses the deep integration of computer vision and natural language processing technologies to accurately identify and locate video clips that match user query content from massive video data. This technology is regarded as an important branch in the field of video content understanding and retrieval. Its core task is to automatically retrieve and return the start and end time of video clips containing target events, scenes or actions in unedited video sequences based on natural language descriptions or keywords provided by users. Video clip localization technology not only greatly improves the retrieval efficiency and accuracy of video content, but also provides strong technical support for multiple fields such as video analysis, intelligent monitoring, and online education.

[0050] In response to the issues mentioned in the background technology, such as how to quickly locate target segments in longer videos in real-time tasks, how to construct a stable and efficient cross-modal hashing process, and how to optimize the model architecture, reduce computing overhead, and improve resource utilization efficiency, the present invention provides a method for building a lightweight model by combining knowledge distillation technology and cross-modal hashing technology, thereby greatly reducing its computing burden so that it can be configured in edge devices. It can not only efficiently capture the deep semantics of video content, but also ensure rapid response and accuracy to real-time tasks.

[0051] The overall concept of the present invention is: after obtaining the video and the query statement, use preprocessing to select the video frame and mark the query statement, and obtain the corresponding feature representation through the image feature processing model and the natural language feature processing model; through cross-modal dynamic hash coding, the feature representations of two different modalities of video and query statement are mapped to the same Hamming space, and dynamic hashing can adjust the hash coding according to the data input; use knowledge distillation technology to extract knowledge from a teacher model with large parameters and high performance, and transfer it to a student model with small parameters and low complexity, so as to build a lightweight video clip positioning model. The present invention constructs a lightweight model by integrating dynamic hash coding technology and distillation learning technology, which realizes the efficient capture of the deep semantics of the video content while ensuring the rapid response and accuracy of real-time tasks.

[0052] Embodiment 1

[0053] like Figure 1 As shown, this embodiment provides a target video segment positioning method for real-time tasks, including the following steps:

[0054] Step 1: Obtain video data and query statements, and preprocess the video and query statements;

[0055] In step 1, the preprocessing of the video and query statement includes:

[0056] Considering that the video has a large number of frames and the changes between adjacent frames are small and smooth, the video frames are sampled to reduce the amount of calculation when obtaining the video feature representation.

[0057] The query sentence is a sentence with a complete grammatical structure. Unrelated auxiliary words and modal particles are removed, and then the segmentation operation is performed to obtain each word block.

[0058] Taking into account the differences in quality of different videos, video denoising and video normalization operations are performed on the video to detect and remove noise in the video. At the same time, the size, frame rate, etc. of the video data are unified to a certain standard to facilitate subsequent processing.

[0059] Step 2: Extract features from the preprocessed video frames and query statements respectively to obtain feature representations of different video clips and text feature representations of query statements; specifically including:

[0060] Step 201: The process of extracting video features includes:

[0061] Step 2011: Perform video preprocessing and adjust the resolution of the video frames to a uniform size to ensure the consistency of the C3D model input. Then standardize the video frame rate to a fixed frame rate, such as 25 frames per second, to match the settings during model training. Finally, normalize the video and scale the pixel values ​​to the [0, 1] range to reduce the impact of illumination and contrast changes on feature extraction.

[0062] Step 212: Segment the preprocessed video into multiple continuous video segments in chronological order. The length of the segments is set according to the requirements of the C3D model to ensure that the model can process and effectively extract features. For video segments that are not long enough, zero padding or loop padding can be used to fill them to the specified length.

[0063] Step 2013: Use the pre-trained C3D model as the backbone network to extract features from each video clip. The C3D model is used to capture both spatial and temporal information in the video, and the video clip is converted into a high-dimensional feature vector through convolutional layers, pooling layers, and fully connected layers.

[0064] Step 202: The process of extracting query text features includes:

[0065] Step 221: Load the pre-trained GloVe vector model and map each word in the query text to its corresponding GloVe word vector. Assume that the sentence text contains N words, and the word vector dimension of each word is d , then the word vector representation of the entire sentence can be expressed as a matrix N×d ,in Is the first i The word vector of each word.

[0066] Step 222: Initialize the BiGRU model, which consists of two GRU layers, and convert the word vector matrix obtained in step 2021 into As the input of the BiGRU model, the BiGRU model processes this sequence and outputs a hidden state at each time step. For the hidden state of each time step output by the model, the hidden state of the last time step is selected as the vector representation of the entire sentence.

[0067] Step 223: Based on the query statement vector feature representation, PCA or other dimensionality reduction techniques are used to reduce the high-dimensional vector h N Mapping to a low-dimensional space to reduce the amount of computation and remove redundant information.

[0068] Step 3: Map the feature representations of the two different modalities, video clips and text, to the same low-dimensional binary hash space through cross-modal hashing technology to obtain hash codes of video clip features and query statement features;

[0069] In step 3, in order to maintain the feature expression capability and reduce the computational complexity, the video clip features and query statement features obtained in step 2 are mapped to a low-dimensional binary space, as follows:

[0070] The video clip features Map to a low-dimensional binary hash space to get the hash code of the video clip ,in is the hash code length:

[0071] ,

[0072] in and is a trainable parameter, Represents a symbolic function.

[0073] Similarly, the query feature Map to a low-dimensional binary hash space to get the query hash code :

[0074] ，

[0075] in and is a trainable parameter.

[0076] In order to make the hash code adapt to different input data, the dynamic hash module is introduced. The dynamic hash optimization can dynamically adjust the hash code according to the changes in input features. The dynamic hash function generation formula is:

[0077] ,

[0078] ,

[0079] in, and , and is the time step Dynamic parameters of

[0080] The dynamic parameters are updated through a small neural network, and the formula for the dynamic hash update network is:

[0081] ,

[0082] ,

[0083] in, is the learning rate, and is the update calculated based on the current input features and the hash coding error; similarly, and according to , Update method to update.

[0084] Step 4: Build a video clip localization teacher model; specifically include:

[0085] Step 401, preprocessor: preprocess the input video data and query text data;

[0086] Step 402, feature extractor: extract features from the preprocessed data to obtain features of two different modalities: video features and query text;

[0087] Step 403, feature encoder: encode features of different modalities and map them to the same dimension;

[0088] Step 404, feature interactor: learning the interaction between the encoded video and the query text, by learning the semantic understanding of the video and the query sentence;

[0089] Step 405, segment predictor: predict the target segment based on the semantic understanding of the video and the query statement, give a matching score to each segment, and rank them according to the final score to obtain the video segment that best matches the query.

[0090] Each module is composed of different models. The training process and the loss function are determined by the model. Continuous training is carried out through standard data sets related to video clip positioning. Finally, a teacher model with high accuracy and strong generalization can be obtained. It has a complex structure and a large number of parameters.

[0091] Step 5: Transfer the knowledge of the teacher model to the lightweight student model through knowledge distillation;

[0092] In order to achieve edge deployment and improve the performance of the model, this embodiment introduces a distillation learning method to transfer the knowledge of a teacher model to a lightweight student model.

[0093] Specifically, the matching score between the video clip and the query output by the teacher model is denoted as The student model also outputs the matching score between the video clip and the query, denoted as , the loss function is constructed based on the output difference between the teacher model and the student model. The distillation loss makes the output of the student model as close as possible to the output of the teacher model, which is expressed as:

[0094] ,

[0095] According to step 2, the loss function includes not only the distillation loss but also the hash coding loss. The hash coding loss ensures that the hash code similarity between the video clip and the query is consistent with the original feature similarity. The total loss function is:

[0096] ,

[0097] in is a weight coefficient used to balance the hash coding loss and the distillation loss. The distillation learning method retains the knowledge of the large teacher model in the lightweight student model, ensuring the accuracy and robustness of the model. And through the optimization of the distillation loss function, the student model can keep the performance of the teacher model as close as possible while maintaining high efficiency.

[0098] Convert the trained lightweight student model into a format suitable for running on edge devices, such as TensorFlowLite and ONNX, and further optimize the model through quantization and pruning.

[0099] For edge devices with appropriate computing power and storage space, such as surveillance cameras and smart cameras, the optimized student model can be integrated into them, so that the camera is not only responsible for capturing video images, but also able to perform tasks such as image processing, target detection, video analysis, and target fragment positioning.

[0100] Embodiment 2

[0101] This embodiment provides a video segment positioning system for real-time tasks, including:

[0102] A data acquisition module, which is used to acquire historical video data and query statements;

[0103] A teacher model training module, which is used to obtain a matching score between a first video segment and a query based on historical video data, a query statement and a trained teacher model; when the teacher model is trained, the hash coding loss of the hash code is dynamically adjusted according to different feature representation changes when calculating the feature representations of different video segments and query statements mapped to a binary hash space;

[0104] A knowledge distillation module, which is used to transfer the knowledge of the teacher model to the lightweight student model through knowledge distillation, and obtain a matching score between the second video clip and the query through the student model;

[0105] A video positioning module is used to calculate the distillation loss based on the matching scores of the first video clip and the query and the matching scores of the second video clip and the query; a total loss function is obtained by combining the hash coding loss and the distillation loss, and the video clip features are positioned with the strategy of minimizing the total loss function as the video clip corresponding to the query statement.

[0106] Embodiment 3

[0107] This embodiment provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the steps in the video segment positioning method for real-time tasks as described above are implemented.

[0108] Embodiment 4

[0109] This embodiment provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the steps in the video segment positioning method for real-time tasks as described above are implemented.

[0110] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A video segment localization method for real-time tasks, characterized in that: The steps include: Get historical video data and query statements; A matching score between the first video segment and the query is obtained based on historical video data, the query statement and the trained teacher model; when the teacher model is trained, the feature representations of different video segments and the query statement are mapped to a binary hash space, and the hash coding loss of the hash coding is dynamically adjusted according to different feature representation changes; The knowledge of the teacher model is transferred to the lightweight student model through knowledge distillation, and the matching score between the second video clip and the query is obtained through the student model; A distillation loss is calculated based on a matching score between the first video clip and the query and a matching score between the second video clip and the query; The total loss function is obtained by combining the hash coding loss and the distillation loss. The video clip features located by minimizing the total loss function are used as the video clip corresponding to the query statement. When mapping the feature representations of different video clips and query statements to the binary hash space, the features of different video clips and query statements are mapped to the binary hash space according to the characteristics of the features of the different video clips and query statements to obtain hash codes of the video clips and hash codes of the query statements; A dynamic hash function is introduced to dynamically adjust the hash function according to the error between the current feature and the hash code to make the hash code adapt to the features of different modes. The hash function parameters corresponding to the minimum error between the current feature and the hash code are used as the optimal hash function parameters to obtain the optimal hash function.

2. The video segment positioning method for real-time tasks according to claim 1, characterized in that: The feature representation extraction process of different video clips includes: Split the video into multiple continuous video segments in chronological order, and the length of the segments is set according to the requirements of the C3D model; The pre-trained C3D model is used as the backbone network to extract features from each video clip. The C3D model is used to simultaneously capture the spatial and temporal information in the video. The video clips are converted into high-dimensional feature vectors through convolutional layers, pooling layers, and fully connected layers, and the features of each video clip are obtained in the output layer.

3. The video segment positioning method for real-time tasks according to claim 1, characterized in that: The process of extracting the feature representation of the query statement includes: Load the pre-trained GloVe vector model and map each word in the query text to its corresponding GloVe word vector; The obtained Glove word vector matrix is ​​used as the input of the BiGRU model, and a hidden state is output at each time step. For the hidden state of each time step output by the BiGRU model, the hidden state of the last time step is selected as the vector representation of the entire sentence.

4. The video segment positioning method for real-time tasks according to claim 1, characterized in that: When the teacher model is trained, the interaction between the two modalities of video and query statement is learned, and the matching score of the query statement corresponding to each video clip is predicted based on the learned interaction results. The video clip with the highest matching score is taken as the video clip with the highest query matching degree.

5. The video segment positioning method for real-time tasks as claimed in claim 1, characterized in that: The expression of the total loss function is: ， ， in, is the total loss function, is the weight coefficient, is the hash encoding loss, is the distillation loss, Output the matching score between the video clip and the query to the teacher model, Outputs the matching score between the video clip and the query for the student model.

6. The video segment positioning method for real-time tasks according to claim 1, characterized in that: After obtaining the historical video data and the query statements, the video data and the query statements are preprocessed, and the video data is preprocessed, including sampling the video frames, performing video denoising and video normalization operations on the video; Remove irrelevant auxiliary words and modal particles in the query sentence, and then perform segmentation operation to obtain each word block.

7. A video segment localization system for real-time tasks, characterized in that: The video segment positioning method for real-time tasks according to any one of claims 1 to 6 comprises: A data acquisition module, which is used to acquire historical video data and query statements; A teacher model training module, which is used to obtain a matching score between a first video segment and a query based on historical video data, a query statement and a trained teacher model; when the teacher model is trained, the hash coding loss of the hash code is dynamically adjusted according to different feature representation changes when calculating the feature representations of different video segments and query statements mapped to a binary hash space; A knowledge distillation module, which is used to transfer the knowledge of the teacher model to the lightweight student model through knowledge distillation, and obtain a matching score between the second video clip and the query through the student model; A video positioning module is used to calculate the distillation loss based on the matching scores of the first video clip and the query and the matching scores of the second video clip and the query; a total loss function is obtained by combining the hash coding loss and the distillation loss, and the video clip features are positioned with the strategy of minimizing the total loss function as the video clip corresponding to the query statement.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps in the real-time task-oriented video segment positioning method as described in any one of claims 1 to 6 are implemented.

9. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps in the real-time task-oriented video segment positioning method according to any one of claims 1 to 6 are implemented.

Images