Video frame classification model training method and video clip retrieval method and device

Abstract

The embodiment of the invention provides a training method of a video frame classification model and a retrieval method and device of video clips, and relates to the technical field of video processing, and the training method of the video frame classification model comprises the following steps: obtaining a sample video and a sample text; inputting the sample text into a text feature extraction network in a video frame classification model to obtain sample text features; inputting each to-be-processed video frame in the sample video into the visual feature extraction network to obtain a sample visual feature; fusing the sample text features and the sample visual features based on an attention mechanism by using a fusion network to obtain sample fusion features; inputting the sample fusion features into a classification network to obtain a first prediction classification result; and performing model parameter adjustment on the video frame classification model according to the difference between the first prediction classification result and the real classification result until model convergence is achieved, and obtaining a trained video frame classification model. According to the scheme, the video clip meeting the description of the user can be retrieved from the video.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of video processing, in particular to a video frame classification model training method and a video segment retrieval method and device. BACKGROUND

[0002] At present, users can install cameras in some scenes to collect videos in the scenes, and sometimes the users also need to find out the video segments of interest from the videos, for example, from the videos recorded by a home camera, a video segment of family members coming home is queried; from the videos recorded by a camera installed on the roadside, a video segment of a specific vehicle passing by is found.

[0003] However, the videos collected by the cameras are usually long and contain a large number of video frames, and therefore, how to retrieve the video segments meeting the user description from the videos is a problem to be solved. SUMMARY

[0004] The embodiments of the present application aim to provide a video frame classification model training method and a video segment retrieval method and device to retrieve the video segments meeting the user description from the videos. The specific technical solutions are as follows:

[0005] The embodiments of the present application first provide a video frame classification model training method, and the method comprises:

[0006] obtaining a sample video and a sample text;

[0007] inputting the sample text into a text feature extraction network in a video frame classification model to be trained to obtain a sample text feature, and inputting each video frame to be processed in the sample video into a visual feature extraction network in the video frame classification model to obtain a sample visual feature of each video frame to be processed; wherein the video frame classification model further comprises a fusion network and a classification network;

[0008] using the fusion network to fuse the sample text feature and each sample visual feature based on an attention mechanism to obtain a sample fusion feature;

[0009] inputting the sample fusion feature into the classification network to obtain a first predicted classification result of each video frame to be processed; wherein the classification result of one video frame to be processed indicates that the video frame to be processed is one of sample categories; the sample categories comprise a start frame of a video segment meeting the sample text, remaining frames in the video segment meeting the sample text except the start frame, and external frames not meeting the sample text;

[0010] According to a difference between the first prediction classification result of each to-be-processed video frame and a true classification result of the to-be-processed video frame, the video frame classification model is model-tuned until model convergence is reached, and a trained video frame classification model is obtained.

[0011] In an embodiment, the video frame classification model further comprises an entity recognition layer and a target detection network.

[0012] Before the to-be-processed video frames in the sample video are input into the visual feature extraction network in the video frame classification model to obtain sample visual features of each to-be-processed video frame, the method further comprises:

[0013] The sample text is input into the entity recognition layer to obtain sample entities contained in the sample text.

[0014] The target detection network is used to perform target detection on each video frame in the sample video to obtain a prediction detection result of whether each video frame in the sample video contains the sample entities, and a video frame in which the prediction detection result indicates that the sample entities are contained is taken as a to-be-processed video frame.

[0015] In an embodiment, the remaining frames in the video segment that meet the sample text, except for the start frame, comprise an end frame and internal frames of the video segment that meet the sample text; and the sample category further comprises a single frame that meets the sample text.

[0016] In an embodiment, the classification network comprises a classification layer and a post-processing layer.

[0017] The sample fusion features are input into the classification network to obtain a first prediction classification result of each to-be-processed video frame, comprising:

[0018] The sample fusion features are input into the classification layer to obtain a second prediction classification result of each to-be-processed video frame.

[0019] The post-processing layer is used to correct the second prediction classification result of each to-be-processed video frame to obtain the first prediction classification result of each to-be-processed video frame.

[0020] In an embodiment, the post-processing layer is used to correct the second prediction classification result of each to-be-processed video frame to obtain the first prediction classification result of each to-be-processed video frame, comprising:

[0021] The post-processing layer is used to correct the second prediction classification result of each to-be-processed video frame based on a conditional random field algorithm to obtain the first prediction classification result of each to-be-processed video frame.

[0022] Or,

[0023] The second prediction classification result of each to-be-processed video frame is corrected based on a preset rule by using the post-processing layer to obtain a first prediction classification result of each to-be-processed video frame, wherein the preset rule includes:

[0024] The classification result indicates that the classification results of the two frames before and after the single frame of the video frame both indicate external frames;

[0025] The classification result indicates that the classification result of the frame after the start frame of the video frame indicates an intermediate frame or an end frame;

[0026] The classification result indicates that the classification result of the frame after the intermediate frame of the video frame indicates an intermediate frame or an end frame;

[0027] The classification result indicates that the classification result of the frame after the end frame of the video frame indicates an external frame;

[0028] The classification result indicates that the classification result of the frame after the external frame of the video frame does not indicate an end frame.

[0029] In an embodiment, the sample fusion feature is obtained by fusing the sample text feature and each sample visual feature based on an attention mechanism by using the fusion network, including:

[0030] The sample text feature is transformed by using a first matrix in the fusion network to obtain a first to-be-processed feature, and the sample text feature is transformed by using a second matrix in the fusion network to obtain a second to-be-processed feature;

[0031] The obtained first to-be-processed feature is used as a key matrix, the obtained second to-be-processed feature is used as a value matrix, and each sample visual feature is used as a query matrix, and the first to-be-processed feature, the second to-be-processed feature, and each sample visual feature are fused based on an attention mechanism to obtain a sample fusion feature.

[0032] The application also provides a video segment retrieval method, including:

[0033] Obtaining to-be-retrieved text and to-be-retrieved video;

[0034] The to-be-retrieved text is input into a text feature extraction network in a pre-trained video frame classification model to obtain to-be-used text features, and each to-be-classified video frame in the to-be-retrieved video is input into a visual feature extraction network in the video frame classification model to obtain to-be-used visual features of each to-be-classified video frame, wherein the video frame classification model further includes a fusion network and a classification network, and the video frame classification model is obtained by training based on the training method of any one of the video frame classification models described above;

[0035] The fusion network is used to fuse the text features and the extracted visual features of each to-be-utilized video frame based on an attention mechanism to obtain to-be-utilized fusion features.

[0036] The to-be-utilized fusion features are input into the classification network to obtain a first classification result of each to-be-classified video frame; the classification result of one to-be-classified video frame indicates that the to-be-classified video frame is one of preset categories; the preset categories include a start frame of a video segment that meets the to-be-retrieved text, remaining frames of the video segment that meet the to-be-retrieved text except the start frame, and external frames that do not meet the to-be-retrieved text.

[0037] According to the first classification result of each to-be-classified video frame, a video segment that meets the to-be-retrieved text in the to-be-retrieved video is determined.

[0038] In one embodiment, the video frame classification model further includes an entity recognition layer and a target detection network.

[0039] Before inputting each to-be-classified video frame in the to-be-retrieved video into the visual feature extraction network in the video frame classification model to obtain to-be-utilized visual features of each to-be-classified video frame, the method further includes:

[0040] The to-be-retrieved text is input into the entity recognition layer to obtain a to-be-retrieved entity contained in the to-be-retrieved text.

[0041] Each video frame in the to-be-retrieved video is detected by the target detection network to obtain a detection result of whether each video frame in the to-be-retrieved video contains the to-be-retrieved entity, and a video frame containing the to-be-retrieved entity in the detection result is taken as a to-be-classified video frame.

[0042] In one embodiment, the remaining frames of the video segment that meet the to-be-retrieved text except the start frame include an end frame and internal frames of the video segment that meet the to-be-retrieved text; and the preset categories further include a single frame that meets the to-be-retrieved text.

[0043] In one embodiment, the determination of the video segment that meets the to-be-retrieved text in the to-be-retrieved video according to the first classification result of each to-be-classified video frame includes:

[0044] For each to-be-classified video frame in the to-be-retrieved video whose first classification result is the start frame, a to-be-classified video frame located after the to-be-classified video frame and closest to the to-be-classified video frame is determined, and the first classification result of the to-be-classified video frame is the end frame.

[0045] determine a video clip composed of the video frame whose first classification result is the start frame, the video frame whose first classification result is the end frame, and other video frames located between the video frame whose first classification result is the start frame and the video frame whose first classification result is the end frame as the video clip conforming to the text to be searched.

[0046] In an embodiment, the classification network comprises a classification layer and a post-processing layer.

[0047] inputting the fusion features into the classification network to obtain a first classification result of each video frame to be classified, comprising:

[0048] inputting the fusion features into the classification layer to obtain a second classification result of each video frame to be classified;

[0049] correcting the second classification result of each video frame to be classified by using the post-processing layer to obtain the first classification result of each video frame to be classified.

[0050] In an embodiment, correcting the second classification result of each video frame to be classified by using the post-processing layer to obtain the first classification result of each video frame to be classified, comprises:

[0051] correcting the second classification result of each video frame to be classified by using the post-processing layer based on a conditional random field algorithm to obtain the first classification result of each video frame to be classified;

[0052] or,

[0053] correcting the second classification result of each video frame to be classified by using the post-processing layer based on a preset rule to obtain the first classification result of each video frame to be classified; wherein the preset rule comprises:

[0054] the classification result indicates that the classification results of the frames before and after the single frame both indicate external frames;

[0055] the classification result indicates that the classification result of the frame after the start frame indicates an intermediate frame or an end frame;

[0056] the classification result indicates that the classification result of the frame after the intermediate frame indicates an intermediate frame or an end frame;

[0057] the classification result indicates that the classification result of the frame after the end frame indicates an external frame;

[0058] the classification result indicates that the classification result of the frame after the external frame does not indicate an end frame.

[0059] In an embodiment, the fusion network is utilized to fuse the text features and the extracted visual features based on an attention mechanism to obtain fusion features.

[0060] The first matrix in the fusion network is utilized to transform the text features to obtain first fusion features, and the second matrix in the fusion network is utilized to transform the text features to obtain second fusion features.

[0061] The first fusion features are taken as a key matrix, the second fusion features are taken as a value matrix, and the extracted visual features are taken as a query matrix, and the first fusion features, the second fusion features, and the visual features are fused based on an attention mechanism to obtain fusion features.

[0062] The embodiment of the present application further provides a training device of a video frame classification model, and the device comprises:

[0063] A sample acquisition module is configured to acquire a sample video and a sample text.

[0064] A sample feature extraction module is configured to input the sample text into a text feature extraction network in a to-be-trained video frame classification model to obtain sample text features, and input each to-be-processed video frame in the sample video into a visual feature extraction network in the video frame classification model to obtain sample visual features of each to-be-processed video frame, wherein the video frame classification model further comprises a fusion network and a classification network.

[0065] A sample feature fusion module is configured to utilize the fusion network to fuse the sample text features and the sample visual features based on an attention mechanism to obtain sample fusion features.

[0066] A classification prediction module is configured to input the sample fusion features into the classification network to obtain a first predicted classification result of each to-be-processed video frame, wherein a classification result of a to-be-processed video frame indicates that the to-be-processed video frame is one of sample categories, and the sample categories comprise a start frame of a video segment conforming to the sample text, remaining frames of the video segment conforming to the sample text except the start frame, and external frames not conforming to the sample text.

[0067] A model parameter adjustment module is configured to perform model parameter adjustment on the video frame classification model according to a difference between the first predicted classification result of each to-be-processed video frame and an actual classification result of the to-be-processed video frame until model convergence is achieved to obtain a trained video frame classification model.

[0068] In an embodiment, the video frame classification model further comprises an entity recognition layer and a target detection network; and the apparatus further comprises:

[0069] a sample entity recognition module, configured to input the sample text into the entity recognition layer to obtain sample entities contained in the sample text, before the sample feature extraction module inputs each to-be-processed video frame in the sample video into the visual feature extraction network in the video frame classification model to obtain sample visual features of each to-be-processed video frame.

[0070] a target prediction module, configured to perform target detection on each video frame in the sample video by using the target detection network to obtain a predicted detection result of whether each video frame in the sample video contains the sample entities, and take a video frame, in which the predicted detection result indicates that the sample entities are contained, as a to-be-processed video frame.

[0071] In an embodiment, the remaining frames in the video clip in accordance with the sample text, except for the start frame, comprise an end frame and an internal frame in the video clip in accordance with the sample text; and the sample category further comprises a single frame in accordance with the sample text.

[0072] In an embodiment, the classification network comprises a classification layer and a post-processing layer.

[0073] The classification prediction module comprises:

[0074] a classification prediction submodule, configured to input the sample fusion features into the classification layer to obtain a second predicted classification result of each to-be-processed video frame.

[0075] a prediction correction submodule, configured to correct the second predicted classification result of each to-be-processed video frame by using the post-processing layer to obtain a first predicted classification result of each to-be-processed video frame.

[0076] In an embodiment, the prediction correction submodule is specifically configured to:

[0077] correct the second predicted classification result of each to-be-processed video frame based on a conditional random field algorithm by using the post-processing layer to obtain the first predicted classification result of each to-be-processed video frame, or correct the second predicted classification result of each to-be-processed video frame based on a preset rule by using the post-processing layer to obtain the first predicted classification result of each to-be-processed video frame; wherein the preset rule comprises:

[0078] the classification result of the video frame representing a single frame indicates that the classification results of the front and rear frames of the video frame representing a single frame both indicate external frames;

[0079] the classification result of the video frame representing a start frame indicates that the classification result of the next frame of the video frame representing a start frame indicates an internal frame or an end frame;

[0080] The classification result of the video frame after the intermediate frame indicates the classification result of the intermediate frame or the end frame;

[0081] The classification result of the video frame after the end frame represents an external frame;

[0082] The classification result of the next frame of the video frame whose classification result indicates an external frame does not indicate an end frame.

[0083] In one embodiment, the sample fusion module includes:

[0084] A sample feature transformation submodule, configured to transform the sample text feature using the first matrix in the fusion network to obtain a first feature to be processed; and to transform the sample text feature using the second matrix in the fusion network to obtain a second feature to be processed;

[0085] The sample feature fusion submodule is used to use the obtained first feature to be processed as the key matrix, the obtained second feature to be processed as the value matrix, and the extracted visual features of each sample as the query matrix, and to fuse the obtained first feature to be processed, the second feature to be processed and the visual features of each sample based on the attention mechanism to obtain the sample fusion feature.

[0086] The present application also provides a video clip retrieval device, the device comprising:

[0087] An information acquisition module is used to acquire the text and video to be retrieved;

[0088] The module for extracting features to be utilized is configured to input the text to be retrieved into a text feature extraction network in a pre-trained video frame classification model to obtain text features to be utilized; and input each video frame to be classified in the video to be retrieved into a visual feature extraction network in the video frame classification model to obtain visual features to be utilized for each video frame to be classified; wherein the video frame classification model further includes: a fusion network and a classification network; and the video frame classification model is trained based on the training method of the above-mentioned video frame classification model;

[0089] A feature fusion module to be used, configured to use the fusion network to fuse the text features to be used and the extracted visual features to be used based on an attention mechanism to obtain fused features to be used;

[0090] The classification module is configured to input the to-be-utilized fusion features into the classification network to obtain a first classification result of each to-be-classified video frame; wherein the classification result of each to-be-classified video frame indicates that the to-be-classified video frame is one of preset categories; the preset categories include a start frame of a video segment that matches the to-be-retrieved text, remaining frames of the video segment that match the to-be-retrieved text except the start frame, and external frames that do not match the to-be-retrieved text.

[0091] The video segment determination module is configured to determine, according to the first classification result of each to-be-classified video frame, a video segment that matches the to-be-retrieved text in the to-be-retrieved video.

[0092] In an embodiment, the video frame classification model further includes an entity recognition layer and a target detection network; and the device further includes:

[0093] The to-be-retrieved entity recognition module is configured to, before the to-be-utilized feature extraction module inputs each to-be-classified video frame in the to-be-retrieved video into the visual feature extraction network in the video frame classification model to obtain the to-be-utilized visual feature of each to-be-classified video frame, input the to-be-retrieved text into the entity recognition layer to obtain a to-be-retrieved entity contained in the to-be-retrieved text.

[0094] The target detection module is configured to perform target detection on each video frame in the to-be-retrieved video by using the target detection network to obtain a detection result of whether each video frame in the to-be-retrieved video contains the to-be-retrieved entity, and to take a video frame containing the to-be-retrieved entity as a to-be-classified video frame.

[0095] In an embodiment, the remaining frames of the video segment that match the to-be-retrieved text except the start frame include an end frame of the video segment that matches the to-be-retrieved text and internal frames; and the preset categories further include a single frame that matches the to-be-retrieved text.

[0096] In an embodiment, the video segment determination module is specifically configured to:

[0097] For each to-be-classified video frame whose first classification result is the start frame in the to-be-retrieved video, determine a to-be-classified video frame located after the to-be-classified video frame and closest to a to-be-classified video frame whose first classification result is the end frame; and determine a video segment composed of the to-be-classified video frame whose first classification result is the start frame, the to-be-classified video frame whose first classification result is the end frame, and other to-be-classified video frames located between the to-be-classified video frame whose first classification result is the start frame and the to-be-classified video frame whose first classification result is the end frame, as the video segment that matches the to-be-retrieved text.

[0098] In an embodiment, the classification network includes a classification layer and a post-processing layer.

[0099] The classification module comprises:

[0100] The classification sub-module is configured to input the to-be-utilized fusion feature into the classification layer to obtain a second classification result of each to-be-classified video frame.

[0101] The result correction sub-module is configured to correct the second classification result of each to-be-classified video frame by using the post-processing layer to obtain a first classification result of each to-be-classified video frame.

[0102] In one embodiment, the result correction sub-module is specifically configured to:

[0103] The post-processing layer is used to correct the second classification result of each to-be-classified video frame based on a conditional random field algorithm to obtain the first classification result of each to-be-classified video frame, or the post-processing layer is used to correct the second classification result of each to-be-classified video frame based on a preset rule to obtain the first classification result of each to-be-classified video frame; wherein the preset rule comprises:

[0104] The classification result indicates that the classification results of the frames before and after the single-frame video frame both represent external frames;

[0105] The classification result indicates that the classification result of the frame after the start frame represents an intermediate frame or an end frame;

[0106] The classification result indicates that the classification result of the frame after the intermediate frame represents an intermediate frame or an end frame;

[0107] The classification result indicates that the classification result of the frame after the end frame represents an external frame;

[0108] The classification result indicates that the classification result of the frame after the external frame does not represent an end frame.

[0109] In one embodiment, the to-be-utilized feature fusion module comprises:

[0110] The to-be-utilized feature transformation sub-module is configured to transform the to-be-utilized text feature by using a first matrix in the fusion network to obtain a first to-be-utilized feature, and transform the to-be-utilized text feature by using a second matrix in the fusion network to obtain a second to-be-utilized feature.

[0111] The to-be-utilized feature fusion sub-module is configured to take the obtained first to-be-utilized feature as a key matrix, take the obtained second to-be-utilized feature as a value matrix, take the extracted to-be-utilized visual features as query matrices, fuse the obtained first to-be-utilized feature, second to-be-utilized feature and to-be-utilized visual features based on an attention mechanism to obtain the to-be-utilized fusion feature.

[0112] The embodiment of the present application further provides an electronic device, comprising:

[0113] a memory for storing a computer program;

[0114] a processor for executing the program stored on the memory to implement the training method of the video frame classification model or the retrieval method of the video segment.

[0115] The embodiment of the present application further provides a computer readable storage medium, wherein the computer readable storage medium stores a computer program, and the computer program is executed by a processor to implement the training method of the video frame classification model or the retrieval method of the video segment.

[0116] The embodiment of the present application further provides a computer program product containing instructions, which, when executed on a computer, cause the computer to perform the training method of the video frame classification model or the retrieval method of the video segment.

[0117] The embodiment of the present application has the following beneficial effects:

[0118] The training method of the video frame classification model provided by the embodiment of the present application extracts the text features of the sample text and the visual features of the to-be-processed video frame in the sample video respectively; fuses the sample text features and the extracted sample visual features based on an attention mechanism to obtain sample fusion features; determines the classification result of each to-be-processed video frame based on the sample fusion features, and the classification result can represent whether the to-be-processed video frame meets the sample text; then, according to the difference between the first predicted classification result of each to-be-processed video frame and the real classification result of the to-be-processed video frame, the video frame classification model is adjusted until the model converges. In the training process, the video frame classification model can learn the correlation between the text features and the visual features, and in the subsequent process, the user can input a text to describe a video segment of interest, and then the trained video frame classification model can be used to determine the video frame meeting the text from the video, and further determine the video segment meeting the user description.

[0119] Of course, implementing any product or method of the present application does not necessarily require all the advantages described above. BRIEF DESCRIPTION OF DRAWINGS

[0120] In order to more clearly illustrate the technical solutions of the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiment or prior art description. Obviously, the drawings in the following description are only some embodiments of the present application, and other embodiments can also be obtained by those skilled in the art based on these drawings.

[0121] Figure 1 A flowchart of one of the training methods of the video frame classification model provided in the embodiments of the present application;

[0122] Figure 2 A flowchart of another of the training methods of the video frame classification model provided in the embodiments of the present application;

[0123] Figure 3 A flowchart of a first of the retrieval methods of the video clip provided in the embodiments of the present application;

[0124] Figure 4 A flowchart of a second of the retrieval methods of the video clip provided in the embodiments of the present application;

[0125] Figure 5 A flowchart of a third of the retrieval methods of the video clip provided in the embodiments of the present application;

[0126] Figure 6 A structural diagram of the training device of the video frame classification model provided in the embodiments of the present application;

[0127] Figure 7 A structural diagram of the retrieval device of the video clip provided in the embodiments of the present application;

[0128] Figure 8 A structural diagram of the electronic device provided in the embodiments of the present application. DETAILED DESCRIPTION

[0129] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, but not all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art based on the present application are within the scope of protection of the present application.

[0130] In the current life, the videos stored by users often contain a large number of video frames. If there is a very effective retrieval method to help users automatically retrieve the video clips meeting the user description from a large video database, it is very valuable.

[0131] In order to retrieve the video clips meeting the user description from the videos, the embodiments of the present application provide a training method of a video frame classification model, a retrieval method and device of a video clip. The two methods can be applied to electronic devices with data processing capability.

[0132] The training method of the video frame classification model includes:

[0133] obtaining a sample video and a sample text;

[0134] inputting the sample text into a text feature extraction network in the video frame classification model to be trained to obtain sample text features, and inputting each video frame in the sample video into a visual feature extraction network in the video frame classification model to obtain sample visual features of each video frame to be processed; wherein the video frame classification model further comprises a fusion network and a classification network;

[0135] The sample text features and the sample visual features are fused based on an attention mechanism by using the fusion network to obtain sample fusion features.

[0136] The sample fusion features are input into the classification network to obtain a first predicted classification result of each video frame to be processed; wherein the classification result of one video frame to be processed indicates that the video frame to be processed is one of the sample categories; the sample categories include a start frame of a video segment conforming to the sample text, remaining frames in the video segment conforming to the sample text except the start frame, and external frames not conforming to the sample text.

[0137] The video frame classification model is model-tuned according to the difference between the first predicted classification result of each video frame to be processed and the true classification result of the video frame to be processed until model convergence is reached, thereby obtaining a trained video frame classification model.

[0138] In this embodiment, the video frame classification model can learn the correlation between the text features and the visual features, and a user can subsequently describe a video segment of interest by inputting text, and the trained video frame classification model can be used to determine video frames conforming to the text from the video, thereby determining the video segment described by the user.

[0139] The training method of the video frame classification model provided by the embodiments of the present application will be described below with reference to the accompanying drawings, as shown in FIG. 1, which comprises the following steps: Figure 1

[0140] S101, obtaining a sample video and a sample text;

[0141] The sample video can be any video containing multiple video frames. For example, it can be a video recorded by a camera in a home scene, a video recorded by a camera in a security scene, a video recorded by a camera installed on a roadside, etc. The sample text can be a text describing any video segment in the sample video, for example, there is a video segment in the sample video recording a person wearing a yellow clothes running in the living room while holding a toy gun, and the sample text can be "a person wearing a yellow clothes running in the living room while holding a toy gun".

[0142] ​The sample video can be determined according to an application scenario of the video segment retrieval method. For example, if the video segment retrieval method is used to find a specific video segment from a video recorded by a camera installed on a roadside, a plurality of videos of road scenes can be obtained as sample videos.

[0143] To improve processing efficiency, the sample video can be obtained by sampling from the original video, for example, by sampling one frame every 3 frames from the original video.

[0144] S102, input the sample text into a text feature extraction network in the video frame classification model to be trained to obtain sample text features; and input each video frame in the sample video into a visual feature extraction network in the video frame classification model to obtain sample visual features of each video frame to be processed;

[0145] The video frame classification model further includes a fusion network and a classification network.

[0146] In actual implementation, the input text can be subjected to word embedding processing to obtain a word vector corresponding to the text, such as encoding by using a pre-trained BERT (Bidirectional Encoder Representations from Transformers, a pre-trained language model based on a Transformer architecture) model, and then encoding the word vector to obtain the text features. For example, the word vector can be encoded by using a TextCNN (text convolutional neural network), a transformer, or the like. In this way, the BERT model will also be fine-tuned under supervision during training of the video frame classification model.

[0147] The text features can include features of each word in the text, and the features of the words have the same dimension. For example, the sample text has a length of S, that is, contains S words, and the dimension of the feature of each word is D, and the dimension of the sample text features can be denoted as (S, D).

[0148] Each video frame to be processed can be all video frames in the sample video, or can be video frames obtained by screening the video frames in the sample video. The specific screening process will be described below.

[0149] The visual feature extraction network can be any model for encoding a video, such as a ViViT (Video Vision Transformer) model. In this case, each video frame in the sample video to be processed can be input into the visual feature extraction network at the same time to obtain the sample visual feature of each video frame to be processed. Alternatively, the visual feature extraction network can be a model for encoding a video frame. In this case, each video frame to be processed can be input into the visual feature extraction network respectively to obtain the sample visual feature of each video frame to be processed.

[0150] The sample visual feature of each video frame to be processed is multi-dimensional, and the dimensions of the sample visual features of the video frames to be processed can be equal. For example, there are N video frames to be processed, and the dimension of the sample visual feature of each video frame to be processed is D. The dimension of the sample visual feature of each video frame to be processed can be denoted as (N, D). The feature of each word in the sample text can also be multi-dimensional. In order to better align the visual features with the text features, the dimension of the visual feature of each video frame to be processed can be consistent with the dimension of the feature of each word in the sample text, for example, both are 768 dimensions.

[0151] In S103, the sample text feature and the sample visual features are fused based on the attention mechanism by using the fusion network to obtain a sample fusion feature.

[0152] The attention mechanism in this embodiment can be a cross-attention mechanism. In one implementation, the following process can be included:

[0153] In step A1, the sample text feature is transformed by using a first matrix in the fusion network to obtain a first processed feature, and the sample text feature is transformed by using a second matrix in the fusion network to obtain a second processed feature.

[0154] The first matrix and the second matrix in the fusion network are model parameters, and can be adjusted in the training process.

[0155] In step A2, the first processed feature obtained is used as a key matrix, the second processed feature obtained is used as a value matrix, and the sample visual features extracted are used as a query matrix. The first processed feature, the second processed feature, and the sample visual features are fused based on the attention mechanism to obtain a sample fusion feature.

[0156] The formula of the attention mechanism is as follows:

[0157]

[0158] wherein Attention represents an attention mechanism; Q, K, and V represent a query matrix, a key matrix, and a value matrix respectively; and Softmax represents a normalized exponential function. a dimension of the visual feature of each input video frame.

[0159] If the dimension of the text feature is (S, D) and the dimension of the visual feature is (N, D), the dimension of the fusion feature is also (N, D). In this way, there is a part corresponding to each of the N video frames in the fusion feature, so that the classification result of each video frame can be determined according to the fusion feature.

[0160] The purpose of steps S102-S103 is to cross-modal fusion of the text feature and the visual feature, so the fusion network has a cross-modal attention structure, and the text feature and the visual feature can be aligned in the fusion process. In the process of training the video frame classification model, the visual feature extraction network and the text feature extraction network can perform contrastive learning, that is, when a video frame conforms to the description of the sample text, the similarity between the visual feature extracted by the visual feature extraction network for the video frame and the text feature extracted by the text feature extraction network for the sample text will be higher and higher, and in the case where a video frame does not conform to the description of the sample text, the similarity between the extracted visual feature and the text feature will be lower and lower.

[0161] After the above fusion process, the information of the text used to query the required video segment can be embedded in the visual feature of the video frame, so that the video frame can be classified according to the text, and the whole process is not limited by the length of the text.

[0162] S104, inputting the sample fusion feature into a classification network to obtain a first predicted classification result of each to-be-processed video frame;

[0163] wherein the classification result of a to-be-processed video frame indicates that the to-be-processed video frame is one of the sample categories; and the sample categories include: a start frame of a video segment conforming to the sample text, remaining frames in the video segment conforming to the sample text except the start frame, and external frames not conforming to the sample text.

[0164] The classification network can include a fully connected layer, and the classification result of a to-be-processed video frame can specifically include a probability value of the to-be-processed video frame belonging to each of the sample categories; and the category indicated by the classification result is the category with the largest probability value. Alternatively, the classification result of a to-be-processed video frame can also only include an identifier of the category to which the to-be-processed video frame belongs.

[0165] Since the sample categories include the start frame, the remaining frame and the external frame, thus, only the start frame is needed to be obtained from the sample video, and then the continuous remaining frame after and adjacent to the start frame is obtained, so that the video segment conforming to the sample text is obtained, thereby improving the efficiency of determining the video segment conforming to the text. The remaining frame at this time can also be referred to as an internal frame.

[0166] In an implementation manner, the remaining frames except the start frame in the video segment conforming to the sample text can be specifically divided into: the end frame and the internal frame of the video segment conforming to the sample text; and the sample categories can further include: a single frame conforming to the sample text. If the classification result of a to-be-processed video frame indicates a single frame, it indicates that the classification results of the to-be-processed video frames adjacent to the to-be-processed video frame all indicate external frames, that is, the classification results of the previous to-be-processed video frame and the next to-be-processed video frame of the to-be-processed video frame all indicate external frames. In the case that the sample video is obtained by sampling from an original video, the time interval between each two frames in the sample video can be relatively long, and at this time, the single frame condition is prone to occur.

[0167] In this way, only the first end frame after the start frame is needed to be determined from each to-be-processed video frame for each start frame in the sample video, and each video frame located between the start frame and the end frame is determined from each to-be-processed video frame, so that the video segment conforming to the sample text is obtained, thereby further improving the efficiency of determining the video segment conforming to the text. In addition, each single frame in the sample video segment can also be regarded as a video segment conforming to the sample text.

[0168] That is, there can be two cases for the sample categories:

[0169] Case one: the sample categories include three categories of start frame, internal frame and external frame;

[0170] Case two: the sample categories include five categories of start frame, internal frame, end frame, single frame and external frame.

[0171] S105, according to the difference between the first prediction classification result of each to-be-processed video frame and the true classification result of the to-be-processed video frame, model parameter adjustment is performed on the video frame classification model until model convergence is reached, and a trained video frame classification model is obtained.

[0172] The true classification result of each to-be-processed video frame can be set by a technician in advance. For example, for the above case one, it can be first determined whether each to-be-processed video frame conforms to the description of the sample text, if not, the to-be-processed video frame is labeled as an external frame; if yes, it can be further determined whether the to-be-processed video frame is a start frame or an internal frame;

[0173] For the second case, it can also be determined whether each to-be-processed video frame conforms to the description of the sample text. If not, the to-be-processed video frame is labeled as an external frame. If yes, it can be further determined whether the to-be-processed video frame is a start frame, an internal frame, an end frame, or a single frame.

[0174] The parts in the video frame classification model can be optimized end to end to avoid error propagation in different stages, and finally a very good model can be obtained.

[0175] Specifically, the model loss can be calculated according to the difference between the first prediction classification result of each to-be-processed video frame and the true classification result of the to-be-processed video frame, and then the video frame classification model is adjusted according to the obtained model loss, and the steps of inputting the sample text into the text feature extraction network in the to-be-trained video frame classification model to obtain the sample text feature, and inputting each to-be-processed video frame in the sample video into the visual feature extraction network in the video frame classification model to obtain the sample visual feature of each to-be-processed video frame are returned. In this way, after multiple rounds of model adjustment until the model converges, the trained video frame classification model can be obtained.

[0176] In this embodiment, through the above training process, the video frame classification model can learn the correlation between the text feature and the visual feature. Subsequently, the user can input a text to describe a video segment of interest, and the trained video frame classification model can be used to determine the video frame that conforms to the text from the video, and then the video segment that conforms to the user's description can be determined.

[0177] In an embodiment of the present application, the video frame classification model can further include a target detection network.

[0178] As shown in Figure 2 Before the above step of inputting each to-be-processed video frame in the sample video into the visual feature extraction network in the video frame classification model to obtain the sample visual feature of each to-be-processed video frame, the method can further include:

[0179] S201, inputting the sample text into the entity recognition layer to obtain the sample entity contained in the sample text;

[0180] In an implementation manner, the keyword representing the entity can be set in advance, and then the entity recognition layer can retrieve the keyword contained in the sample text from the sample text as the sample entity according to the keyword set in advance.

[0181] In another implementation, the entity recognition layer can perform entity recognition on the input text based on a NER (Named Entity Recognition) algorithm. For example, the input text is: "Help me find the clip of xxx doing what behavior", and the extracted entity is "person". Alternatively, a large language model such as a GPT (Generative Pre-trained Transformer) model can be used for entity recognition. In order to improve processing efficiency, a lightweight algorithm such as Pytorch-NLU, xmnlp, etc. can be used for entity recognition in actual implementation.

[0182] In S202, the target detection network is used to perform target detection on each video frame in the sample video to obtain a predicted detection result of whether each video frame in the sample video contains the sample entity, and a video frame containing the sample entity in the predicted detection result is taken as a to-be-processed video frame.

[0183] The target detection network can be any model with target detection capability, i.e., capable of identifying the target of interest contained in an image from the image. The target detection network can be a YOLO (You Only Look Once) model, an SSD (Single Shot MultiBox Detector) model, etc. In order to improve processing efficiency, a lightweight target detection model such as an SL-YOLO model can be selected.

[0184] In one implementation, the target detection network can identify the target contained in each video frame in the sample video, and then determine whether there is a target consistent with the sample entity in the video frame. If there is, the video frame is determined as a to-be-processed video frame, otherwise, the predicted classification result of the to-be-processed video frame can be directly determined as an external frame.

[0185] For example, the sample entity is a child, and a video frame contains only a cat in the target. It is determined that there is no target consistent with the sample entity in the video frame. If the target contained in the video frame contains a child, it is determined that there is a target consistent with the sample entity in the video frame.

[0186] In this case, the types of targets that can be detected by the target detection network should include the entities that can be recognized by the entity recognition layer. For example, the entities that can be recognized by the entity recognition layer include adults, children, cats, and dogs, and the types of targets that can be detected by the target detection network should also include adults, children, cats, and dogs.

[0187] In another implementation, the target detection network can also directly detect whether each video frame in the sample video contains the sample entity. In this way, compared with detecting the target first and then screening, this manner can improve the processing speed and avoid analyzing some video frames that do not contain the sample entity for a long time.

[0188] The target detection network can be trained end-to-end with the text feature extraction network, the sample text feature, the fusion network, and the classification network, that is, the target detection network is also tuned during the model tuning of the video frame classification model in step S105. At this time, the model tuning of the video frame classification model can be performed according to the difference between the first predicted classification result of each video frame in the sample video and the true classification result of the video frame. The first predicted classification result of the video frame in the sample video other than the to-be-processed video frame can be regarded as indicating an external frame. When labeling each video frame in the sample video, if the video frame does not contain the sample entity, the true classification result of the video frame can be determined to indicate an external frame, and if the video frame contains the sample entity, the true classification result of the video frame can be further determined to indicate which one of the start frame, the intermediate frame, the end frame, or the external frame.

[0189] In this embodiment, by using the target detection network to detect the target in each video frame in the sample video, the predicted detection result of whether each video frame in the sample video contains the sample entity is obtained, and the predicted detection result indicating the video frame containing the sample entity is taken as the to-be-processed video frame, so that the video frames in the sample video can be preliminarily screened, the number of video frames to be processed in the subsequent stage is reduced, and the processing speed is improved.

[0190] In an embodiment of the present application, the classification network can further include a classification layer and a post-processing layer.

[0191] As shown in FIG. 4, in step S104, the sample fusion feature is input into the classification network to obtain the first predicted classification result of each to-be-processed video frame, which includes: Figure 2

[0192] S1041, the sample fusion feature is input into the classification layer to obtain the second predicted classification result of each to-be-processed video frame.

[0193] In this embodiment, the classification layer can be a full connection layer. The second predicted classification result is actually a classification result preliminarily predicted by the video frame classification model. It can be understood that if the classification result of a video frame is a single frame, the classification results of the previous frame and the next frame should both be external frames, and the classification result of a video frame between the start frame and the end frame should be an intermediate frame. Since the second predicted classification result of each to-be-processed video frame is only a preliminary classification result, there can be an irregular classification result. ​

[0194] For example, the following cases are all non-compliant:

[0195] The first case: the second prediction classification result of a to-be-processed video frame represents a single frame, while the second prediction classification result of the previous or next frame does not represent an external frame, such as also representing a single frame.

[0196] The second case: the second prediction classification result of a to-be-processed video frame represents a start frame, while the second prediction classification result of the previous frame does not represent an external frame, such as representing an internal frame.

[0197] The third case: the second prediction classification result of a to-be-processed video frame represents an intermediate frame, while the second prediction classification result of the previous frame does not represent an intermediate frame or a start frame, such as representing an end frame.

[0198] The fourth case: the second prediction classification result of a to-be-processed video frame represents an end frame, while the second prediction classification result of the previous frame does not represent an intermediate frame or a start frame, such as also representing an end frame.

[0199] Therefore, it is necessary to correct the second prediction classification result of each to-be-processed video frame.

[0200] S1042, using the post-processing layer, correcting the second prediction classification result of each to-be-processed video frame to obtain the first prediction classification result of each to-be-processed video frame.

[0201] Specifically, the process of using the post-processing layer to correct the second prediction classification result of each to-be-processed video frame to obtain the first prediction classification result of each to-be-processed video frame can include the following two ways:

[0202] The first way: using the post-processing layer, based on a conditional random field algorithm (CRF), to correct the second prediction classification result of each to-be-processed video frame to obtain the first prediction classification result of each to-be-processed video frame.

[0203] The conditional random field algorithm is a mathematical model. When labeling the true classification results of the video frames in the pre-labeled sample video, the categories represented by the true classification results of the video frames should all be compliant, so that after training the video classification model using the sample video, the conditional random field algorithm can learn the rules that the classification results of the video frames comply with. Finally, the trained conditional random field algorithm can adjust the categories represented by the input second prediction classification results of each to-be-processed video frame to be compliant.

[0204] The second prediction classification result of each to-be-processed video frame is corrected based on a preset rule by using the post-processing layer to obtain the first prediction classification result of each to-be-processed video frame. The preset rule is a rule that should be followed when the classification results of the video frames in a video are logical. For example, the preset rule includes:

[0205] The classification result indicates that the classification results of the two frames before and after the video frame of the single frame all indicate external frames;

[0206] The classification result indicates that the classification result of the frame after the video frame of the start frame indicates an intermediate frame or an end frame;

[0207] The classification result indicates that the classification result of the frame after the video frame of the intermediate frame indicates an intermediate frame or an end frame;

[0208] The classification result indicates that the classification result of the frame after the video frame of the end frame indicates an external frame;

[0209] The classification result indicates that the classification result of the frame after the video frame of the external frame does not indicate an end frame.

[0210] That is, the second prediction classification result of each to-be-processed video frame can be adjusted to be in line with the preset rule. Specifically, each to-be-processed video frame can be detected in time sequence according to the timestamps, and it is determined whether the to-be-processed video frame is in line with the preset rule. If not, the second prediction classification result of the to-be-processed video frame is adjusted to indicate an external frame to obtain the first prediction classification result of the to-be-processed video frame. If yes, the second prediction classification result of the to-be-processed video frame can be directly used as the first prediction classification result.

[0211] In this embodiment, the second prediction classification result of each to-be-processed video frame is obtained first, and then the second prediction classification result of each to-be-processed video frame is corrected to obtain the first prediction classification result of each to-be-processed video frame, which can avoid the occurrence of illogical classification results, thereby further improving the accuracy of the model.

[0212] The embodiments of the present application also provide a video segment retrieval method, as shown in Figure 3 The method comprises the following steps.

[0213] S301, obtaining to-be-retrieved text and to-be-retrieved video;

[0214] The to-be-retrieved text can be input by a user and used to describe the video segment to be retrieved.

[0215] In order to improve the processing efficiency, the to-be-retrieved video can also be obtained by sampling from an original video.

[0216] S302, input the text features of the to-be-searched text into a text feature extraction network in a pre-trained video frame classification model to obtain to-be-utilized text features, and input each to-be-classified video frame in the to-be-searched video into a visual feature extraction network to obtain to-be-utilized visual features of each to-be-classified video frame;

[0217] The video frame classification model further includes a visual feature extraction network, a fusion network, and a classification network, and is obtained through the training method of the video frame classification model.

[0218] S303, the fusion network is used to fuse the to-be-utilized text features and the extracted to-be-utilized visual features based on an attention mechanism to obtain to-be-utilized fusion features.

[0219] Similarly, the fusion network is used to fuse the to-be-utilized text features and the extracted to-be-utilized visual features based on an attention mechanism to obtain to-be-utilized fusion features, including:

[0220] Step B1, the to-be-utilized text features are transformed by using a first matrix in the fusion network to obtain first to-be-utilized features, and the to-be-utilized text features are transformed by using a second matrix in the fusion network to obtain second to-be-utilized features.

[0221] Step B2, the obtained first to-be-utilized features are used as a key matrix, the obtained second to-be-utilized features are used as a value matrix, and the extracted to-be-utilized visual features are used as a query matrix, and the obtained first to-be-utilized features, the second to-be-utilized features, and the to-be-utilized visual features are fused based on an attention mechanism to obtain to-be-utilized fusion features.

[0222] S304, the to-be-utilized fusion features are input into the classification network to obtain a first classification result of each to-be-classified video frame.

[0223] The classification result of a to-be-classified video frame indicates that the to-be-classified video frame is one of preset categories; the preset categories include a start frame of a video segment that meets the to-be-searched text, remaining frames in the video segment that meet the to-be-searched text except the start frame, and external frames that do not meet the to-be-searched text.

[0224] In an implementation manner, the remaining frames in the video segment that meet the to-be-searched text except the start frame include an end frame and internal frames of the video segment that meet the to-be-searched text; and the preset categories further include a single frame that meets the to-be-searched text.

[0225] The embodiment can classify video frames frame by frame and realize frame-level positioning by classifying each to-be-classified video frame.

[0226] That is, the preset categories can have two cases:

[0227] Case 1: The preset categories include: start frame, internal frame and external frame;

[0228] Case 2: The preset categories include: start frame, inner frame, end frame, single frame and outer frame.

[0229] S305 , determining video segments in the video to be retrieved that match the text to be retrieved based on the first classification result of each video frame to be classified.

[0230] In one embodiment, determining a video segment in the video to be retrieved that matches the text to be retrieved based on the first classification result of each video frame to be classified includes:

[0231] Step C1, for each video frame to be classified whose first classification result in the video to be retrieved is a start frame, determining a video frame to be classified whose first classification result is an end frame that is closest to the video frame to be classified;

[0232] Step C2, determining the video frame to be classified whose first classification result is a start frame, the video frame to be classified whose first classification result is an end frame, and the video clip consisting of other video frames to be classified between the video frame to be classified whose first classification result is a start frame and the video frame to be classified whose first classification result is an end frame, as the video clip that meets the text to be retrieved.

[0233] In other words, during the specific execution process, we only need to find the start frame and then the end frame closest to the start frame according to the time stamp sequence of each video frame in the searched video to obtain the video segment that matches the searched text. This can quickly determine the video segment that matches the searched text.

[0234] In addition, the video frame whose first classification result is a single frame can be ignored, or can also be used as a video segment that meets the description of the text to be retrieved, which can be set according to actual conditions.

[0235] After determining the video segment that matches the text to be retrieved, the video segment can be marked on the timeline of the video to be retrieved to remind the user.

[0236] In one implementation, the video frame classification model further includes: an entity recognition network and an object detection network; Figure 4 As shown, before inputting each to-be-classified video frame in the to-be-retrieved video into the visual feature extraction network to obtain the to-be-utilized visual features of each to-be-classified video frame, the method further includes:

[0237] S401, inputting the text to be retrieved into the entity recognition layer to obtain the entities to be retrieved contained in the text to be retrieved;

[0238] S402, target detection network is used to detect targets in each video frame in the video to be searched, to obtain a detection result of whether each video frame in the video to be searched contains the entity to be searched, and to take the video frame containing the entity to be searched as a video frame to be classified.

[0239] In combination with subsequent classification of each video frame to be classified, the video frame classification model can go beyond the single target-based search mode, and can not only detect the video frame containing the entity to be searched, but also further detect the video segment conforming to the entire text. For example, for the text to be searched: "find the person wearing yellow clothes and running in the living room with a toy gun", not only the "running person" can be detected, but also the "person wearing yellow clothes and running in the living room with a toy gun" can be detected.

[0240] In an implementation manner, the classification network comprises a classification layer and a post-processing layer.

[0241] In the above step S304, the fusion feature to be used is input into the classification network to obtain a first classification result of each video frame to be classified, which comprises:

[0242] S3041, the fusion feature to be used is input into the classification layer to obtain a second classification result of each video frame to be classified;

[0243] S3042, the post-processing layer is used to correct the second classification result of each video frame to be classified to obtain the first classification result of each video frame to be classified.

[0244] Similarly, the process of correcting the second classification result of each video frame to be classified by using the post-processing layer to obtain the first classification result of each video frame to be classified can comprise the following two modes:

[0245] Mode one: the post-processing layer is used to correct the second classification result of each video frame to be classified based on a conditional random field algorithm to obtain the first classification result of each video frame to be classified;

[0246] Since the conditional random field algorithm can learn the rules that the classification results of the video frames conform to in the training process. Finally, the trained conditional random field algorithm can adjust the categories represented by the second classification results of the video frames to be classified to be in compliance.

[0247] Mode two: the post-processing layer is used to correct the second classification result of each video frame to be classified based on a preset rule to obtain the first classification result of each video frame to be classified; wherein the preset rule is a rule that the video frames in a video should conform to, and the preset rule can comprise:

[0248] The classification result indicates that the classification result of the frame after the video frame of the start frame indicates the intermediate frame or the end frame.

[0249] The classification result indicates that the classification result of the frame after the video frame of the start frame indicates the intermediate frame or the end frame.

[0250] The classification result indicates that the classification result of the frame after the video frame of the intermediate frame indicates the intermediate frame or the end frame.

[0251] The classification result indicates that the classification result of the frame after the video frame of the end frame indicates the external frame.

[0252] The classification result indicates that the classification result of the frame after the video frame of the external frame does not indicate the end frame.

[0253] That is, the second classification result of each to-be-classified video frame is adjusted to meet the preset rule. Specifically, each to-be-classified video frame can be detected in chronological order of the timestamp, and it is determined whether it meets the preset rule. If it does not meet the preset rule, the second classification result of the to-be-classified video frame is adjusted to indicate an external frame, and the first classification result of the to-be-classified video frame is obtained; if it does meet the preset rule, the second classification result of the to-be-classified video frame can be directly used as the first classification result.

[0254] Of course, in specific implementation, the post-processing layer can also directly find the to-be-classified video frame that meets the preset rule from the second classification result of each to-be-classified video frame, so as to obtain the to-be-retrieved text video segment that meets the preset rule.

[0255] In this embodiment, because the video frame classification model can learn the correlation between the text features and the visual features through the training process described above, when it is necessary to retrieve the to-be-retrieved video, the to-be-retrieved text describing the video segment of interest can be input, and the trained video frame classification model can be used to determine the video frame that meets the to-be-retrieved text from the video, and then the video segment that meets the description of the user can also be determined.

[0256] The overall framework of the embodiment of the present application can be as shown in Figure 5 Firstly, the user can input a question (S501), that is, a to-be-retrieved text, such as: “Help me find the video segment of the girl in a yellow dress going home”. The question input by the user can be respectively input into a lightweight semantic understanding module 11 (that is, the entity recognition layer described above) and a text encoder 12 (that is, the text feature extraction network described above); the text encoder 12 outputs a text feature 20; the lightweight semantic understanding module 11 can preliminarily analyze the question of the user (S502) to obtain an extracted target 30, that is, the to-be-retrieved entity described above.

[0257] After step S503, the lightweight target detection network filtering is performed to obtain the preliminary screening related video segment 40, that is, the target detection network is used to detect the target in each video frame in the video to be searched to obtain the detection result of whether each video frame in the video to be searched contains the entity to be searched, and the video frame containing the detection result of the entity to be searched is taken as a video frame to be classified.

[0258] The preliminary screening related video segment 40 is input into the visual encoder 14 (that is, the visual feature extraction network described above) to obtain the visual feature 50.

[0259] After step S504, the feature attention calculation is performed, that is, the text feature 20 and the visual feature 50 are fused based on the attention mechanism to obtain the fusion feature to be used.

[0260] Step S505, video frame classification, that is, the fusion feature to be used is input into the classification network to obtain the first classification result of each video frame to be classified.

[0261] Step S506, CRF rule constraint, that is, the post-processing layer is used to correct the second classification result of each video frame to be classified based on the conditional random field algorithm to obtain the first classification result of each video frame to be classified, or the second prediction classification result of each video frame to be processed is corrected based on the preset rule to obtain the first prediction classification result of each video frame to be processed.

[0262] Step S507, return the corresponding start and end frames to obtain the video segment 60 meeting the requirements, that is, for each video frame to be classified in the video to be searched, the first classification result of which is the start frame, the closest video frame to be classified with the first classification result as the end frame is determined after the video frame to be classified, and the video segment between the video frame to be classified with the first classification result as the start frame and the video frame to be classified with the first classification result as the end frame is determined as the video segment meeting the text to be searched.

[0263] In an actual application scenario, the embodiment can quickly locate a specific behavior from a video, for example, for the behavior of "staying for a long time", the text to be searched can be input as "a man in a black jacket lingers in front of the bank for 3 minutes and then climbs over the fence". The video frame classification model can classify each video frame in the video to accurately find the start frame and the end frame, so as to return the video segment meeting the requirements.

[0264] Through the embodiment, family event backtracking can also be performed, that is, a specific behavior of a family member is retrieved from a video recorded by a home camera. For example, input the text to be retrieved: "a child wearing school uniform takes yogurt from a refrigerator and then sits at a desk and takes out a book". The classification result of the video frame classification model can also reflect the order of actions, that is, the text to be retrieved includes two actions, taking yogurt from the refrigerator and sitting at the desk and taking out a book. The video frame classification model can identify the start frame and the end frame of "taking yogurt from the refrigerator" and the start frame and the end frame of "sitting at the desk and taking out a book". The start frame of "taking yogurt from the refrigerator" can represent the action of "opening the refrigerator", and the end frame can represent the action of "closing the refrigerator". The start frame of "sitting at the desk and taking out a book" can represent the action of "sitting down", and the end frame can represent the action of "taking out a book". It can be seen that the frame-level behavior feature positioning can be realized through the scheme.

[0265] The embodiment of the present application proposes a novel video retrieval scheme. With the understanding ability of the model, an effective training and retrieval algorithm structure is constructed. Moreover, the scheme can classify video frames one by one, has high detection accuracy, and can improve the user experience and the application value of the product.

[0266] In the technical scheme of the present application, the operations of obtaining, storing, using, processing, transmitting, providing and disclosing of user personal information are all performed after obtaining the authorization of the user.

[0267] The embodiment of the present application also provides a training device of a video frame classification model, as shown in Figure 6 The embodiment of the present application also provides a training device of a video frame classification model, as shown in

[0268] The sample acquisition module 601 is configured to acquire a sample video and a sample text.

[0269] The sample feature extraction module 602 is configured to input the sample text into a text feature extraction network in a video frame classification model to be trained to obtain sample text features, and input each video frame in the sample video into a visual feature extraction network in the video frame classification model to obtain sample visual features of each video frame to be processed. The video frame classification model further includes a fusion network and a classification network.

[0270] The sample feature fusion module 603 is configured to fuse the sample text features and the sample visual features based on an attention mechanism by using the fusion network to obtain sample fusion features.

[0271] The classification prediction module 604 is configured to input the sample fusion feature into the classification network to obtain a first predicted classification result of each to-be-processed video frame; wherein the classification result of one to-be-processed video frame indicates that the to-be-processed video frame is one of the sample categories; and the sample categories include a start frame of a video segment conforming to the sample text, remaining frames except the start frame in the video segment conforming to the sample text, and external frames not conforming to the sample text.

[0272] The model parameter adjustment module 605 is configured to perform model parameter adjustment on the video frame classification model according to a difference between the first predicted classification result of each to-be-processed video frame and a true classification result of the to-be-processed video frame, until model convergence is achieved, to obtain a trained video frame classification model.

[0273] In an embodiment, the video frame classification model further includes an entity recognition layer and a target detection network; and the apparatus further includes:

[0274] The sample entity recognition module is configured to, before the sample feature extraction module 602 inputs each to-be-processed video frame in the sample video into the visual feature extraction network in the video frame classification model to obtain a sample visual feature of each to-be-processed video frame, input the sample text into the entity recognition layer to obtain a sample entity contained in the sample text.

[0275] The target prediction module is configured to perform target detection on each video frame in the sample video by using the target detection network to obtain a predicted detection result of whether each video frame in the sample video contains the sample entity, and take a video frame containing the sample entity as a to-be-processed video frame according to the predicted detection result.

[0276] In an embodiment, the remaining frames except the start frame in the video segment conforming to the sample text include an end frame and internal frames of the video segment conforming to the sample text; and the sample categories further include a single frame conforming to the sample text.

[0277] In an embodiment, the classification network includes a classification layer and a post-processing layer.

[0278] The classification prediction module 604 includes:

[0279] The classification prediction submodule is configured to input the sample fusion feature into the classification layer to obtain a second predicted classification result of each to-be-processed video frame.

[0280] The prediction correction submodule is configured to correct the second predicted classification result of each to-be-processed video frame by using the post-processing layer to obtain the first predicted classification result of each to-be-processed video frame.

[0281] In one embodiment, the prediction correction submodule is specifically configured to:

[0282] The second prediction classification result of each to-be-processed video frame is corrected based on a conditional random field algorithm by using the post-processing layer to obtain the first prediction classification result of each to-be-processed video frame, or the second prediction classification result of each to-be-processed video frame is corrected based on a preset rule by using the post-processing layer to obtain the first prediction classification result of each to-be-processed video frame; wherein the preset rule comprises:

[0283] The classification result indicates that the classification results of the two frames before and after the video frame of the single frame both indicate external frames;

[0284] The classification result indicates that the classification result of the frame after the video frame of the start frame indicates an intermediate frame or an end frame;

[0285] The classification result indicates that the classification result of the frame after the video frame of the intermediate frame indicates an intermediate frame or an end frame;

[0286] The classification result indicates that the classification result of the frame after the video frame of the end frame indicates an external frame;

[0287] The classification result indicates that the classification result of the frame after the video frame of the external frame does not indicate an end frame.

[0288] In one embodiment, the sample fusion module comprises:

[0289] The sample feature transformation submodule is configured to transform and process the sample text features by using a first matrix in the fusion network to obtain first to-be-processed features, and transform and process the sample text features by using a second matrix in the fusion network to obtain second to-be-processed features;

[0290] The sample feature fusion submodule is configured to take the obtained first to-be-processed features as a key matrix, take the obtained second to-be-processed features as a value matrix, and take the extracted sample visual features as a query matrix, and fuse the obtained first to-be-processed features, second to-be-processed features and sample visual features based on an attention mechanism to obtain sample fusion features.

[0291] The embodiments of the present application also provide a training device of a video frame classification model, as shown in Figure 7 The device comprises:

[0292] The information acquisition module 701 is configured to acquire to-be-retrieved text and to-be-retrieved video;

[0293] The to-be-utilized feature extraction module 702 is configured to input the to-be-retrieved text into a text feature extraction network in a pre-trained video frame classification model to obtain to-be-utilized text features, and input each to-be-classified video frame in the to-be-retrieved video into a visual feature extraction network in the video frame classification model to obtain to-be-utilized visual features of each to-be-classified video frame. The video frame classification model further includes a fusion network and a classification network, and is obtained by training based on the training method of the video frame classification model.

[0294] The to-be-utilized feature fusion module 703 is configured to fuse the to-be-utilized text features and the extracted to-be-utilized visual features based on an attention mechanism by using the fusion network to obtain to-be-utilized fusion features.

[0295] The classification module 704 is configured to input the to-be-utilized fusion features into the classification network to obtain a first classification result of each to-be-classified video frame. The classification result of one to-be-classified video frame indicates that the to-be-classified video frame is one of preset categories. The preset categories include a start frame of a video segment that meets the to-be-retrieved text, remaining frames of the video segment that meet the to-be-retrieved text except the start frame, and external frames that do not meet the to-be-retrieved text.

[0296] The video segment determination module 705 is configured to determine a video segment that meets the to-be-retrieved text in the to-be-retrieved video according to the first classification result of each to-be-classified video frame.

[0297] In an embodiment, the video frame classification model further includes an entity recognition layer and a target detection network. The apparatus further includes:

[0298] The to-be-retrieved entity recognition module is configured to input the to-be-retrieved text into the entity recognition layer to obtain a to-be-retrieved entity contained in the to-be-retrieved text before the to-be-utilized feature extraction module 702 inputs each to-be-classified video frame in the to-be-retrieved video into the visual feature extraction network in the video frame classification model to obtain to-be-utilized visual features of each to-be-classified video frame.

[0299] The target detection module is configured to perform target detection on each video frame in the to-be-retrieved video by using the target detection network to obtain a detection result of whether each video frame in the to-be-retrieved video contains the to-be-retrieved entity, and take a video frame containing the to-be-retrieved entity as a to-be-classified video frame.

[0300] In an embodiment, the remaining frames of the video segment that meet the to-be-retrieved text except the start frame include an end frame and internal frames of the video segment that meet the to-be-retrieved text. The preset categories further include a single frame that meets the to-be-retrieved text.

[0301] In an embodiment, the video segment determination module 705 is specifically configured to:

[0302] For each video frame to be classified with a first classification result as a start frame in the video to be searched, determine a video frame to be classified with a first classification result as an end frame closest to the video frame to be classified, and determine a video segment composed of the video frame to be classified with a first classification result as a start frame, the video frame to be classified with a first classification result as an end frame, and other video frames to be classified between the video frame to be classified with a first classification result as a start frame and the video frame to be classified with a first classification result as an end frame as a video segment conforming to the text to be searched.

[0303] In an embodiment, the classification network comprises a classification layer and a post-processing layer.

[0304] The classification module 704 comprises:

[0305] A classification sub-module is configured to input the fusion feature to be used into the classification layer to obtain a second classification result of each video frame to be classified.

[0306] A result correction sub-module is configured to correct the second classification result of each video frame to be classified by using the post-processing layer to obtain a first classification result of each video frame to be classified.

[0307] In an embodiment, the result correction sub-module is specifically configured to:

[0308] The post-processing layer is used to correct the second classification result of each video frame to be classified based on a conditional random field algorithm to obtain a first classification result of each video frame to be classified, or the post-processing layer is used to correct the second classification result of each video frame to be classified based on a preset rule to obtain a first classification result of each video frame to be classified; wherein the preset rule comprises:

[0309] The classification result indicates that the classification results of the frames before and after a single frame both represent external frames;

[0310] The classification result indicates that the classification result of the frame after a start frame represents an intermediate frame or an end frame;

[0311] The classification result indicates that the classification result of the frame after an intermediate frame represents an intermediate frame or an end frame;

[0312] The classification result indicates that the classification result of the frame after an end frame represents an external frame;

[0313] The classification result indicates that the classification result of the frame after an external frame does not represent an end frame.

[0314] In an embodiment, the to-be-utilized feature fusion module 703 comprises:

[0315] The to-be-utilized feature transformation sub-module is configured to transform the to-be-utilized text feature by using a first matrix in the fusion network to obtain a first to-be-utilized feature, and transform the to-be-utilized text feature by using a second matrix in the fusion network to obtain a second to-be-utilized feature.

[0316] The to-be-utilized feature fusion sub-module is configured to take the obtained first to-be-utilized feature as a key matrix, take the obtained second to-be-utilized feature as a value matrix, take each to-be-utilized visual feature as a query matrix, fuse the obtained first to-be-utilized feature, second to-be-utilized feature and each to-be-utilized visual feature based on an attention mechanism, and obtain a to-be-utilized fusion feature.

[0317] The embodiments of the present application also provide an electronic device, as shown in the accompanying drawings, comprising: Figure 8

[0318] The memory 801 is configured to store a computer program.

[0319] The processor 802 is configured to execute the program stored in the memory 801, and implement the steps of the training method of any video frame classification model or the retrieval method of a video segment.

[0320] The electronic device can further comprise a communication bus and / or a communication interface, and the processor 802, the communication interface and the memory 801 can communicate with each other through the communication bus.

[0321] The communication bus mentioned in the electronic device can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For the convenience of representation, only one thick line is shown in the figure, but it does not mean that there is only one bus or only one type of bus.

[0322] The communication interface is configured to communicate between the electronic device and other devices.

[0323] The memory can include a Random Access Memory (RAM) and can also include a Non-Volatile Memory (NVM), such as at least one disk memory. Optionally, the memory can also be at least one storage device located away from the aforementioned processor.

[0324] ​The processor described above can be a general processor, including a central processing unit (CPU), a network processor (NP), etc.; can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component.

[0325] In another embodiment provided in the present application, a computer readable storage medium is also provided, and the computer readable storage medium stores a computer program. The computer program is executed by a processor to implement the steps of the training method of the video frame classification model or the retrieval method of the video segment.

[0326] In another embodiment provided in the present application, a computer program product containing instructions is also provided, and when the computer program product is run on a computer, the computer is caused to execute the training method of the video frame classification model or the retrieval method of the video segment in any of the above embodiments.

[0327] In the above embodiments, all or part of the embodiments can be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part of the embodiments can be implemented in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network or other programmable device. The computer instructions can be stored in a computer readable storage medium or transmitted from one computer readable storage medium to another computer readable storage medium, for example, the computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center through wired (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (such as infrared, wireless, microwave, etc.) mode. The computer readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server, data center, etc. containing one or more available media sets. The available medium can be a magnetic medium (for example, floppy disk, hard disk, magnetic tape), an optical medium (for example, DVD), or a solid state disk (SSD) and the like.

[0328] It should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply the existence of any such actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or device comprising the element.

[0329] Each embodiment in this specification is described in a related manner. Similar portions between the various embodiments can be referenced to each other. Each embodiment focuses on the differences from other embodiments. In particular, the device, electronic device, and readable storage medium embodiments are generally similar to the method embodiments, so their descriptions are relatively simple. For related portions, reference can be made to the descriptions of the method embodiments.

[0330] The above description is only a preferred embodiment of the present application and is not intended to limit the scope of protection of the present application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application are included in the scope of protection of the present application.

Claims

1. A method for training a video frame classification model, characterized in that: The method comprises: Get sample videos and sample texts; Inputting the sample text into a text feature extraction network in a video frame classification model to be trained to obtain sample text features; and inputting each to-be-processed video frame in the sample video into a visual feature extraction network in the video frame classification model to obtain sample visual features of each to-be-processed video frame; wherein the video frame classification model further includes: a fusion network and a classification network; Using the fusion network, the sample text features and each sample visual feature are fused based on the attention mechanism to obtain a sample fusion feature; Inputting the sample fusion features into the classification network to obtain a first predicted classification result for each to-be-processed video frame; wherein the classification result of a to-be-processed video frame indicates that the to-be-processed video frame belongs to one of the sample categories; the sample categories include: the start frame of the video segment that conforms to the sample text, the remaining frames in the video segment that conforms to the sample text except the start frame, and external frames that do not conform to the sample text; According to the difference between the first predicted classification result of each video frame to be processed and the actual classification result of the video frame to be processed, the video frame classification model is adjusted until the model converges, thereby obtaining a trained video frame classification model.

2. The method according to claim 1, characterized in that The video frame classification model also includes: an entity recognition layer and an object detection network; Before inputting each to-be-processed video frame in the sample video into the visual feature extraction network in the video frame classification model to obtain sample visual features of each to-be-processed video frame, the method further includes: Inputting the sample text into the entity recognition layer to obtain sample entities contained in the sample text; The target detection network is used to perform target detection on each video frame in the sample video to obtain a predicted detection result of whether each video frame in the sample video contains the sample entity, and the video frame whose predicted detection result indicates that it contains the sample entity is used as the video frame to be processed.

3. The method according to claim 1 or 2, characterized in that The remaining frames except the start frame in the video segment that conforms to the sample text include: the end frame and internal frames of the video segment that conforms to the sample text; the sample category also includes: a single frame that conforms to the sample text.

4. The method according to claim 3, characterized in that The classification network includes: a classification layer and a post-processing layer; Inputting the sample fusion features into the classification network to obtain a first prediction classification result for each video frame to be processed, including: Inputting the sample fusion features into the classification layer to obtain a second prediction classification result for each video frame to be processed; The post-processing layer is used to correct the second prediction classification result of each video frame to be processed to obtain the first prediction classification result of each video frame to be processed.

5. The method according to claim 4, characterized in that Using the post-processing layer, the second prediction classification result of each video frame to be processed is corrected to obtain the first prediction classification result of each video frame to be processed, including: Using the post-processing layer, based on the conditional random field algorithm, the second prediction classification result of each to-be-processed video frame is corrected to obtain the first prediction classification result of each to-be-processed video frame; or, The post-processing layer is used to modify the second prediction classification result of each video frame to be processed based on preset rules to obtain the first prediction classification result of each video frame to be processed; wherein the preset rules include: The classification results of the two frames before and after the video frame of a single frame both represent external frames; The classification result of the video frame after the start frame represents the classification result of the video frame representing the middle frame or the end frame; The classification result of the video frame after the intermediate frame indicates the classification result of the intermediate frame or the end frame; The classification result of the video frame after the end frame represents an external frame; The classification result of the next frame of the video frame whose classification result indicates an external frame does not indicate an end frame.

6. The method according to claim 1, characterized in that The fusion network is used to fuse the sample text features and the visual features of each sample based on the attention mechanism to obtain sample fusion features, including: Using the first matrix in the fusion network to transform the sample text features to obtain a first feature to be processed; using the second matrix in the fusion network to transform the sample text features to obtain a second feature to be processed; The obtained first feature to be processed is used as the key matrix, the obtained second feature to be processed is used as the value matrix, and the extracted visual features of each sample are used as the query matrix. The obtained first feature to be processed, the second feature to be processed and the visual features of each sample are fused based on the attention mechanism to obtain the sample fusion feature.

7. A video clip retrieval method, characterized in that: The method comprises: Get the text and video to be retrieved; Inputting the text to be retrieved into a text feature extraction network in a pre-trained video frame classification model to obtain text features to be utilized; and inputting each video frame to be classified in the video to be retrieved into a visual feature extraction network in the video frame classification model to obtain visual features to be utilized for each video frame to be classified; wherein the video frame classification model further includes: a fusion network and a classification network; and the video frame classification model is trained based on the method according to any one of claims 1 to 6; Using the fusion network, based on the attention mechanism, the text features to be used and the extracted visual features to be used are fused to obtain fused features to be used; Inputting the fused features to be utilized into the classification network to obtain a first classification result for each video frame to be classified; wherein the classification result of a video frame to be classified indicates that the video frame to be classified belongs to one of the preset categories; the preset categories include: the starting frame of the video segment that matches the text to be retrieved, the remaining frames of the video segment that match the text to be retrieved except the starting frame, and external frames that do not match the text to be retrieved; According to the first classification result of each video frame to be classified, a video segment in the video to be retrieved that matches the text to be retrieved is determined.

8. The method according to claim 7, characterized in that The video frame classification model also includes: an entity recognition layer and an object detection network; Before inputting each to-be-classified video frame in the to-be-retrieved video into a visual feature extraction network in the video frame classification model to obtain a to-be-utilized visual feature of each to-be-classified video frame, the method further comprises: Inputting the text to be retrieved into the entity recognition layer to obtain the entities to be retrieved contained in the text to be retrieved; The target detection network is used to perform target detection on each video frame in the video to be retrieved, and a detection result of whether each video frame in the video to be retrieved contains the entity to be retrieved is obtained, and the video frame indicated by the detection result as containing the entity to be retrieved is used as the video frame to be classified.

9. The method according to claim 7 or 8, characterized in that The remaining frames except the start frame in the video segment that matches the text to be retrieved include: the end frame and internal frames of the video segment that matches the text to be retrieved; the preset category also includes: a single frame that matches the text to be retrieved.

10. The method according to claim 9, characterized in that The step of determining, based on the first classification result of each video frame to be classified, a video segment in the video to be retrieved that matches the text to be retrieved comprises: For each video frame to be classified whose first classification result is a start frame in the video to be retrieved, determining a video frame to be classified whose first classification result is an end frame that is closest to the video frame to be classified; Determine the video frame to be classified whose first classification result is a start frame, the video frame to be classified whose first classification result is an end frame, and a video clip consisting of other video frames to be classified between the video frame to be classified whose first classification result is a start frame and the video frame to be classified whose first classification result is an end frame, as a video clip that meets the text to be retrieved.

11. The method according to claim 9, characterized in that The classification network includes: a classification layer and a post-processing layer; Inputting the fusion features to be utilized into the classification network to obtain a first classification result for each video frame to be classified, including: Inputting the fusion features to be utilized into the classification layer to obtain a second classification result for each video frame to be classified; The post-processing layer is used to correct the second classification result of each video frame to be classified to obtain the first classification result of each video frame to be classified.

12. The method according to claim 11, characterized in that Using the post-processing layer, the second classification result of each video frame to be classified is corrected to obtain the first classification result of each video frame to be classified, including: Using the post-processing layer, based on the conditional random field algorithm, the second classification result of each video frame to be classified is corrected to obtain the first classification result of each video frame to be classified; or, The post-processing layer is used to modify the second classification result of each video frame to be classified based on preset rules to obtain the first classification result of each video frame to be classified; wherein the preset rules include: The classification results of the two frames before and after the video frame of a single frame both represent external frames; The classification result of the video frame after the start frame represents the classification result of the video frame representing the middle frame or the end frame; The classification result of the video frame after the intermediate frame indicates the classification result of the intermediate frame or the end frame; The classification result of the video frame after the end frame represents an external frame; The classification result of the next frame of the video frame whose classification result indicates an external frame does not indicate an end frame.

13. The method according to claim 7, characterized in that The fusion network is used to fuse the text features to be used and the extracted visual features to be used based on the attention mechanism to obtain the fusion features to be used, including: Using the first matrix in the fusion network to transform the text feature to be used, to obtain a first feature to be used; using the second matrix in the fusion network to transform the text feature to be used, to obtain a second feature to be used; The obtained first feature to be used is used as the key matrix, the obtained second feature to be used is used as the value matrix, and the extracted visual features to be used are used as the query matrix. The obtained first feature to be used, the second feature to be used and the visual features to be used are fused based on the attention mechanism to obtain the fused feature to be used.

14. A training device for a video frame classification model, characterized in that: The device comprises: Sample acquisition module, used to obtain sample videos and sample texts; A sample feature extraction module is configured to input the sample text into a text feature extraction network in a video frame classification model to be trained to obtain sample text features; and input each to-be-processed video frame in the sample video into a visual feature extraction network in the video frame classification model to obtain sample visual features of each to-be-processed video frame; wherein the video frame classification model further comprises: a fusion network and a classification network; A sample feature fusion module is used to use the fusion network to fuse the sample text features and the visual features of each sample based on the attention mechanism to obtain a sample fusion feature; a classification prediction module, configured to input the sample fusion features into the classification network to obtain a first predicted classification result for each to-be-processed video frame; wherein the classification result of a to-be-processed video frame indicates that the to-be-processed video frame belongs to one of the sample categories; the sample categories include: the start frame of the video segment that conforms to the sample text, the remaining frames of the video segment that conforms to the sample text except the start frame, and external frames that do not conform to the sample text; The model parameter adjustment module is used to adjust the model parameters of the video frame classification model according to the difference between the first predicted classification result of each video frame to be processed and the actual classification result of the video frame to be processed until the model converges to obtain a trained video frame classification model.

15. A video clip retrieval device, characterized in that: The device comprises: An information acquisition module is used to acquire the text and video to be retrieved; A feature extraction module for use, configured to input the text to be retrieved into a text feature extraction network in a pre-trained video frame classification model to obtain text features to be used; and input each video frame to be classified in the video to be retrieved into a visual feature extraction network in the video frame classification model to obtain visual features to be used for each video frame to be classified; wherein the video frame classification model further comprises: a fusion network and a classification network; and the video frame classification model is trained based on the method according to any one of claims 1 to 6; A feature fusion module to be used, configured to use the fusion network to fuse the text features to be used and the extracted visual features to be used based on an attention mechanism to obtain fused features to be used; a classification module, configured to input the fused features to be utilized into the classification network to obtain a first classification result for each video frame to be classified; wherein the classification result of a video frame to be classified indicates that the video frame to be classified belongs to one of preset categories; the preset categories include: the start frame of the video segment that matches the text to be retrieved, the remaining frames of the video segment that match the text to be retrieved except the start frame, and external frames that do not match the text to be retrieved; The video segment determination module is used to determine the video segment that matches the text to be retrieved in the video to be retrieved according to the first classification result of each video frame to be classified.

16. An electronic device, characterized in that: include: Memory for storing computer programs; A processor, configured to implement the method described in any one of claims 1 to 6 or any one of claims 7 to 13 when executing a program stored in a memory.

17. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 6 or any one of claims 7 to 13 is implemented.

18. A computer program product comprising instructions, characterized in that When the computer program product is run on a computer, the computer is enabled to execute the method according to any one of claims 1 to 6 or any one of claims 7 to 13.

Images