Video search method, apparatus, and computer readable storage medium

Abstract

The application discloses a video search method and device and a computer readable storage medium. The method comprises the following steps: obtaining a video search text, and performing text feature extraction on the video search text to obtain first text features; mapping the first text features to a video feature space corresponding to video information to obtain second text features; performing video feature extraction on each candidate video in a candidate video library to obtain first video features of each candidate video; mapping the first video features to a text feature space corresponding to text information to obtain second video features of each candidate video; and searching for a target candidate video corresponding to the video search text in the candidate video library based on a first similarity relationship between the first text features and the second video features and a second similarity relationship between the first video features and the second text features. The method can greatly improve the accuracy of searching for videos based on text.

Description

Technical Field

[0001] This application relates to the field of video technology, specifically to a video search method, apparatus, and computer-readable storage medium. Background Technology

[0002] In recent years, with the accelerating pace of life, people have tended to choose efficient and convenient channels for information acquisition. Short videos, with their ability to provide a large amount of information in a short time and their ability to meet the fragmented information consumption needs of current internet users, have gradually gained popularity among a large number of internet users.

[0003] Currently, users can enter text information in the search bar of short video apps to search for videos they want to watch. However, the videos found based on the user's text input are not yet accurate enough. Summary of the Invention

[0004] This application provides a video search method, apparatus, and computer-readable storage medium, which can improve the accuracy of video search.

[0005] The first aspect of this application provides a video search method, the method comprising:

[0006] Obtain the video search text and extract text features from the video search text to obtain the first text feature;

[0007] The first text features are mapped to the video feature space corresponding to the video information to obtain the second text features;

[0008] For each candidate video in the candidate video library, video features are extracted to obtain the first video feature of each candidate video.

[0009] The first video features are mapped to the text feature space corresponding to the text information to obtain the second video features of each candidate video.

[0010] Based on the first similarity relationship between the first text feature and the second video feature, and the second similarity relationship between the first video feature and the second text feature, the system searches for target candidate videos in the candidate video library that correspond to the video search text.

[0011] Accordingly, a second aspect of this application provides a video search device, the device comprising:

[0012] The acquisition unit is used to acquire video search text and extract text features from the video search text to obtain the first text feature;

[0013] The first mapping unit is used to map the first text features to the video feature space corresponding to the video information to obtain the second text features;

[0014] The extraction unit is used to extract video features from each candidate video in the candidate video library to obtain the first video feature of each candidate video.

[0015] The second mapping unit is used to map the first video features to the text feature space corresponding to the text information to obtain the second video features of each candidate video.

[0016] The search unit is used to search for target candidate videos corresponding to the video search text in the candidate video library based on a first similarity relationship between the first text features and the second video features, and a second similarity relationship between the first video features and the second text features.

[0017] In some embodiments, the acquiring unit includes:

[0018] The first acquisition subunit is used to acquire video search text and use the trained first long short-term memory unit to extract text features from the video search text to obtain multiple transition text features.

[0019] The first calculation subunit is used to calculate the average value of the multiple transition text features to obtain the first text feature.

[0020] In some embodiments, the first mapping unit includes:

[0021] The second acquisition subunit is used to acquire video memory features, which are features that record visual semantics in video frames during video feature extraction.

[0022] The transfer subunit is used to transfer the first text features to the video feature space based on the video memory features to obtain the second text features.

[0023] In some embodiments, the migration subunit includes:

[0024] The first calculation module is used to calculate the Euclidean distance between the video memory features and the first text features to obtain similar features;

[0025] The mapping module is used to perform fully connected mapping on the similar features based on the trained fully connected network to obtain selected features;

[0026] The second calculation module is used to calculate the second text feature based on the selected feature and the first text feature.

[0027] In some embodiments, the second computing module includes:

[0028] The first calculation submodule is used to calculate the average value of the selection features to obtain the average selection features;

[0029] The second calculation submodule is used to calculate the product of the average selection feature and the first text feature to obtain the second text feature.

[0030] In some embodiments, the extraction unit includes:

[0031] The first extraction subunit is used to extract video frame features from the video frames of each candidate video in the candidate video library to obtain the video frame features corresponding to each candidate video.

[0032] The processing subunit is used to perform attention processing on the video frame features corresponding to each candidate video using the video memory features to obtain the first video feature of each candidate video.

[0033] In some embodiments, extracting a subunit includes:

[0034] The segmentation module is used to divide each candidate video in the candidate video library into multiple video frames.

[0035] The processing module is used to perform convolution processing on multiple video frames corresponding to each candidate video using a preset neural network model to obtain the video frame features corresponding to each candidate video.

[0036] In some embodiments, the second mapping unit includes:

[0037] The first mapping subunit is used to map the first video features of each candidate video based on the trained second long short-term memory unit to obtain multiple hidden features of each candidate video.

[0038] The second calculation subunit is used to calculate the second video feature of each candidate video based on multiple hidden features of each candidate video and the corresponding predicted sentence length.

[0039] In some embodiments, the video search method provided in this application further includes:

[0040] The third acquisition subunit is used to acquire training sample data, which includes sample video data and sample text data corresponding to the sample video data.

[0041] The second extraction subunit is used to extract video frame features from the sample video data, and to perform attention processing on the extracted video frame features using the video memory features to be trained, so as to obtain the first sample video features of the sample video data.

[0042] The second mapping subunit is used to input the first sample video features into the first long short-term memory unit to be trained for mapping processing, and to calculate the second sample video features based on the output hidden features.

[0043] The second mapping subunit is used to input the sample text data into the second long short-term memory unit to be trained for mapping processing, and to calculate the mean of the output features to obtain the first sample text features.

[0044] The third computational subunit is used to input the Euclidean distance between the first sample text features and the video memory features to be trained into the fully connected network to be trained, and to calculate the second sample text features based on the output features of the fully connected network to be trained and the first sample text features.

[0045] The training subunit is used to train the video memory features to be trained, the first long short-term memory unit, the second long short-term memory unit, and the fully connected network based on the first sample video features, the second sample video features, the first sample text features, and the second sample text features.

[0046] In some embodiments, the training subunit includes:

[0047] The third calculation module is used to calculate the first sample similarity between the first sample video features and the second sample text features, and to calculate the second sample similarity between the second sample video features and the first sample text features.

[0048] The fourth calculation module is used to calculate the mean of the first sample similarity and the second sample similarity to obtain the target sample similarity;

[0049] The fifth calculation module is used to calculate a first loss function value based on the similarity of the target samples, wherein the first loss function value represents the contrast loss;

[0050] The acquisition module is used to acquire the value of the second loss function that constrains the second long short-term memory unit;

[0051] The adjustment module is used to perform gradient backpropagation optimization based on the first loss function value and the second loss function value, so as to adjust the video memory features to be trained, the first long short-term memory unit, the second long short-term memory unit, and the fully connected network.

[0052] In some embodiments, the search unit includes:

[0053] The fourth calculation subunit is used to calculate the similarity between the first text feature and the second video feature to obtain the first similarity.

[0054] The fifth calculation subunit is used to calculate the similarity between the second text feature and the first video feature to obtain the second similarity.

[0055] The sixth calculation subunit is used to calculate the average of the first similarity and the second similarity to obtain the target similarity between the video search text and each candidate video;

[0056] A subunit is defined to determine the target candidate video corresponding to the video search text in the candidate video library based on the target similarity.

[0057] A third aspect of this application also provides a computer-readable storage medium storing a plurality of instructions adapted for loading by a processor to perform the steps in the video search method provided in the first aspect of this application.

[0058] A fourth aspect of this application provides a computer device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the video search method provided in the first aspect of this application.

[0059] The fifth aspect of this application provides a computer program product, including a computer program / instructions, which, when executed by a processor, implement the steps in the video search method provided in the first aspect.

[0060] The video search method provided in this application embodiment obtains video search text and extracts text features from the video search text to obtain a first text feature; maps the first text feature to the video feature space corresponding to the video information to obtain a second text feature; extracts video features from each candidate video in the candidate video library to obtain a first video feature for each candidate video; maps the first video feature to the text feature space corresponding to the text information to obtain a second video feature for each candidate video; and searches for a target candidate video corresponding to the video search text in the candidate video library based on a first similarity relationship between the first text feature and the second video feature, and a second similarity relationship between the first video feature and the second text feature.

[0061] Therefore, the video search method provided in this application calculates the similarity of video features and text features separately in the video space and text space during video search, avoiding information loss caused by alignment in a single space. This significantly improves the accuracy of text-based video searches. Attached Figure Description

[0062] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0063] Figure 1 This is a schematic diagram of a video search scenario in this application;

[0064] Figure 2 This is a flowchart illustrating the video search method provided in this application;

[0065] Figure 3 This is a schematic diagram of the model structure of the video-text similarity calculation model provided in this application;

[0066] Figure 4 This is another flowchart illustrating the video search method provided in this application;

[0067] Figure 5 This is a schematic diagram of the video search device provided in this application;

[0068] Figure 6 This is a schematic diagram of the structure of the computer device provided in this application. Detailed Implementation

[0069] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0070] This invention provides a video search method, apparatus, computer-readable storage medium, and computer device. The video search method can be used in a video search apparatus. The video search apparatus can be integrated into a computer device, which can be a terminal or a server. The terminal can be a mobile phone, tablet computer, laptop computer, smart TV, wearable smart device, personal computer (PC), or in-vehicle terminal, etc. The server can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, Content Delivery Network (CDN), and big data and artificial intelligence platforms. The server can also be a node in a blockchain.

[0071] Please see Figure 1 This is a schematic diagram of a scenario for the video search method provided in this application. As shown in the figure, server A receives video search text sent by terminal B. The video search text can be the search text entered by the user in a video application installed on the terminal. Then, server A extracts text features from the video search text to obtain a first text feature; maps the first text feature to the video feature space corresponding to the video information to obtain a second text feature; extracts video features from each candidate video in the candidate video library to obtain a first video feature for each candidate video; maps the first video feature to the text feature space corresponding to the text information to obtain a second video feature for each candidate video; and searches for a target candidate video corresponding to the video search text in the candidate video library based on a first similarity relationship between the first text feature and the second video feature, and a second similarity relationship between the first video feature and the second text feature. Further, server A can send the target candidate video to terminal B for display. It can be understood that there can be one or more target candidate videos.

[0072] It should be noted that, Figure 1 The illustrated video search scenario diagram is merely an example. The video search scenario described in this application is intended to more clearly illustrate the technical solution of this application and does not constitute a limitation on the technical solution provided in this application. Those skilled in the art will understand that, with the evolution of video search scenarios and the emergence of new business scenarios, the technical solution provided in this application is equally applicable to similar technical problems.

[0073] The implementation scenarios described above will be explained in detail below.

[0074] In various video search applications and products, the ability to accurately search and locate corresponding videos using text is crucial. However, due to the significant differences between video and text modalities, establishing accurate correspondences between videos and texts with different distributions is a core issue in video search research.

[0075] In related technologies, when performing video search based on text, videos and text can be mapped to the same space, bringing video-text pairs representing the same semantics closer together and separating those with different semantics. However, because videos contain a large amount of visual information such as entity description, spatial layout, and edge texture, while text not only describes entities but is also adept at describing abstract concepts such as relationships and emotions, forcibly aligning videos and text in a common space, their common parts (e.g., entities) are preserved, while information specific to each modality (e.g., spatial layout in videos and abstract concepts in text) is weakened during model training. This leads to information loss during alignment in the common space, causing text to incorrectly match videos with the same entities while ignoring other features. This reduces the accuracy of text-based video search. To address the aforementioned problem of low accuracy in video search, this application provides a video search method to improve the accuracy of video search.

[0076] This application embodiment will be described from the perspective of a video search device, which can be integrated into a computer device. The computer device can be a terminal or a server. The terminal can be a mobile phone, tablet computer, laptop computer, smart TV, wearable smart device, personal computer (PC), or in-vehicle terminal, etc. The server can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery network (CDN) acceleration services, and big data and artificial intelligence platforms. Figure 2 The diagram shown is a flowchart of the video search method provided in this application, which includes:

[0077] Step 101: Obtain the video search text and extract text features from the video search text to obtain the first text feature.

[0078] In this embodiment, when a user inputs text information into a video application installed on a terminal to perform a video search, the video application can send the user-input text information to a video search device for the search. The video application can be a short video application or other video playback applications. After the video search device obtains the video search text from the terminal, it can extract text features from the video search text to obtain first text features. This text feature extraction can involve extracting features of the video search text within the space corresponding to the text modality. The space corresponding to the text modality can be called the text space or symbol space.

[0079] Specifically, in some embodiments, video search text is obtained, and text feature extraction is performed on the video search text to obtain a first text feature, including:

[0080] 1. Obtain video search text and use the first long short-term memory unit after training to extract text features from the video search text to obtain multiple transition text features;

[0081] 2. Calculate the average value of multiple transition text features to obtain the first text feature.

[0082] In this embodiment, the extraction of first text features in the symbol space of the video search text can be achieved using a text encoder based on Long Short-Term Memory (LSTM). The text encoder can contain one or more LSTM units. Specifically, this application also provides a video-text similarity calculation model, where the aforementioned LSTM-based text encoder is a module within this model. To distinguish it from other LSTM modules in the model, the LSTM in the text encoder is referred to here as the first LSTM unit. During the training of the aforementioned video-text similarity model, the parameters of the LSTM in the text encoder are trained and adjusted accordingly. The training process of the video-text similarity calculation model will be described in detail below.

[0083] When a video search device acquires video search text, it can input the video search text and candidate videos into a video-text similarity calculation model for similarity calculation. At this point, the text encoder in the model can extract text features in symbolic space from the acquired video search text. Specifically, it uses a trained first long short-term memory unit to extract features from the video search text, obtaining multiple transitional text features. The mean of these multiple transitional text features is then calculated to obtain the text features of the video search text in symbolic space. To distinguish it from other text features, this can be referred to as the first text feature.

[0084] Step 102: Map the first text features to the video feature space corresponding to the video information to obtain the second text features.

[0085] In this embodiment of the application, after extracting the text features of the video search text in the symbol space, namely the aforementioned first text features, the text features of the video search text in the symbol space and related visual features can be further linked together to perform feature selection at the dimensional level, thereby transferring the text features distributed in the symbol space to the video space (also known as the latent space) to obtain the text features in the latent space.

[0086] Specifically, the aforementioned video-text similarity calculation model may also include a latent space feature selection module. After the text encoder extracts text features from the video search text to obtain the first text feature, the first text feature can be input into the latent space feature selection module for feature mapping to obtain the text feature in the latent space, which can be referred to as the second text feature.

[0087] In some embodiments, mapping the first text features to the video feature space corresponding to the video information to obtain the second text features includes:

[0088] 1. Obtain video memory features, which are features that record the visual semantics in video frames during video feature extraction;

[0089] 2. Based on the video memory features, the first text features are transferred to the video feature space to obtain the second text features.

[0090] In this embodiment, the first text features of the video search text in the symbolic space are mapped to the latent space corresponding to the video. Specifically, video memory features can be used for this mapping process. Here, video memory features can be features learned during the training of the aforementioned video-text similarity calculation model that record important visual semantics in video frames during video feature extraction. These features can extract important visual semantics from the text features in the symbolic space, thereby obtaining the second text features of the video search text in the latent space.

[0091] In some embodiments, the second text features are obtained by transferring the first text features to the video feature space based on video memory features, including:

[0092] 2.1 Calculate the Euclidean distance between the video memory features and the first text features to obtain similar features;

[0093] 2.2. Based on the trained fully connected network, similar features are mapped using fully connected methods to obtain selected features;

[0094] 2.3. The second text feature is calculated based on the selected features and the first text feature.

[0095] In this embodiment, the aforementioned latent space feature selection module further includes a fully connected network, which can specifically be a multilayer perceptron (MLP). During the training of the video-text similarity calculation model, the parameters of this fully connected network can also be adjusted simultaneously to obtain the trained fully connected network. The fully connected network in the latent space feature selection module can be a single layer or multiple layers.

[0096] After the latent space feature selection module receives the first text feature, it can first calculate the Euclidean distance between the aforementioned video memory feature and the first text feature to obtain the similarity feature between the video memory feature and the first text feature in the channel dimension, which can be called the similarity feature. Then, this similarity feature is input into the trained fully connected network for feature selection to obtain the selected feature. Further, the second text feature can be calculated based on the selected feature and the first text feature.

[0097] In some embodiments, a second text feature is calculated based on the selected feature and the first text feature, including:

[0098] 2.3.1 Calculate the average value of the selection features to obtain the average selection feature;

[0099] 2.3.2 Calculate the product of the average selected feature and the first text feature to obtain the second text feature.

[0100] In this embodiment of the application, the specific process of calculating the second text feature based on the selection feature and the first text feature can be as follows: first, calculate the mean of the selection feature to obtain the average selection feature, and then further calculate the product between the average selection feature and the first text feature to obtain the second text feature.

[0101] Step 103: Extract video features from each candidate video in the candidate video library to obtain the first video feature of each candidate video.

[0102] In this embodiment, while extracting the text features of the video search text in the symbolic space and latent space, video features can also be extracted from the candidate videos in the candidate video library. Specifically, a first video feature in the latent space and a second video feature in the symbolic space can be extracted from the candidate videos. The candidate video library can be all video data stored on the server of the video application, or it can be video data filtered based on a certain video type.

[0103] In some embodiments, video features are extracted from each candidate video in the candidate video library to obtain a first video feature for each candidate video, including:

[0104] 1. Extract video frame features from the video frames of each candidate video in the candidate video library to obtain the video frame features corresponding to each candidate video;

[0105] 2. Attention processing is performed on the video frame features corresponding to each candidate video using video memory features to obtain the first video feature of each candidate video.

[0106] The first video feature in the latent space of the candidate videos in the candidate video library is extracted. Specifically, this extraction can be performed using a memory-based video encoder in the video-text similarity calculation model. Specifically, the video search device can input the candidate videos into the memory-based video encoder to extract video frame features for each video frame of the candidate videos, obtaining multiple video frame features corresponding to the candidate videos.

[0107] In this embodiment, the memory-based video encoder also has a memory storage module, which consists of multiple learnable video memory features. When the video encoder extracts video features from candidate videos, the memory storage module uses the learned video memory features to perform attention processing on multiple video frame features of the candidate videos to obtain the first video feature of the candidate video in the latent space.

[0108] In some embodiments, video frame feature extraction is performed on the video frames of each candidate video in the candidate video library to obtain the video frame features corresponding to each candidate video, including:

[0109] 1.1 Divide each candidate video in the candidate video library into multiple video frames;

[0110] 1.2. A pre-defined neural network model is used to perform convolution processing on multiple video frames corresponding to each candidate video to obtain the video frame features corresponding to each candidate video.

[0111] In this embodiment of the application, the memory-based video encoder may further include a convolutional neural network layer. When extracting video frame features from video frames of candidate videos, the convolutional neural network can be used to perform convolution processing on the video frames to obtain video frame features.

[0112] Specifically, when candidate videos are obtained, they can be divided into multiple video frames. Then, a convolutional neural network is used to perform convolution processing on each video frame to obtain the video frame features of each candidate video.

[0113] Step 104: Map the first video features to the text feature space corresponding to the text information to obtain the second video features of each candidate video.

[0114] After extracting the first video feature of the candidate video in the latent space, the first video feature can be further mapped to the text feature space corresponding to the text information, that is, mapped to the symbolic feature space, to obtain the second video feature of the candidate video in the symbolic feature space.

[0115] Specifically, in the embodiments of this application, the video-text similarity calculation model provided in this application may also include a symbol generator, which can be used to map the first video features to the symbol space to obtain the second video features of the candidate video.

[0116] In some embodiments, mapping the first video features to the text feature space corresponding to the text information to obtain the second video features for each candidate video includes:

[0117] 1. Based on the trained second long short-term memory unit, the first video features of each candidate video are mapped to obtain multiple hidden features of each candidate video;

[0118] 2. The second video feature of each candidate video is calculated based on the multiple hidden features of each candidate video and the corresponding predicted sentence length.

[0119] In this embodiment, the video-text similarity calculation model includes a symbol generator that can contain multiple second long short-term memory units (LSTMs). During the training of the video-text similarity model, the parameters of these LSTMs can be adjusted to obtain trained LSTMs. Thus, when the first video feature is input to the symbol generator, the trained LSTMs can be used to map the first video feature, obtaining multiple hidden features corresponding to each candidate video. Then, based on the predicted sentence length, these hidden features can be averaged to obtain the second video feature for each candidate video.

[0120] In some embodiments, before obtaining video memory features, which are features recorded in the visual semantics of video frames during video feature extraction, the method further includes:

[0121] a. Obtain training sample data, which includes sample video data and sample text data corresponding to the sample video data;

[0122] b. Extract video frame features from the sample video data, and use the video memory features to be trained to perform attention processing on the extracted video frame features to obtain the first sample video features of the sample video data.

[0123] c. Input the first sample video features into the first long short-term memory unit to be trained for mapping processing, and calculate the second sample video features based on the output hidden features;

[0124] d. Input the sample text data into the second long short-term memory unit to be trained for mapping processing, and calculate the mean of the output features to obtain the first sample text features;

[0125] e. Input the Euclidean distance between the first sample text features and the video memory features to be trained into the fully connected network to be trained, and calculate the second sample text features based on the output features of the fully connected network to be trained and the first sample text features.

[0126] f. Train the video memory features to be trained, the first long short-term memory unit, the second long short-term memory unit, and the fully connected network based on the first sample video features, the second sample video features, the first sample text features, and the second sample text features.

[0127] As mentioned above, the video search method provided in this application is based on a trained video-text similarity calculation model. The aforementioned video memory features can be learned during model training, and the trained first long short-term memory unit, trained second long short-term memory unit, and trained fully connected network can all be trained during the training of the aforementioned video-text similarity calculation model. The model structure and training process are described in detail below.

[0128] like Figure 3The figure shows a schematic diagram of the video-text similarity calculation model provided in this application. As shown, the video-text similarity calculation model provides in this application takes a video frame sequence and text as input, and outputs the similarity value between the video frame sequence and the text. The model structure specifically includes a video encoder 10, a symbol generator 20, a text encoder 30, and a latent space feature selector 40. The video encoder 10 includes a convolutional neural network, a memory storage module, and an attention module. The convolutional neural network is used to extract features from the video frame sequence, obtaining the video frame features corresponding to each video frame. Here, the convolutional neural network can be a pre-trained neural network; its parameters are not updated during the training of the video-text similarity calculation model. The memory storage module provides video memory features, i.e., video memory vectors, which are used to record important visual semantic information. The attention module performs attention calculations on the video memory vectors and continuous video frame features to integrate the continuous video frame features and obtain the global features of the input video, i.e., the first video feature. The symbol generator 20 includes a second long short-term memory unit (LSSMU), specifically multiple LLSMUs. These LLSMUs project video features distributed in the latent space onto the symbol space to obtain video features in the symbol space, i.e., second video features. The text encoder 30 includes a first LLSMU and a first computation module. The first LLSMU can be a bidirectional LLSMU, used to generate multiple latent features of the text. The first computation module averages these multiple latent features to obtain text features in the symbol space, i.e., first text features. The latent space feature selector 40 includes a memory storage module, a second computation module, a fully connected network, and a third computation module. The memory storage module still provides video memory features, i.e., video memory vectors. The second computation module calculates the Euclidean distance between the video memory features and the first text features. The fully connected network transforms the Euclidean distance into a selection feature of the video memory features on the channel dimension of the first text features. The third computation module calculates the mean of the selected features and the product of this mean and the first video features to obtain text features in the latent space, i.e., second text features. The fourth calculation module is used to calculate the first similarity between the first text feature and the second video feature, calculate the second similarity between the second text feature and the first video feature, and calculate the mean of the first similarity and the second similarity to obtain the similarity between the video and the text.

[0129] The training process for this model begins by acquiring training samples, which include multiple positive and negative sample pairs. In positive sample pairs, the video and text are related, with a similarity of 1; in negative sample pairs, they are unrelated, with a similarity of 0. These sample pairs are then used as input to the model, and the corresponding similarity values ​​are used as outputs to train the video-text similarity calculation model, resulting in the trained model. This leads to the learned video memory features, the trained first long short-term memory unit (LSTM), the trained second LSM, and the trained fully connected network.

[0130] In some embodiments, training is performed on the video memory features to be trained, the first long short-term memory unit, the second long short-term memory unit, and the fully connected network based on the first sample video features, the second sample video features, the first sample text features, and the second sample text features, including:

[0131] f1. Calculate the first sample similarity between the first sample video features and the second sample text features, and calculate the second sample similarity between the second sample video features and the first sample text features;

[0132] f2. Calculate the mean of the similarity of the first sample and the similarity of the second sample to obtain the similarity of the target sample;

[0133] f3. Calculate the first loss function value based on the similarity of the target samples. The first loss function value represents the contrast loss.

[0134] f4. Obtain the value of the second loss function that constrains the second long short-term memory unit;

[0135] f5. Gradient backpropagation optimization is performed based on the first loss function value and the second loss function value to adjust the video memory features to be trained, the first long short-term memory unit, the second long short-term memory unit, and the fully connected network.

[0136] In this embodiment, when training the video-text similarity calculation model provided in this application, the model can be trained under constraints based on contrastive loss and generation loss used to train the symbol generator. Contrastive loss can bring positive sample pairs closer together and distance negative sample pairs further apart, while the generation loss function can improve the accuracy of the generated symbol space video features.

[0137] Step 105: Based on the first similarity relationship between the first text feature and the second video feature, and the second similarity relationship between the first video feature and the second text feature, search for target candidate videos in the candidate video library that correspond to the video search text.

[0138] The process involves calculating the first video feature in the latent space and the second video feature in the symbolic space for each candidate video, as well as the first text feature in the symbolic space and the second text feature in the latent space for the video search text. Then, similarity calculations are performed between the latent space video features and text features (i.e., between the first video feature and the second text feature); and between the symbolic space video features and text features (i.e., between the second video feature and the first text feature). Based on the similarity relationship between the two spaces, the similarity between the video search text and each candidate video is calculated. Finally, the similarity between the video search text and each candidate video determines the target video to be recommended to the user. Here, the target video can be one or multiple.

[0139] In some embodiments, searching for target candidate videos corresponding to the video search text in the candidate video library based on a first similarity relationship between the first text features and the second video features, and a second similarity relationship between the first video features and the second text features, includes:

[0140] 1. Calculate the similarity between the first text feature and the second video feature to obtain the first similarity score;

[0141] 2. Calculate the similarity between the second text feature and the first video feature to obtain the second similarity.

[0142] 3. Calculate the mean of the first similarity and the second similarity to obtain the target similarity between the video search text and each candidate video;

[0143] 4. Based on the target similarity, determine the target candidate video corresponding to the video search text in the candidate video library.

[0144] In this embodiment, after calculating the first similarity and second similarity of each candidate video and the video search text in different spaces, the target similarity between the video search text and each candidate video can be determined based on the average of the similarities in the two spaces. Furthermore, video search can be performed based on the target similarity between the video search text and each candidate video; specifically, candidate videos with a target similarity greater than a preset threshold can be identified as the target videos corresponding to the video search text.

[0145] As described above, the video search method provided in this application embodiment obtains video search text and extracts text features from the video search text to obtain a first text feature; maps the first text feature to the video feature space corresponding to the video information to obtain a second text feature; extracts video features from each candidate video in the candidate video library to obtain a first video feature for each candidate video; maps the first video feature to the text feature space corresponding to the text information to obtain a second video feature for each candidate video; and searches for a target candidate video corresponding to the video search text in the candidate video library based on a first similarity relationship between the first text feature and the second video feature, and a second similarity relationship between the first video feature and the second text feature.

[0146] Therefore, the video search method provided in this application calculates the similarity of video features and text features separately in the video space and text space during video search, avoiding information loss caused by alignment in a single space. This significantly improves the accuracy of text-based video searches. The video search method provided in this application can deeply mine the unique information of both video and text, maintaining the integrity of data in the original space for a given modality, thereby reducing information loss during video and text alignment and ultimately enhancing the performance of video-text search.

[0147] This application also provides a video search method that can be used in a computer device, which can be a terminal or a server. For example... Figure 4 The diagram shown is another flowchart illustrating the video search method provided in this application. The method specifically includes:

[0148] Step 201: The computer device acquires training sample data.

[0149] In related technologies, text-based video retrieval methods map videos and texts to a common space for alignment, leading to the loss of information belonging to each modality. Specifically, for example, a video might contain visual information such as "a group of people," "food," "balloons," and "cake," while the corresponding correct text might contain abstract concepts like "a group of people," "happy," and "birthday party." Traditional methods based on a common space map videos and texts to the same space, where only common entities like "a group of people" are retained for similarity calculation. Information such as "birthday party" and "happy" in the text, and balloons and cake in the video, are unlikely to contribute to increasing similarity. Therefore, the video might have a higher similarity score to text containing "food" and "a group of people" than the correct text, resulting in retrieval failure.

[0150] To address this issue, this application provides a video retrieval method that aims to avoid the loss of modality-specific information during video and text alignment, thereby improving the accuracy of video search. This method fully leverages the complementarity of video and text by mimicking the associative nature of picture-based storytelling and sentence reading, thus enhancing video-text retrieval performance. The core idea of ​​this method is to reconstruct the alignment of the common space into alignments in two distinct spaces: the latent space and the symbolic space. The latent space contains visual information (such as spatial layout and entities) learned from a pre-trained convolutional neural network. The symbolic space captures abstract knowledge generated by language (such as concepts and emotions). This method preserves the integrity of the original spatial information for data from one modality, then maps data from another modality to this space for similarity calculation, and vice versa.

[0151] In this application embodiment, a video-text similarity calculation model that can achieve the above-mentioned functions is provided. Please refer to the following: Figure 3 This is a schematic diagram of the model structure for calculating video-text similarity provided in this application. The model will be referred to as the similarity calculation model below. Before performing video search, the similarity calculation model can be trained. Specifically, sample data for training the similarity calculation model can be obtained, which may include positive and negative sample data. Positive sample data consists of relevant video-text pairs, while negative sample data consists of irrelevant video-text pairs.

[0152] Step 202: The computer equipment uses training sample data to train a similarity calculation model.

[0153] After obtaining the training sample data, the similarity calculation model can be trained using the training sample data, and then the trained similarity calculation model can be deployed to perform video search.

[0154] Specifically, the sample data can be input into the similarity calculation model to be trained to extract video features in the latent space and video features in the symbolic space of the sample videos, as well as text features in the latent space and text features in the symbolic space of the sample texts. Then, the similarity data corresponding to each sample data is calculated based on these features. Further, the following contrastive learning loss function can be constructed based on the similarity data of each sample pair:

[0155] L rank =[Δ+s(v + , t - )-s(v + , t + )] + +[Δ+s(v - , t -)-s(v + , t + )] +

[0156] Where (v+, t+) represents a related video-text pair (positive sample pair), and (v+, t-) represents an unrelated video-text pair (negative sample pair). Δ is a predefined threshold used to control the similarity distance between positive and negative sample pairs; s() is the similarity operator, calculated using the following formula:

[0157]

[0158] Among them, v s Let t be the video feature in the symbolic space, t be the text feature in the symbolic space, and v be the video feature in the latent space. L Textual features in the latent space.

[0159] In addition, since the similarity calculation model also includes a symbol generation module, it can be jointly trained using the loss function for visual description generation and the contrastive learning loss function mentioned above. The loss function for visual description generation is as follows:

[0160]

[0161] Among them, h i c represents the i-th hidden layer in LSTM. i This represents the nth word to be predicted. p(c i |h i ) represents the predicted probability of the i-th word based on the i-th hidden layer. N w This represents the number of words in the sentence to be predicted.

[0162] Thus, the loss function for joint training is as follows:

[0163] L = L rank +λL generate

[0164] Here, λ is the parameter that balances the two loss functions.

[0165] Furthermore, the similarity calculation model can be trained based on the aforementioned joint training loss function to obtain the trained similarity calculation model. During training, the parameters of the first LSTM in the text encoder, the second LSTM in the symbol generator, and the fully connected network in the latent space feature extractor can be trained and corrected simultaneously to obtain the trained first LSTM, the trained second LSTM, and the trained fully connected network. Simultaneously, the memory vector in the memory storage module can also be learned.

[0166] Step 203: When the video search text is received, the computer device uses the text encoder in the similarity calculation model to encode the video search text to obtain the first text feature.

[0167] After training the similarity calculation model, the model can be deployed online. When a user inputs video search text, the model can be used to calculate the accurate similarity between the video search text and each video in the candidate video library, and then an accurate video search can be performed based on the similarity.

[0168] Specifically, when a video search text is received, the computer device can use the text encoder in the above model to encode the video search text and obtain the first text feature in the symbol space.

[0169] Specifically, for the input text, a sentence is first transformed into a series of vectors using a word embedding model. These word vectors are then fed into a bidirectional LSTM model (i.e., the first LSTM trained previously) to generate global text features t containing sentence information. This process is as follows:

[0170]

[0171]

[0172]

[0173]

[0174] Among them, w i x is the output of the i-th hidden layer. i w is the input to the i-th hidden layer. i-1 This is the output of the (i-1)th hidden layer. i+1 For the output of the (i+1)th hidden layer, N w The length of the sentence.

[0175] Step 204: The computer device uses the latent space feature selector in the similarity calculation model to select the first text feature and obtain the second text feature.

[0176] Furthermore, the first textual feature of the video search text in the symbolic space can be fed into the latent space feature selector in the similarity calculation model for feature selection, resulting in the second textual feature. In this latent space feature selector, the first textual feature and related visual features can be linked for dimensional feature selection, thereby transferring the textual features distributed in the symbolic space to the latent space, resulting in the second textual feature located in the latent space. Specifically, we first use Euclidean distance to obtain the similarity vector d between the memory vector mentioned earlier and the first textual feature in the channel dimension.i =|tm i Next, the similarity vector is fed into the trained fully connected network, which can specifically be a two-layer fully connected network, to obtain the selection vector of the memory vector for the first text feature channel dimension. Then, the average selection vector v is multiplied by the text vector t corresponding to the first text feature to obtain the latent space text feature t containing visual information. L This yields the second text feature in the latent space. The specific process and formula are shown below:

[0177] v i =LS(d i );

[0178]

[0179]

[0180] Where di is the i-th similarity vector, 1≤i≤K. K is the number of memory vectors. LS() uses a two-layer fully connected network for mapping processing.

[0181] Step 205: The computer device uses the video encoder in the similarity calculation model to perform video encoding on each candidate video in the candidate video library to obtain the first video feature of each candidate video.

[0182] For each candidate video in the candidate video library, it can be input into the video encoder in the similarity calculation model for encoding, that is, to extract video features in the latent space and obtain the first video feature of each candidate video.

[0183] Specifically, each candidate video can be first divided into multiple video frames, and then a pre-defined convolutional neural network can be used to extract the video frame features of each video frame. For a single candidate video, the number of extracted video frame features can be denoted as N. f These features can then be mapped into vectors v of appropriate dimensions. i (i = {1, ..., N} f In this embodiment of the application, a memory storage module M is designed in the model. This module consists of K learnable vectors M = {m1,...,m}. K These vectors are used to record important visual semantic information. In this embodiment, a memory storage module can be used to integrate consecutive frame features to obtain a global vector representing the entire video information, as shown below:

[0184]

[0185]

[0186]

[0187] Step 206: The computer device uses the symbol generator in the similarity calculation model to perform symbol space feature mapping on the first video features of each candidate video to obtain the second video features of each candidate video.

[0188] Furthermore, the first video feature in the latent space of each candidate video can be fed into the symbol generation module of the similarity calculation model, which generates the text annotation corresponding to each candidate video. Simultaneously, the visual features distributed in the latent space are projected onto the symbol space to obtain the second video feature v in the symbol space. S Specifically, this module can be built using a simple and effective LSTM-based text generator, as shown below:

[0189] h i =LSTM(W e ω i-1 v)

[0190]

[0191] Among them, W e These are learnable parameters, where L is the length of the predicted sentence, and w is the length of the predicted sentence. i-1 It is the embedding of the previously predicted word, h i It is the i-th hidden layer.

[0192] Step 207: The computer device calculates the similarity between the first text feature and the second video feature of each candidate video to obtain the first similarity between the video search text and each candidate video.

[0193] Furthermore, after obtaining the first text features of the video search text in the symbolic space and the second text features in the latent space, as well as the second video features of each candidate video in the symbolic space and the first video features in the latent space, the similarity between the video search text and each candidate video can be calculated in each space.

[0194] Specifically, the similarity between the video search text and each candidate video can be calculated in the symbolic space, that is, the similarity between the first text feature and the second video feature of each candidate video can be calculated to obtain the first similarity between the video search text and each candidate video in the symbolic space.

[0195] Step 208: The computer device calculates the similarity between the second text feature and the first video feature of each candidate video to obtain the second similarity between the video search text and each candidate video.

[0196] Then, we can further calculate the similarity between the second text feature of the video search text in the latent space and the first video feature of each candidate video in the latent space, and obtain the second similarity between the video search text and each candidate video in the latent space.

[0197] Step 209: The computer device calculates the mean of the two similarities between the video search text and each candidate video to obtain the target similarity between the video search text and each candidate video.

[0198] Furthermore, to more accurately represent the similarity between the video search text and each candidate video, the average of different spatial similarities can be used to determine the target similarity between the video search text and each candidate video. That is, the average of the first and second similarities between the video search text and each candidate video is calculated to obtain the target similarity between the video search text and each candidate video.

[0199] Step 210: The computer device determines the target candidate video related to the video search text based on the target similarity between the video search text and each candidate video.

[0200] Furthermore, one or more (e.g., a preset number) of candidate videos with the highest similarity to the target text in the video search can be identified as the searched target candidate videos. These target candidate videos can then be pushed to the corresponding terminals for display.

[0201] The video search method provided in this application, through in-depth exploration of the impact of information unique to video and text modalities on cross-modal retrieval tasks, proposes a video-text feature alignment method based on complementary space learning. It utilizes a simple and effective symbol generator and latent feature selector to map video and text to symbolic and latent spaces, enriching feature representation capabilities. Using only a single external feature to describe the video, without requiring additional data such as audio or optical flow features, this patent achieves retrieval results comparable to or even better than multi-feature methods with a smaller model and faster speed.

[0202] As described above, the video search method provided in this application obtains video search text and extracts text features from the video search text to obtain first text features; maps the first text features to the video feature space corresponding to the video information to obtain second text features; extracts video features from each candidate video in the candidate video library to obtain first video features for each candidate video; maps the first video features to the text feature space corresponding to the text information to obtain second video features for each candidate video; and searches for target candidate videos corresponding to the video search text in the candidate video library based on a first similarity relationship between the first text features and the second video features, and a second similarity relationship between the first video features and the second text features.

[0203] Therefore, the video search method provided in this application calculates the similarity of video features and text features separately in the video space and text space during video search, avoiding information loss caused by alignment in a single space. This significantly improves the accuracy of text-based video searches.

[0204] To better implement the above video search method, this application also provides a video search device that can be integrated into a terminal or server.

[0205] For example, such as Figure 5 The diagram shown is a structural schematic of a video search device provided in an embodiment of this application. The video search device may include an acquisition unit 301, a first mapping unit 302, an extraction unit 303, a second mapping unit 304, and a search unit 305, as follows:

[0206] The acquisition unit 301 is used to acquire video search text and extract text features from the video search text to obtain the first text feature;

[0207] The first mapping unit 302 is used to map the first text features to the video feature space corresponding to the video information to obtain the second text features;

[0208] The extraction unit 303 is used to extract video features from each candidate video in the candidate video library to obtain the first video feature of each candidate video.

[0209] The second mapping unit 304 is used to map the first video features to the text feature space corresponding to the text information to obtain the second video features of each candidate video.

[0210] Search unit 305 is used to search for target candidate videos corresponding to video search text in the candidate video library based on a first similarity relationship between the first text features and the second video features, and a second similarity relationship between the first video features and the second text features.

[0211] In some embodiments, the acquiring unit includes:

[0212] The first acquisition subunit is used to acquire video search text and use the trained first long short-term memory unit to extract text features from the video search text to obtain multiple transition text features.

[0213] The first calculation subunit is used to calculate the average value of multiple transition text features to obtain the first text feature.

[0214] In some embodiments, the first mapping unit includes:

[0215] The second acquisition subunit is used to acquire video memory features, which are features that record the visual semantics in video frames during video feature extraction.

[0216] The transfer subunit is used to transfer the first text features to the video feature space based on the video memory features to obtain the second text features.

[0217] In some embodiments, the migration subunit includes:

[0218] The first calculation module is used to calculate the Euclidean distance between the video memory features and the first text features to obtain similar features;

[0219] The mapping module is used to perform fully connected mapping on similar features based on the trained fully connected network to obtain selected features;

[0220] The second calculation module is used to calculate the second text feature based on the selected features and the first text feature.

[0221] In some embodiments, the second computing module includes:

[0222] The first calculation submodule is used to calculate the average value of the selection features to obtain the average selection features;

[0223] The second calculation submodule is used to calculate the product of the average selection feature and the first text feature to obtain the second text feature.

[0224] In some embodiments, the extraction unit includes:

[0225] The first extraction subunit is used to extract video frame features from the video frames of each candidate video in the candidate video library to obtain the video frame features corresponding to each candidate video.

[0226] The processing subunit is used to perform attention processing on the video frame features corresponding to each candidate video using video memory features to obtain the first video feature of each candidate video.

[0227] In some embodiments, extracting a subunit includes:

[0228] The segmentation module is used to divide each candidate video in the candidate video library into multiple video frames.

[0229] The processing module is used to perform convolution processing on multiple video frames corresponding to each candidate video using a preset neural network model to obtain the video frame features corresponding to each candidate video.

[0230] In some embodiments, the second mapping unit includes:

[0231] The first mapping subunit is used to map the first video features of each candidate video based on the trained second long short-term memory unit to obtain multiple hidden features of each candidate video.

[0232] The second computational subunit is used to calculate the second video features of each candidate video based on multiple hidden features of each candidate video and the corresponding predicted sentence length.

[0233] In some embodiments, the video search method provided in this application further includes:

[0234] The third acquisition subunit is used to acquire training sample data, which includes sample video data and sample text data corresponding to the sample video data.

[0235] The second extraction subunit is used to extract video frame features from the sample video data, and to perform attention processing on the extracted video frame features using the video memory features to be trained, so as to obtain the first sample video features of the sample video data.

[0236] The second mapping subunit is used to input the first sample video features into the first long short-term memory unit to be trained for mapping processing, and to calculate the second sample video features based on the output hidden features.

[0237] The second mapping subunit is used to input the sample text data into the second long short-term memory unit to be trained for mapping processing, and to calculate the mean of the output features to obtain the first sample text features.

[0238] The third computational subunit is used to input the Euclidean distance between the first sample text features and the video memory features to be trained into the fully connected network to be trained, and to calculate the second sample text features based on the output features of the fully connected network to be trained and the first sample text features.

[0239] The training subunit is used to train the video memory features to be trained, the first long short-term memory unit, the second long short-term memory unit, and the fully connected network based on the first sample video features, the second sample video features, the first sample text features, and the second sample text features.

[0240] In some embodiments, the training subunit includes:

[0241] The third calculation module is used to calculate the first sample similarity between the first sample video features and the second sample text features, and to calculate the second sample similarity between the second sample video features and the first sample text features.

[0242] The fourth calculation module is used to calculate the mean of the similarity of the first sample and the similarity of the second sample to obtain the similarity of the target sample.

[0243] The fifth calculation module is used to calculate the first loss function value based on the similarity of the target samples. The first loss function value represents the contrast loss.

[0244] The acquisition module is used to acquire the value of the second loss function that constrains the second long short-term memory unit;

[0245] The adjustment module is used to perform gradient backpropagation optimization based on the first loss function value and the second loss function value, so as to adjust the video memory features to be trained, the first long short-term memory unit, the second long short-term memory unit, and the fully connected network.

[0246] In some embodiments, the search unit includes:

[0247] The fourth calculation subunit is used to calculate the similarity between the first text feature and the second video feature to obtain the first similarity.

[0248] The fifth calculation subunit is used to calculate the similarity between the second text feature and the first video feature to obtain the second similarity.

[0249] The sixth calculation subunit is used to calculate the mean of the first similarity and the second similarity to obtain the target similarity between the video search text and each candidate video;

[0250] The determination subunit is used to identify the target candidate video corresponding to the video search text in the candidate video library based on the target similarity.

[0251] In practice, each of the above units can be implemented as an independent entity or can be arbitrarily combined to be implemented as the same or several entities. For the specific implementation of each of the above units, please refer to the previous method embodiments, which will not be repeated here.

[0252] As described above, the video search device provided in this application embodiment acquires video search text through acquisition unit 301 and extracts text features from the video search text to obtain first text features; first mapping unit 302 maps the first text features to the video feature space corresponding to the video information to obtain second text features; extraction unit 303 extracts video features from each candidate video in the candidate video library to obtain first video features for each candidate video; second mapping unit 304 maps the first video features to the text feature space corresponding to the text information to obtain second video features for each candidate video; and search unit 305 searches for target candidate videos corresponding to the video search text in the candidate video library based on the first similarity relationship between the first text features and the second video features, and the second similarity relationship between the first video features and the second text features.

[0253] Therefore, the video search method provided in this application calculates the similarity of video features and text features separately in the video space and text space during video search, avoiding information loss caused by alignment in a single space. This significantly improves the accuracy of text-based video searches.

[0254] This application also provides a computer device, which can be a terminal or a server, such as... Figure 6 The diagram shown is a structural schematic of the computer device provided in this application. Specifically:

[0255] The computer device may include components such as a processing unit 401 with one or more processing cores, a storage unit 402 with one or more storage media, a power module 403, and an input module 404. Those skilled in the art will understand that... Figure 6 The computer device structure shown does not constitute a limitation on the computer device and may include more or fewer components than shown, or combine certain components, or have different component arrangements. Wherein:

[0256] The processing unit 401 is the control center of the computer device. It connects various parts of the computer device via various interfaces and lines, and performs various functions and processes data by running or executing software programs and / or modules stored in the storage unit 402, and by calling data stored in the storage unit 402. Optionally, the processing unit 401 may include one or more processing cores; preferably, the processing unit 401 may integrate an application processor and a modem processor, wherein the application processor mainly handles the operating system, user interface, and applications, and the modem processor mainly handles wireless communication. It is understood that the modem processor may not be integrated into the processing unit 401.

[0257] Storage unit 402 can be used to store software programs and modules. Processing unit 401 executes various functional applications and data processing by running the software programs and modules stored in storage unit 402. Storage unit 402 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function (such as sound playback, image playback, and web page access); the data storage area may store data created based on the use of the computer device. In addition, storage unit 402 may include high-speed random access memory and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device. Accordingly, storage unit 402 may also include a memory controller to provide processing unit 401 with access to storage unit 402.

[0258] The computer equipment also includes a power supply module 403 that supplies power to various components. Preferably, the power supply module 403 can be logically connected to the processing unit 401 through a power management system, thereby enabling functions such as charging, discharging, and power consumption management through the power management system. The power supply module 403 may also include one or more DC or AC power supplies, recharging systems, power fault detection circuits, power converters or inverters, power status indicators, and other arbitrary components.

[0259] The computer device may also include an input module 404, which can be used to receive input numeric or character information and generate keyboard, mouse, joystick, optical or trackball signal inputs related to object settings and function control.

[0260] Although not shown, the computer device may also include a display unit, etc., which will not be described in detail here. Specifically, in this embodiment, the processing unit 401 in the computer device loads the executable files corresponding to the processes of one or more applications into the storage unit 402 according to the following instructions, and the processing unit 401 runs the applications stored in the storage unit 402 to realize various functions, as follows:

[0261] The process involves: acquiring video search text and extracting text features from it to obtain first text features; mapping the first text features to the video feature space corresponding to the video information to obtain second text features; extracting video features from each candidate video in the candidate video library to obtain first video features for each candidate video; mapping the first video features to the text feature space corresponding to the text information to obtain second video features for each candidate video; and searching for target candidate videos corresponding to the video search text in the candidate video library based on the first similarity relationship between the first text features and the second video features, as well as the second similarity relationship between the first video features and the second text features.

[0262] It should be noted that the computer device provided in this application embodiment and the method in the above embodiment belong to the same concept. The specific implementation of each of the above operations can be found in the previous embodiments, and will not be repeated here.

[0263] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be performed by instructions, or by instructions controlling related hardware. These instructions can be stored in a computer-readable storage medium and loaded and executed by a processor.

[0264] Therefore, embodiments of the present invention provide a computer-readable storage medium storing a plurality of instructions that can be loaded by a processor to execute steps in any of the methods provided in the embodiments of the present invention. For example, the instructions can execute the following steps:

[0265] The process involves: acquiring video search text and extracting text features from it to obtain first text features; mapping the first text features to the video feature space corresponding to the video information to obtain second text features; extracting video features from each candidate video in the candidate video library to obtain first video features for each candidate video; mapping the first video features to the text feature space corresponding to the text information to obtain second video features for each candidate video; and searching for target candidate videos corresponding to the video search text in the candidate video library based on the first similarity relationship between the first text features and the second video features, as well as the second similarity relationship between the first video features and the second text features.

[0266] For details on the implementation of each of the above operations, please refer to the previous examples, which will not be repeated here.

[0267] The computer-readable storage medium may include: read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0268] Since the instructions stored in the computer-readable storage medium can execute the steps of any of the methods provided in the embodiments of the present invention, the beneficial effects that any of the methods provided in the embodiments of the present invention can achieve can be realized, as detailed in the preceding embodiments, and will not be repeated here.

[0269] According to one aspect of this application, a computer program product or computer program is provided, comprising computer instructions stored in a storage medium. A processor of a computer device reads the computer instructions from the storage medium and executes the computer instructions, causing the computer device to perform the methods provided in various optional implementations of the video search method described above.

[0270] The video search method, apparatus, and computer-readable storage medium provided in the embodiments of the present invention have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A video search method, characterized in that, The method includes: Obtain the video search text and extract text features from the video search text to obtain the first text feature; Mapping the first text features to the video feature space corresponding to the video information to obtain the second text features includes: acquiring video memory features, which are features learned during the training of the video-text similarity calculation model that record the visual semantics in video frames during video feature extraction; calculating the Euclidean distance between the video memory features and the first text features to obtain similar features; performing a fully connected mapping on the similar features based on the trained fully connected network to obtain selected features; and calculating the second text features based on the selected features and the first text features. For each candidate video in the candidate video library, video features are extracted to obtain the first video feature of each candidate video. This includes: extracting video frame features from the video frames of each candidate video in the candidate video library to obtain the video frame features corresponding to each candidate video; and performing attention processing on the video frame features corresponding to each candidate video based on the video memory features to obtain the first video feature of each candidate video. Mapping the first video features to the text feature space corresponding to the text information to obtain the second video features of each candidate video includes: mapping the first video features based on the trained second long short-term memory unit to obtain multiple hidden features of each candidate video; calculating the second video features of each candidate video based on the multiple hidden features of each candidate video and the corresponding predicted sentence length, wherein the first video features contain the global features of the candidate video. Based on the first similarity relationship between the first text feature and the second video feature, and the second similarity relationship between the first video feature and the second text feature, the system searches for target candidate videos in the candidate video library that correspond to the video search text.

2. The method according to claim 1, characterized in that, The process of acquiring video search text and extracting text features from the video search text to obtain the first text feature includes: The video search text is obtained, and the first long short-term memory unit after training is used to extract text features from the video search text to obtain multiple transition text features; The average value of the multiple transition text features is calculated to obtain the first text feature.

3. The method according to claim 1, characterized in that, The step of calculating the second text feature based on the selected feature and the first text feature includes: Calculate the average value of the selection features to obtain the average selection features; The second text feature is obtained by multiplying the average selection feature by the first text feature.

4. The method according to claim 1, characterized in that, The step of extracting video frame features from each candidate video in the candidate video library to obtain the video frame features corresponding to each candidate video includes: Each candidate video in the candidate video library is divided into multiple video frames; A pre-defined neural network model is used to perform convolution processing on multiple video frames corresponding to each candidate video to obtain the video frame features corresponding to each candidate video.

5. The method according to any one of claims 2 to 4, characterized in that, The step of acquiring video memory features, which are features recorded in the visual semantics of video frames during video feature extraction, further includes: Acquire training sample data, which includes sample video data and sample text data corresponding to the sample video data; Video frame features are extracted from the sample video data, and attention processing is performed on the extracted video frame features using the video memory features to be trained, to obtain the first sample video features of the sample video data. The first sample video features are input into the first long short-term memory unit to be trained for mapping processing, and the second sample video features are calculated based on the output hidden features. The sample text data is input into the second long short-term memory unit to be trained for mapping processing, and the mean of the output features is calculated to obtain the first sample text features. The Euclidean distance between the first sample text features and the video memory features to be trained is input into the fully connected network to be trained, and the second sample text features are calculated based on the output features of the fully connected network to be trained and the first sample text features. The video memory features to be trained, the first long short-term memory unit, the second long short-term memory unit, and the fully connected network are trained based on the first sample video features, the second sample video features, the first sample text features, and the second sample text features.

6. The method according to claim 5, characterized in that, The step of training the video memory features to be trained, the first long short-term memory unit, the second long short-term memory unit, and the fully connected network based on the first sample video features, the second sample video features, the first sample text features, and the second sample text features includes: Calculate the first sample similarity between the first sample video features and the second sample text features, and calculate the second sample similarity between the second sample video features and the first sample text features; Calculate the mean of the similarity between the first sample and the similarity between the second sample to obtain the similarity of the target sample; A first loss function value is calculated based on the similarity of the target samples, and the first loss function value represents the contrast loss; Obtain the value of the second loss function that constrains the second long short-term memory unit; Gradient backpropagation optimization is performed based on the first loss function value and the second loss function value to adjust the video memory features to be trained, the first long short-term memory unit, the second long short-term memory unit, and the fully connected network.

7. The method according to claim 1, characterized in that, The step of searching for target candidate videos corresponding to the video search text in the candidate video library based on the first similarity relationship between the first text features and the second video features, and the second similarity relationship between the first video features and the second text features, includes: Calculate the similarity between the first text feature and the second video feature to obtain the first similarity; Calculate the similarity between the second text feature and the first video feature to obtain the second similarity; Calculate the mean of the first similarity and the second similarity to obtain the target similarity between the video search text and each candidate video; Based on the target similarity, a target candidate video corresponding to the video search text is determined in the candidate video library.

8. A video search device, characterized in that, The device includes: The acquisition unit is used to acquire video search text and extract text features from the video search text to obtain the first text feature; The first mapping unit is used to map the first text features to the video feature space corresponding to the video information to obtain the second text features; The first mapping unit includes: The second acquisition subunit is used to acquire video memory features, which are features learned during the training of the video-text similarity calculation model that record the visual semantics in the video frame during video feature extraction. The transfer subunit is used to transfer the first text feature to the video feature space based on the video memory feature to obtain the second text feature; The migration subunit includes: A first computer module is used to calculate the Euclidean distance between the video memory features and the first text features to obtain similar features; The mapping module is used to perform fully connected mapping on the similar features based on the trained fully connected network to obtain selected features; The second calculation module is used to calculate the second text feature based on the selected feature and the first text feature; The extraction unit is used to extract video features from each candidate video in the candidate video library to obtain the first video feature of each candidate video. The extraction unit includes: The first extraction subunit is used to extract video frame features from the video frames of each candidate video in the candidate video library to obtain the video frame features corresponding to each candidate video. The processing unit is used to perform attention processing on the video frame features corresponding to each candidate video based on the video memory features to obtain the first video feature of each candidate video. The second mapping unit is used to map the first video features to the text feature space corresponding to the text information to obtain the second video features of each candidate video. The second mapping unit includes: The first mapping subunit is used to map the first video features based on the trained second long short-term memory unit to obtain multiple hidden features for each candidate video. The second calculation subunit is used to calculate the second video feature of each candidate video based on multiple hidden features of each candidate video and the corresponding predicted sentence length, wherein the first video feature includes the global features of the candidate video. The search unit is used to search for target candidate videos corresponding to the video search text in the candidate video library based on a first similarity relationship between the first text features and the second video features, and a second similarity relationship between the first video features and the second text features.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a plurality of instructions adapted for loading by a processor to perform the steps of the video search method according to any one of claims 1 to 7.

10. A computer device, characterized in that, It includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the video search method according to any one of claims 1 to 7.

11. A computer program product, comprising a computer program / instructions, characterized in that, When the computer program / instructions are executed by the processor, they implement the steps of the video search method according to any one of claims 1 to 7.

Images