Model training method and device, video search method and device, equipment and medium

Abstract

The embodiment of the application provides a model training method and device, a video search method and device, equipment and a medium, and belongs to the technical field of artificial intelligence. The method comprises the following steps: acquiring a training video dataset; wherein the training video dataset comprises at least two training video data; performing feature extraction on each training video data to obtain at least two video frame feature sequences; performing index conversion on the at least two video frame feature sequences to obtain at least two video index sequences; performing classification processing on the at least two training video data according to the at least two video index sequences to obtain a video identifier; and training a preset self-supervised learning model according to the video identifier and the training video dataset to obtain a data conversion model. The embodiment of the application can construct a data conversion model with higher precision.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a model training method and apparatus, a video search method and apparatus, equipment and medium. Background Technology

[0002] Retrieving video content using image or text modal data requires converting the video data into binary data or low-dimensional data that matches the retrieval data. In related technologies, the conversion of video data into binary data primarily employs self-supervised learning to construct a data conversion model. However, discrete optimization problems exist during model training, leading to a decrease in the accuracy of the constructed data conversion model. Summary of the Invention

[0003] The main objective of this application is to propose a model training method and apparatus, a video search method and apparatus, a device and medium, which aim to improve the accuracy of data conversion model construction.

[0004] To achieve the above objectives, a first aspect of this application proposes a model training method, the method comprising:

[0005] Obtain a training video dataset; wherein the training video dataset includes at least two training video datasets;

[0006] Feature extraction is performed on each of the training video data to obtain at least two video frame feature sequences;

[0007] The feature sequences of the at least two video frames are indexed to obtain at least two video index sequences.

[0008] The at least two training video data are classified according to the at least two video index sequences to obtain video identifiers;

[0009] The preset self-supervised learning model is trained based on the video identifier and the training video dataset to obtain the data transformation model.

[0010] In some embodiments, each training video data includes at least two video frames, and the step of extracting features from each training video data to obtain feature sequences of at least two video frames includes:

[0011] Feature extraction is performed on each video frame to obtain video frame features;

[0012] Each video frame feature in the training video data is transformed into a feature sequence to obtain at least two video frame feature sequences.

[0013] In some embodiments, the step of extracting features from each of the video frames to obtain video frame features includes:

[0014] Each video frame is encoded using a global hierarchical encoder to obtain global video-level features;

[0015] The global video-level features are encoded using a backward hierarchical encoder to obtain inverse video frame features;

[0016] The inverse video frame features are decoded using a bidirectional hierarchical decoder to obtain frame-level features;

[0017] The frame-level features are encoded by a forward hierarchical encoder to obtain the video frame features.

[0018] In some embodiments, the bidirectional hierarchical decoder includes: a unidirectional convolutional layer and a bidirectional convolutional layer. The step of decoding the inverse video frame features using the bidirectional hierarchical decoder to obtain frame-level features includes:

[0019] The inverse video frame features are convolved using the unidirectional convolutional layer to obtain convolutional data;

[0020] The frame-level features are obtained by performing convolution processing on the convolutional data through the bidirectional convolutional layer.

[0021] In some embodiments, the step of indexing the at least two video frame feature sequences to obtain at least two video index sequences includes:

[0022] The feature sequences of at least two video frames are subjected to mean pooling to obtain video-level information.

[0023] The neighborhood structure is obtained by performing neighborhood calculation on the video-level information according to the preset neighborhood function.

[0024] The at least two video frame feature sequences are indexed and transformed according to the neighborhood structure to obtain the at least two video index sequences.

[0025] To achieve the above objectives, a second aspect of this application proposes a video search method, the method comprising:

[0026] Obtain raw video data;

[0027] The original video data is input into a data conversion model; wherein the data conversion model is obtained by the model training method of the first aspect;

[0028] The original video data is converted using the data conversion model to obtain video sequence data;

[0029] Retrieve query information;

[0030] Target sequence data is selected from at least two video sequence data based on the query information;

[0031] Target video data is selected from at least two of the original video data based on the target sequence data.

[0032] In some embodiments, filtering target sequence data from at least two video sequence data based on the query information includes:

[0033] The query information is converted into binary to obtain a query sequence;

[0034] The target sequence data is selected from at least two video sequence data based on the query sequence.

[0035] To achieve the above objectives, a third aspect of this application provides a model training apparatus, the apparatus comprising:

[0036] A data acquisition module is used to acquire a training video dataset; wherein the training video dataset includes at least two training video datasets;

[0037] An extraction module is used to extract features from each of the training video data to obtain at least two video frame feature sequences.

[0038] An index conversion module is used to perform index conversion on the at least two video frame feature sequences to obtain at least two video index sequences;

[0039] A classification module is used to classify the at least two training video data according to the at least two video index sequences to obtain video identifiers;

[0040] The training module is used to train a preset self-supervised learning model based on the video identifier and the training video dataset to obtain a data transformation model.

[0041] To achieve the above objectives, a fourth aspect of the present application provides an electronic device, the electronic device including a memory, a processor, a program stored in the memory and executable on the processor, and a data bus for implementing connection communication between the processor and the memory, wherein the program, when executed by the processor, implements the method described in the first aspect or the method described in the second aspect.

[0042] To achieve the above objectives, a fifth aspect of the present application provides a storage medium, which is a computer-readable storage medium for computer-readable storage, wherein the storage medium stores one or more programs that can be executed by one or more processors to implement the method described in the first aspect or the method described in the second aspect.

[0043] The model training method and apparatus, video search method and apparatus, device and medium proposed in this application obtain video identifiers based on the feature sequence of video frames. Then, a pre-defined self-supervised learning model is trained based on the video identifiers and the training video dataset. The trained data conversion model can convert similar video data into similarity hash codes and different video data into different hash codes. Therefore, a more accurate data conversion model is constructed. Attached Figure Description

[0044] Figure 1 This is a flowchart of the model training method provided in the embodiments of this application;

[0045] Figure 2 yes Figure 1 The flowchart of step S102 in the document;

[0046] Figure 3 yes Figure 2 The flowchart of step S201 in the text;

[0047] Figure 4 yes Figure 3 The flowchart of step S303 in the process;

[0048] Figure 5 yes Figure 1 The flowchart of step S103 in the process;

[0049] Figure 6 This is a flowchart of the video search method provided in the embodiments of this application;

[0050] Figure 7 yes Figure 6 The flowchart of step S605 in the process;

[0051] Figure 8 This is a schematic diagram of the structure of the model training device provided in the embodiments of this application;

[0052] Figure 9 This is a schematic diagram of the structure of the video search device provided in the embodiments of this application;

[0053] Figure 10 This is a schematic diagram of the hardware structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0054] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0055] It should be noted that although functional modules are divided in the device schematic diagram and a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first," "second," etc., in the specification, claims, and the aforementioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0056] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0057] First, let's analyze some of the terms used in this application:

[0058] Artificial intelligence (AI) is a new branch of computer science that studies, develops, and applies theories, methods, technologies, and systems to simulate, extend, and expand human intelligence. It aims to understand the essence of intelligence and produce intelligent machines that can react in a way similar to human intelligence. Research in this field includes robotics, speech recognition, image recognition, natural language processing, and expert systems. AI can simulate the information processes of human consciousness and thought. Furthermore, AI utilizes digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceiving the environment, acquiring knowledge, and using that knowledge to achieve optimal results.

[0059] Autoencoders are unsupervised learning models. Based on backpropagation and optimization methods (such as gradient descent), they use the input data X itself as supervision to guide the neural network in attempting to learn a mapping relationship, thereby obtaining a reconstructed output X. R In time series anomaly detection scenarios, anomalies are rare compared to normal occurrences. Therefore, we believe that if the output X reconstructed using an autoencoder... RIf the difference from the original input exceeds a certain threshold, the original time series is considered to have an anomaly. The algorithm model consists of two main parts: an encoder and a decoder. The encoder's role is to encode the high-dimensional input X into low-dimensional latent variables h, thus forcing the neural network to learn the most informative features. The decoder's role is to restore the latent variables h from the hidden layers to their initial dimensions. Ideally, the decoder's output should perfectly or approximately recover the original input, i.e., X. R ≈X.

[0060] Cross-modal retrieval involves using one type of data as a query to retrieve another related type of data. The three main modalities are natural language (written and spoken language, etc.), visual signals (images and videos, etc.), and audio signals (sound encoding and prosody, etc.). Methodologically, cross-modal retrieval falls into two main categories: real-valued representation learning and binary representation learning (also known as cross-modal hashing). Real-valued representation learning directly learns features extracted from different modalities; while binary representation learning maps features extracted from different modalities to a Hamming binary space and then learns from that space.

[0061] Hash code: A hash code is not completely unique. It is an algorithm that makes objects of the same class have different hash codes as much as possible according to their different characteristics, but it does not mean that different objects have completely different hash codes.

[0062] Neighborhood: A neighborhood is a fundamental topological structure on a set. In set theory, it is any open interval centered at a point a, denoted as U(a). In topology and related mathematical fields, the neighborhood is a basic concept in topological spaces. There are works on related research such as the neighborhood axiom (a fundamental concept in modern mathematical topology), open and closed neighborhoods, and centered neighborhoods.

[0063] Long Short-Term Memory (LSTM) networks are a type of recurrent neural network (RNN) primarily designed to address the vanishing and exploding gradient problems during long sequence training. An LSTM is a neural network containing LSTM blocks or other similar units. In literature and other materials, LSTM blocks may be described as intelligent network units because they can memorize values ​​of indefinite duration. Each block contains a gate that determines whether the input is important enough to be remembered and whether it can be output.

[0064] Mean pooling: Mean pooling is the process of averaging all values ​​in a local receptive field. Commonly used pooling methods include max pooling and mean pooling. According to relevant theories, the error in feature extraction mainly comes from two aspects: (1) the increased variance of the estimated value due to the limited neighborhood size; and (2) the shift in the estimated mean caused by the error of the convolutional layer parameters.

[0065] With the development of short videos, efficient video data retrieval has become a key technical challenge in short video search, leading to widespread research in cross-modal retrieval. For example, query data might be in both image and text modalities, with the goal of retrieving matching video content based on these modalities. However, since video data differs from image and text modalities, and these heterogeneous data reside in different feature spaces, retrieving video data based solely on image or text modal data is unlikely to yield accurate results. To eliminate the heterogeneity of different modalities, traditional methods involve learning all data within a single space to convert multimedia stream data into binary hash streams for matching with other low-dimensional data. Related technologies employ self-supervised learning to train data relevance and calculate the mapping from multimodal data to binary data. A pre-defined label set is used to perform discrete optimization of the binary encoding during learning. However, since the pre-defined label set is learned autonomously by the self-supervised learning model, in practical applications, multimodal data with different labels often have similar binary encodings, affecting the accuracy of converting video data into binary video sequences.

[0066] Based on this, embodiments of this application provide a model training method and apparatus, a video search method and apparatus, a device, and a medium. By extracting features from training video data to obtain a video frame feature sequence, and then performing indexing transformation on the video frame feature sequence to obtain a video index sequence, the training video data is classified according to the video index sequence to obtain video identifiers. A preset self-supervised learning model is then trained using the video identifiers and the training video dataset to obtain a data conversion model, making model training more accurate. Furthermore, the data conversion model converts similar video data to obtain similar hash codes, while different video data have different hash codes, thereby improving the accuracy of video data conversion and making video search more accurate.

[0067] The model training method and apparatus, video search method and apparatus, device and medium provided in the embodiments of this application are specifically described through the following embodiments. First, the model training method in the embodiments of this application is described.

[0068] The embodiments of this application can acquire and process relevant data based on artificial intelligence technology. Artificial intelligence (AI) refers to the theories, methods, technologies, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results.

[0069] Foundational technologies for artificial intelligence generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies mainly encompass computer vision, robotics, biometrics, speech processing, natural language processing, and machine learning / deep learning.

[0070] The model training method provided in this application relates to the field of artificial intelligence technology. The model training method provided in this application can be applied to a terminal, a server, or software running on either a terminal or a server. In some embodiments, the terminal can be a smartphone, tablet, laptop, desktop computer, etc.; the server can be configured as an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application implementing the model training method, but is not limited to the above forms.

[0071] This application can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0072] It should be noted that in all specific embodiments of this application, when processing data related to user identity or characteristics, such as user information, user behavior data, user historical data, and user location information, user permission or consent is obtained first. Furthermore, the collection, use, and processing of this data comply with relevant laws, regulations, and standards of the relevant countries and regions. In addition, when embodiments of this application require access to sensitive personal information of users, separate permission or consent from the user is obtained through pop-ups or redirects to confirmation pages. Only after obtaining the user's separate permission or consent is the necessary user-related data for the proper functioning of the embodiments of this application obtained.

[0073] Figure 1 This is an optional flowchart of the model training method provided in the embodiments of this application. Figure 1 The method may include, but is not limited to, steps S101 to S105.

[0074] Step S101: Obtain the training video dataset; wherein the training video dataset includes at least two training video datasets.

[0075] Step S102: Extract features from each training video data to obtain at least two video frame feature sequences;

[0076] Step S103: Perform index transformation on at least two video frame feature sequences to obtain at least two video index sequences;

[0077] Step S104: Classify at least two training video data according to at least two video index sequences to obtain video identifiers;

[0078] Step S105: Train the preset self-supervised learning model based on the video identifier and training video dataset to obtain the data transformation model.

[0079] Steps S101 to S105 of this embodiment involve acquiring at least two training video datasets, extracting features from each training video dataset to obtain at least two video frame feature sequences, and then performing index transformation on these at least two video index sequences. These video index sequences are then used to classify the at least two training video datasets to obtain video identifiers. Since the video identifiers are obtained from the video frame feature sequences, a pre-defined self-supervised learning model is trained using the video identifiers and the training video dataset. The trained data transformation model can convert similar video data into similarity hash codes, while different video data are converted into different hash codes. Finally, video search is performed based on these hash codes to obtain more accurate video data.

[0080] In step S101 of some embodiments, training video data needs to be acquired before training the training video dataset, and at least two training video datasets are aggregated into a training video dataset. Wherein, if the training video data is acquired through a pre-defined third-party internet platform, and the acquired training video data covers various fields, then model training based on the training video dataset will be more accurate.

[0081] In step S102 of some embodiments, the training video data includes multiple video frames. Therefore, feature extraction is performed on each training video data, that is, feature extraction is performed on the video frame. The features extracted from multiple video frames are used to obtain a video frame feature sequence.

[0082] In step S103 of some embodiments, after obtaining the video frame feature sequence of each training video data, an index transformation is performed on each video frame feature sequence to obtain a video index sequence. The index transformation of the video frame feature sequence is essentially vectorizing the video frame features to obtain a feature vector. Each index in the video index sequence corresponds to a feature vector of a video frame feature. Each index in the video index sequence consists of at least two feature elements and their values. If the training video dataset has 256 feature elements, then the feature vector corresponds to 256 feature elements. Each feature element has its own index and a corresponding value. The value of each feature element represents the feature of that video frame, and the presence of feature elements in the video frame is indicated by 0s and 1s in the feature vector. Therefore, the video frame features corresponding to each video frame in the training video data can be determined through the video index sequence, thus allowing the determination of the features of the training video data.

[0083] It should be noted that if the video index sequence is {vi,...,vk}, and vi includes a feature vector representing the i-th video frame, and the feature vector vi = {t1,...,tn}, where n is the number of feature elements, if t2 and t7 in vi have values ​​of 1, and the rest have values ​​of 0, then the video frame feature of the i-th video frame is the feature elements t2 and t7. If t2 represents a person's features, and t7 represents an age feature between 20 and 30 years old, then the video frame feature corresponding to that video frame is that the person's age is between 20 and 30 years old. Therefore, the features of each video frame in the training video data can be determined through the video index sequence, making feature analysis of the training video data much simpler.

[0084] In step S104 of some embodiments, at least two training video data are classified according to at least two video index sequences. That is, the features of video frames in each training video data are clear according to the video index sequences. At least two training video data are classified according to at least two video index sequences to reasonably classify at least two training video data. A corresponding video identifier is constructed according to the size of the classified training video data. The features of each training video data can be determined according to the video identifier, making the association between similar training video data closer.

[0085] In step S105 of some embodiments, the self-supervised learning model performs self-supervised learning on the training video data to construct representations by learning the similarity or dissimilarity of two training video data sets for encoding. Adjacent features in video frames are similar, while video frames with features far apart are dissimilar. Therefore, video identifiers are obtained by classifying the training video data using video index sequences. The self-supervised learning model is then trained based on these video identifiers and the training video dataset, making the data transformation model trained by the self-supervised learning model more accurate in data transformation. Thus, the resulting data transformation model can convert similar video data into similar hash codes and different video data into different hash codes, making video data retrieval more accurate when searching based on hash codes.

[0086] It's important to note that self-supervised learning models use training video datasets to construct pseudo-labels and convert the training video data into hash codes based on these pseudo-labels. However, training video data with different pseudo-labels will produce similar hash codes, leading to errors when searching for video data using hash codes. A better approach is to pre-extract features from the training video data to obtain video frame feature sequences, then index these sequences to obtain video index sequences. These video index sequences are then used to classify the training video data to obtain a video identifier for each training video. The self-supervised learning model is then trained using these video identifiers and the training video dataset to build a data transformation model. Because the training video data is pre-classified based on its features, similar training video data within the dataset are converted into similar hash codes, while different training video data will still have different hash codes. Therefore, searching for video data using hash codes is more accurate.

[0087] Please see Figure 2 In some embodiments, each training video data includes at least two video frames, and step S102 may include, but is not limited to, steps S201 to S202:

[0088] Step S201: Extract features from each video frame to obtain video frame features;

[0089] Step S202: Perform feature sequence transformation on the features of each video frame of the training video data to obtain at least two video frame feature sequences.

[0090] In step S201 of some embodiments, since each training video data includes at least two video frames, when performing feature extraction on the training video data, the training video data is first decomposed to obtain at least two video frames, and feature extraction is performed on each video frame to obtain video frame features.

[0091] In step S202 of some embodiments, after obtaining the video frame features of each video frame, feature sequence transformation is performed on the video frame features of the training video data to aggregate each video frame feature into a video frame feature sequence according to the order of the video frames, and the video frame features of each video frame are determined through the video frame feature sequence. The video frame features extracted for each video frame are different. Therefore, the position of each video frame feature in the video frame feature sequence is determined according to the time order of the video frames, that is, the sequence number is determined, and the corresponding video frame feature is input into the corresponding sequence number position to construct the video frame feature sequence. This makes the construction of the video frame feature sequence simpler, and the feature changes of the training video data can be known through the video frame feature sequence.

[0092] Please see Figure 3In some embodiments, step S201 may include, but is not limited to, steps S301 to S304:

[0093] Step S301: Encode each video frame using a global hierarchical encoder to obtain global video-level features;

[0094] Step S302: Encode the global video-level features using a backward hierarchical encoder to obtain the inverse video frame features;

[0095] Step S303: The inverse video frame features are decoded using a bidirectional hierarchical decoder to obtain frame-level features;

[0096] Step S304: The frame-level features are encoded using a forward hierarchical encoder to obtain video frame features.

[0097] It should be noted that video frame features are obtained by inputting video frames into the encoder and decoder for feature extraction. To transform the training video data into a form that is easier for the computer to process, preprocessing is necessary. The preprocessed training video data should retain as much information as possible from the original training video data, ensuring that the final encoded data can reconstruct the original training video data to the greatest extent possible. The encoder is an autoencoder, which includes a global hierarchical encoder, a backward hierarchical encoder, and a forward hierarchical encoder. The decoder includes a bidirectional hierarchical decoder. The global hierarchical decoder encodes the video frames, that is, it encodes the high-dimensional input video frames into low-dimensional global video-level features. The backward hierarchical encoder encodes the global video-level features to obtain inverse video frame features. The bidirectional hierarchical decoder then decodes the inverse video frame features to reconstruct frame-level features. Finally, the frame-level features are input into the forward hierarchical encoder for encoding to obtain the video frame features. Therefore, by encoding and decoding video frames using a global hierarchical encoder, a backward hierarchical encoder, a forward hierarchical encoder, and a bidirectional hierarchical decoder, and then encoding the decoded frame-level features to obtain video frame features, the feature extraction of video frames becomes more accurate, thereby optimizing the feature extraction of training video data.

[0098] In step S301 of some embodiments, in this embodiment, the global hierarchical encoder includes two global convolutional layers, and the global convolutional layers are long short-term memory networks. By inputting video frames into the global convolutional layers for encoding processing, global video-level features are obtained, and the global video-level features are Vg.

[0099] In step S302 of some embodiments, the backward layered decoder includes backward convolutional layers, which are long short-term memory networks. By inputting global video-level features into the backward convolutional layers, and setting the number of backward convolutional layers to match the number of video frames, with each video frame inputting into a corresponding backward convolutional layer, each backward convolutional layer outputs an inverse video frame feature. Therefore, the output of multiple backward convolutional layers from the training video data input is an inverse video frame feature sequence. Let the inverse video frame feature sequence be (Vr1, Vr2, ..., Vrm).

[0100] In step S303 of some embodiments, the inverse video frame features are input into a bidirectional hierarchical decoder, and the bidirectional hierarchical decoder reconstructs video-level features based on the inverse video frame features to obtain frame-level features. Specifically, if the inverse video frame feature sequence is input to the bidirectional hierarchical decoder for sequential decoding, the resulting frame-level feature sequence is (V1, V2, ..., Vm).

[0101] In step S304 of some embodiments, the forward hierarchical decoder includes forward convolutional layers, and the forward convolutional layers are long short-term memory networks. The number of forward convolutional layers is consistent with the number of video frames. By inputting each video frame into the corresponding forward convolutional layer, one forward convolutional layer outputs one video frame feature. If the video frames in the training video data are input into the forward convolutional layers respectively, the video frame features are output by multiple forward convolutional layers, and the multiple video frame features are aggregated to obtain a video frame feature sequence, let the video frame feature sequence be (Vf1, Vf2, ..., Vfm-1, Vfm). Therefore, by encoding and decoding the video frames through a global hierarchical encoder, a backward hierarchical encoder, a bidirectional hierarchical encoder, and a forward hierarchical encoder to obtain video frame features, the feature extraction of the video frames is more accurate.

[0102] Please see Figure 4 In some embodiments, the bidirectional hierarchical decoder includes: a unidirectional convolutional layer and a bidirectional convolutional layer, and step S303 may include, but is not limited to, steps S401 to S402:

[0103] Step S401: Convolutional processing of the inverse video frame features is performed through a unidirectional convolutional layer to obtain convolutional data;

[0104] Step S402: Convolutional data is processed by bidirectional convolutional layers to obtain frame-level features.

[0105] In step S401 of some embodiments, the unidirectional convolutional layer is a long short-term memory network. The reverse video frame features are convolved by a single convolutional layer to obtain convolutional data, which makes the conversion of reverse video frame features into convolutional data easy, so as to convert the convolutional data into frame-level features.

[0106] In step S402 of some embodiments, the bidirectional convolutional layer is a bidirectional long short-term memory neural network. By constructing unidirectional and bidirectional convolutional layers, the time required to obtain frame-level features by performing convolution processing on convolutional data can be reduced, thereby reducing the computational load of frame-level feature reconstruction.

[0107] It should be noted that by adding a bidirectional long short-term memory neural network to a long short-term memory network, not only can the computation time for frame-level feature reconstruction be reduced, but the video data can also be converted into binary hash codes through a bidirectional hierarchical decoder. Since long short-term memory networks cannot generate binary hash codes, adding a bidirectional long short-term memory neural network, and having the long short-term memory network convert the video data features into hidden variables, and then using the bidirectional long short-term memory neural network to convert the video data into binary hash codes based on the video identifier and the hidden variables, simplifies the video data conversion process.

[0108] Please see Figure 5 In some embodiments, step S103 may also include, but is not limited to, steps S501 to S503:

[0109] Step S501: Perform mean pooling on the feature sequences of at least two video frames to obtain video-level information;

[0110] Step S502: Perform neighborhood calculation on the video-level information according to the preset neighborhood function to obtain the neighborhood structure;

[0111] Step S503: Based on the neighborhood structure, perform index transformation on the feature sequences of at least two video frames to obtain at least two video index sequences.

[0112] In step S501 of some embodiments, since the video frame feature sequence includes video frames of the training video data, at least two video frame feature sequences are subjected to mean pooling, that is, the mean of all values ​​in the local receptive field of at least two video frame feature sequences is calculated to obtain the video-level information of the training video data.

[0113] In step S502 of some embodiments, the neighborhood calculation of the video information is performed according to a preset neighborhood function to obtain the neighborhood structure. The neighborhood structure is used to analyze whether the neighborhoods of each training video are close or far apart, and then the similarity of the features of the training video data is determined based on the neighborhood structure.

[0114] In step S503 of some embodiments, at least two video frame feature sequences are indexed and transformed according to the neighborhood structure to obtain at least two video index sequences. The constructed video index sequences can distinguish between similar training video data and different training video data. Therefore, the video identifiers obtained by classifying the training video data based on the at least two video index sequences are more accurate. Since training video data with the same video identifier are similar, while training video data with different video identifiers are different, a self-supervised learning model is trained based on the video identifiers and the training video dataset to obtain a more accurate data conversion model. Therefore, the data conversion model can convert similar video data into similar hash codes, and different video data into different hash codes, thus making the search for video data based on hash codes more accurate.

[0115] Please see Figure 6 In addition, this application also discloses a video search method, which includes, but is not limited to, steps S601 to S606:

[0116] Step S601: Obtain raw video data;

[0117] Step S602: Input the raw video data into the data conversion model; wherein, the data conversion model is obtained by the model training method described above;

[0118] Step S603: The original video data is converted using a data conversion model to obtain video sequence data;

[0119] Step S604: Obtain query information;

[0120] Step S605: Select the target sequence data from at least two video sequence data based on the query information;

[0121] Step S606: Select target video data from at least two original video data based on the target sequence data.

[0122] Steps S601 to S606, as illustrated in this embodiment, involve inputting raw video data into a data conversion model. The model converts the raw video data into video sequence data, which is then represented as a binary hash code. By converting the multimodal raw video data into binary hash codes, when a query for raw video data is needed, query information is obtained. Based on this information, target sequence data is selected from the video sequence data, and then, based on the target sequence data, target video data is selected from at least two sets of raw video data. The data conversion model is obtained using the aforementioned model training method. Therefore, the data conversion model can convert similar raw video data into similar hash codes, and different raw video data into different hash codes. This allows for more accurate retrieval of raw video data by using the hash code.

[0123] In step S601 of some embodiments, the video search method runs on a server, and the target video is obtained either from user uploads or from a third-party platform. After obtaining the source video data, and the source video data format includes any of the following: mp4, flv, avi, mkv, rm, rmvb, the source video data format is unified to obtain the original video data, and the original video data format is unified to avi format. Therefore, using the unified data format of the original video data simplifies data conversion.

[0124] In step S603 of some embodiments, the data conversion model is obtained by the model training method described above. Therefore, the original video data is input into the data conversion model to convert the data into video sequence data. Since the video sequence data is a binary hash code, similar original video data is converted into similar hash codes, while different original video data is converted into different hash codes, so as to quickly find the original video data based on low-dimensional query information.

[0125] In step S604 of some embodiments, the query information is obtained based on the query data sent by the user terminal. When the user terminal uploads query data, the query data includes any of the following: image data, text data, and voice data. If the query data is image data, OCR recognition is performed on the image data to obtain image content, and the image content is the query information. If the query data is text data, semantic understanding is performed on the text data to obtain text content, and the text content is the query information. If the query data is voice data, voice content recognition is performed on the voice data to obtain voice content, and the voice content is the query information. Therefore, by obtaining the query data sent by the user terminal and obtaining query information through the query data, it is possible to filter target sequence data from at least two video sequence data based on the query information.

[0126] In step S605 of some embodiments, since the query information is low-dimensional information, all data are filtered from at least two video sequence data according to the query information, so as to filter out the target sequence data that matches the query information from at least two video sequence data.

[0127] In step S606 of some embodiments, since the original video data corresponds to the video sequence data, target video data is filtered from at least two original video data according to the target sequence data, so as to obtain target video data that matches the target sequence data from at least two original video data.

[0128] Please see Figure 7 In some embodiments, step S605 may include, but is not limited to, steps S701 to S702:

[0129] Step S701: Convert the query information into binary to obtain the query sequence;

[0130] Step S702: Select the target sequence data from at least two video sequence data according to the query sequence.

[0131] In step S701 of some embodiments, since the target sequence data is a binary hash code, the query information is converted into binary to obtain the query sequence. Then both the query sequence and the video sequence data are binary hash codes, so the video sequence data can be quickly matched according to the query sequence.

[0132] In step S702 of some embodiments, target sequence data is filtered from at least two video sequence data by querying the sequence, and similarity calculation is performed based on the query sequence and at least two video sequence data to obtain the similarity of each video sequence data. The video sequence data with the highest similarity among the at least two video sequence data is obtained as the target sequence data, which makes the filtering of target sequence data simple. Then, target video data with a higher matching degree with the query information is filtered from at least two original video data based on the target sequence data.

[0133] This application embodiment acquires training video data, extracts features from the training video data to obtain at least two video frame feature sequences, performs index transformation on the at least two video frame feature sequences to obtain at least two video index sequences, classifies the training video dataset according to the at least two video index sequences to obtain video identifiers, and trains a preset self-supervised learning model based on the video identifiers and the training video dataset to obtain a data conversion model. It receives raw video data from a user terminal or a third-party platform, inputs the raw video data into the data conversion model, and the data conversion model performs data conversion on the raw video data to obtain video sequence data. The binary hash code of the video sequence data is obtained by receiving query data sent by the user terminal, extracting query information from the query data, performing binary conversion on the query information to obtain a query sequence, and filtering target sequence data from at least two video sequence data based on the query sequence. Finally, target video data is filtered from at least two raw video data based on the target sequence data. Therefore, by pre-constructing video identifiers, a data conversion model is obtained by training a self-supervised learning model based on the video identifiers and the training video dataset. The data conversion model can convert similar original video data into similar video sequence data, and different original video data into different video sequence data. Thus, it is more accurate to find the target sequence data from at least two video sequence data based on the query sequence, and the target video data found from at least two original video data based on the target sequence data is more consistent with the query information.

[0134] Please see Figure 8 This application also provides a model training apparatus that can implement the above-described model training method. The apparatus includes:

[0135] The data acquisition module 801 is used to acquire the training video dataset; wherein the training video dataset includes at least two training video datasets.

[0136] The extraction module 802 is used to extract features from each training video data to obtain at least two video frame feature sequences.

[0137] The index conversion module 803 is used to perform index conversion on at least two video frame feature sequences to obtain at least two video index sequences.

[0138] Classification module 804 is used to classify at least two training video data according to at least two video index sequences to obtain video identifiers;

[0139] Training module 805 is used to train a pre-defined self-supervised learning model based on video identifiers and training video datasets to obtain a data transformation model.

[0140] The specific implementation of this model training device is basically the same as the specific implementation of the model training method described above, and will not be repeated here.

[0141] Please see Figure 9 This application also provides a video search device that can implement the above-described video search method. The device includes:

[0142] Video acquisition module 901 is used to acquire raw video data;

[0143] The input module 902 is used to input the raw video data into the data conversion model; wherein the data conversion model is obtained by the model training method described above;

[0144] The data conversion module 903 is used to convert the original video data using a data conversion model to obtain video sequence data.

[0145] Information acquisition module 904 is used to acquire query information;

[0146] The sequence filtering module 905 is used to filter target sequence data from at least two video sequence data based on query information;

[0147] The video filtering module 906 is used to filter target video data from at least two original video data based on target sequence data.

[0148] This application also provides an electronic device, which includes: a memory, a processor, a program stored in the memory and executable on the processor, and a data bus for communication between the processor and the memory. When the program is executed by the processor, it implements the aforementioned model training method or video search method. This electronic device can be any smart terminal, including tablet computers, in-vehicle computers, etc.

[0149] Please see Figure 10 , Figure 10 The hardware structure of an electronic device according to another embodiment is illustrated. The electronic device includes:

[0150] The processor 101 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this application.

[0151] The memory 102 can be implemented as a read-only memory (ROM), static storage device, dynamic storage device, or random access memory (RAM). The memory 102 can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 102 and is called and executed by the processor 101 to execute the model training method or video search method of the embodiments of this application.

[0152] Input / output interface 103 is used to implement information input and output;

[0153] The communication interface 104 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0154] Bus 105 transmits information between various components of the device (e.g., processor 101, memory 102, input / output interface 103, and communication interface 104);

[0155] The processor 101, memory 102, input / output interface 103 and communication interface 104 are connected to each other within the device via bus 105.

[0156] This application embodiment also provides a storage medium, which is a computer-readable storage medium for computer-readable storage. The storage medium stores one or more programs, which can be executed by one or more processors to implement the above-described model training method or video search method.

[0157] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0158] The model training method and apparatus, video search method and apparatus, device and medium provided in the embodiments of this application obtain video identifiers based on video frame feature sequences. Then, a preset self-supervised learning model is trained based on the video identifiers and training video datasets. The trained data conversion model can convert similar video data into similarity hash codes, and convert different video data into different hash codes. It is more accurate to find target sequence data from at least two video sequence data based on the query sequence, and the target video data found from at least two original video data based on the target sequence data matches the query information more closely.

[0159] The embodiments described in this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided by the embodiments of this application. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.

[0160] It will be understood by those skilled in the art that Figure 1-5 ,or Figure 6-7 The technical solutions shown do not constitute a limitation on the embodiments of this application, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0161] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0162] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0163] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0164] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0165] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of the units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0166] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0167] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0168] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0169] The preferred embodiments of the present application have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present application. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and substance of the embodiments of the present application shall be within the scope of the claims of the present application.

Claims

1. A model training method, characterized in that, The method includes: Obtain a training video dataset; wherein the training video dataset includes at least two training video datasets; Feature extraction is performed on each of the training video data to obtain at least two video frame feature sequences; The feature sequences of the at least two video frames are indexed to obtain at least two video index sequences. The at least two training video data are classified according to the at least two video index sequences to obtain video identifiers; A pre-defined self-supervised learning model is trained based on the video identifier and the training video dataset to obtain a data conversion model; wherein, the data conversion model is used to convert similar video data into similar hash codes and to convert different video data into different hash codes.

2. The method according to claim 1, characterized in that, Each training video data includes at least two video frames, and the feature extraction of each training video data to obtain at least two video frame feature sequences includes: Feature extraction is performed on each video frame to obtain video frame features; Each video frame feature in the training video data is transformed into a feature sequence to obtain at least two video frame feature sequences.

3. The method according to claim 2, characterized in that, The step of extracting features from each video frame to obtain video frame features includes: Each video frame is encoded using a global hierarchical encoder to obtain global video-level features; The global video-level features are encoded using a backward hierarchical encoder to obtain inverse video frame features; The inverse video frame features are decoded using a bidirectional hierarchical decoder to obtain frame-level features; The frame-level features are encoded by a forward hierarchical encoder to obtain the video frame features.

4. The method according to claim 3, characterized in that, The bidirectional hierarchical decoder includes a unidirectional convolutional layer and a bidirectional convolutional layer. The bidirectional hierarchical decoder decodes the inverse video frame features to obtain frame-level features, including: The inverse video frame features are convolved using the unidirectional convolutional layer to obtain convolutional data; The frame-level features are obtained by performing convolution processing on the convolutional data through the bidirectional convolutional layer.

5. The method according to any one of claims 1 to 4, characterized in that, The step of indexing and transforming the feature sequences of the at least two video frames to obtain at least two video index sequences includes: The feature sequences of at least two video frames are subjected to mean pooling to obtain video-level information. The neighborhood structure is obtained by performing neighborhood calculation on the video-level information according to the preset neighborhood function. The at least two video frame feature sequences are indexed and transformed according to the neighborhood structure to obtain the at least two video index sequences.

6. A video search method, characterized in that, The method includes: Obtain raw video data; The original video data is input into a data conversion model; wherein the data conversion model is obtained by the model training method according to any one of claims 1 to 5; The original video data is converted using the data conversion model to obtain video sequence data; Retrieve query information; Target sequence data is selected from at least two video sequence data based on the query information; Target video data is selected from at least two of the original video data based on the target sequence data.

7. The method according to claim 6, characterized in that, The step of filtering target sequence data from at least two video sequence data based on the query information includes: The query information is converted into binary to obtain a query sequence; The target sequence data is selected from at least two video sequence data based on the query sequence.

8. A model training device, characterized in that, The device includes: A data acquisition module is used to acquire a training video dataset; wherein the training video dataset includes at least two training video datasets. An extraction module is used to extract features from each of the training video data to obtain at least two video frame feature sequences. An index conversion module is used to perform index conversion on the at least two video frame feature sequences to obtain at least two video index sequences; A classification module is used to classify the at least two training video data according to the at least two video index sequences to obtain video identifiers; The training module is used to train a preset self-supervised learning model based on the video identifier and the training video dataset to obtain a data conversion model; wherein, the data conversion model is used to convert similar video data into similar hash codes and convert different video data into different hash codes.

9. An electronic device, characterized in that, The electronic device includes a memory, a processor, a program stored in the memory and executable on the processor, and a data bus for enabling communication between the processor and the memory. When the program is executed by the processor, it implements the steps of the method as described in any one of claims 1 to 5, or the steps of the method as described in any one of claims 6 to 7.

10. A storage medium, said storage medium being a computer-readable storage medium for computer-readable storage, characterized in that, The storage medium stores one or more programs, which can be executed by one or more processors to implement the steps of the method as described in any one of claims 1 to 5, or the steps of the method as described in any one of claims 6 to 7.

Images