Model training method, video query method and device
By extracting visual features from video frames using a self-supervised pre-training method, the problem of video query systems relying on labeled datasets is solved, achieving more efficient feature extraction and accurate video retrieval.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- TENCENT TECHNOLOGY (SHENZHEN) CO LTD
- Filing Date
- 2022-09-28
- Publication Date
- 2026-07-10
AI Technical Summary
In existing technologies, video query systems rely on labeled datasets for training, which results in high manpower costs and limited dataset size, affecting model performance and inaccurate retrieval results.
By extracting unlabeled video frames from video data as training samples, and using the first and second encoding networks for visual feature contrast learning, a self-supervised pre-trained visual encoder is achieved, enhancing the feature extraction capability.
Without relying on manual labeling, the training dataset is expanded to improve the accuracy of video queries and user experience, and to avoid retrieving irrelevant video content.
Smart Images
Figure CN116992947B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of artificial intelligence technology, and in particular to a training model method, a video query method, and an apparatus. Background Technology
[0002] With the rapid development of internet technology, multimedia applications are becoming increasingly widespread, and the number of videos is growing dramatically. Users need to sift through this massive amount of video content to find the videos they need. Currently, video-related applications offer video search functions. After users enter their search information in the search bar, the applications can retrieve and display videos that match their search query.
[0003] In related technologies, user-uploaded videos often have descriptions or tags that are not closely related to the actual video content. In such cases, matching the user's query information with the video description or tags may result in irrelevant video searches. Therefore, a visual encoder can be used to obtain the semantic information contained in the video frames. Then, the semantic information of the video frames can be matched with the semantic information of the user's query information to perform video filtering, thereby improving the accuracy of video searches.
[0004] Traditional visual encoders typically utilize labeled datasets for supervised pre-training. While training with labeled datasets can uncover semantic information from videos to some extent, the high human cost of manually labeling the dataset limits the size of such datasets. Conversely, using a small amount of labeled data for pre-training negatively impacts model performance. Summary of the Invention
[0005] This application provides a model training method, a video query method, and an apparatus that can enhance the feature extraction capability of a visual encoder, thereby improving the accuracy of video queries.
[0006] In a first aspect, embodiments of this application provide a model training method, including:
[0007] Acquire first video data and extract at least two video frames from the first video data;
[0008] The first video frame of the at least two video frames is input into the first encoding network to extract the first visual features of the first video frame, wherein the first encoding network includes a visual encoder.
[0009] The second video frame of the at least two video frames is input into the second coding network to extract the second visual features of the second video frame. The first coding network and the second coding network have the same network structure, and the parameters of the second coding network are updated by the parameters of the first coding network based on the momentum shift value.
[0010] Calculate the loss based on the first visual feature and the second visual feature;
[0011] Based on the loss, the parameters of the first encoding network are updated to obtain the trained visual encoder.
[0012] Secondly, embodiments of this application provide a video query method, including:
[0013] Obtain the query text and extract its semantic features to obtain the query semantic features;
[0014] Obtain at least one candidate video;
[0015] Visual features of video frames of the candidate video are extracted using a visual encoder, wherein the visual encoder is trained according to the method described in the first aspect;
[0016] Based on the visual features and the query semantic features, the target video corresponding to the query text is determined from the at least one candidate video.
[0017] Thirdly, embodiments of this application provide a model training apparatus, including:
[0018] An acquisition unit is used to acquire first video data and extract at least two video frames from the first video data.
[0019] A first encoding network is used to take a first video frame from the at least two video frames as input and extract a first visual feature of the first video frame, wherein the first encoding network includes a visual encoder;
[0020] A second encoding network is used to take a second video frame from the at least two video frames as input and extract a second visual feature of the second video frame. The first encoding network and the second encoding network have the same network structure, and the parameters of the second encoding network are updated by the parameters of the first encoding network based on the momentum shift value.
[0021] A calculation unit is configured to calculate the loss based on the first visual feature and the second visual feature;
[0022] An update unit is used to update the parameters of the first encoding network according to the loss, so as to obtain the trained visual encoder.
[0023] Fourthly, embodiments of this application provide a video query device, including:
[0024] The acquisition unit is used to acquire the query text and extract semantic features from the query text to obtain query semantic features;
[0025] The acquisition unit is also used to acquire at least one candidate video;
[0026] A visual encoder is used to extract visual features from video frames of the candidate video, wherein the visual encoder is trained according to the method described in the first aspect;
[0027] The determining unit is configured to determine the target video corresponding to the query text from the at least one candidate video based on the visual features and the query semantic features.
[0028] Fifthly, embodiments of this application provide an electronic device, including: a processor and a memory, the memory being used to store a computer program, and the processor being used to call and run the computer program stored in the memory to perform the methods as described in the first or second aspect.
[0029] In a sixth aspect, embodiments of this application provide a computer-readable storage medium including instructions that, when executed on a computer, cause the computer to perform the methods as described in the first or second aspect.
[0030] In a seventh aspect, embodiments of this application provide a computer program product including computer program instructions that cause a computer to perform the methods described in the first or second aspect.
[0031] Eighthly, embodiments of this application provide a computer program that causes a computer to perform the methods described in the first or second aspect.
[0032] By using the above technical solution, at least two video frames extracted from video data are used as unlabeled training samples. The first and second coding networks are used to extract the visual features of different video frames in the at least two video frames. By comparing and learning the visual features of the different video frames, the semantic information of the video data is mined without relying on manually labeled tags. This allows the training dataset to be expanded with low overhead to achieve large-scale self-supervised pre-training to obtain a visual encoder, thereby enhancing the feature extraction capability of the visual encoder.
[0033] Furthermore, in this embodiment of the application, when using the semantic information of the query text entered by the user to match the semantic information of the video frames obtained by the visual encoder, it no longer relies on the title or description of the video, thereby helping to avoid retrieving irrelevant video content, achieving more accurate text-video retrieval, and thus improving the user's product experience. Attached Figure Description
[0034] Figure 1 A schematic diagram of the system architecture of the solution in an embodiment of this application;
[0035] Figure 2 This is a schematic diagram of the interface for video search on WeChat Video Channel;
[0036] Figure 3 This is a schematic flowchart illustrating a model training method according to an embodiment of this application;
[0037] Figure 4 This is a schematic diagram of a network architecture for a training model according to an embodiment of this application;
[0038] Figure 5 This is a schematic flowchart illustrating a video query method according to an embodiment of this application;
[0039] Figure 6 This is a schematic block diagram of a training model apparatus according to an embodiment of this application;
[0040] Figure 7 This is a schematic block diagram of a video query device according to an embodiment of the present invention;
[0041] Figure 8 This is a schematic block diagram of an electronic device according to an embodiment of this application. Detailed Implementation
[0042] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention.
[0043] It should be understood that in the embodiments of this application, "B corresponding to A" means that B is associated with A. In one implementation, B can be determined based on A. However, it should also be understood that determining B based on A does not mean determining B solely based on A; B can also be determined based on A and / or other information.
[0044] In the description of this application, unless otherwise stated, "at least one" means one or more, and "multiple" means two or more. Additionally, "and / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can mean: A alone, A and B simultaneously, or B alone, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can mean: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or multiple.
[0045] It should also be understood that the descriptions of "first", "second", etc. appearing in the embodiments of this application are only for illustration and to distinguish the objects being described, and there is no order to them. They do not indicate any special limitation on the number of devices in the embodiments of this application, and cannot constitute any limitation on the embodiments of this application.
[0046] It should also be understood that specific features, structures, or characteristics relating to embodiments in the specification are included in at least one embodiment of this application. Furthermore, these specific features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.
[0047] Furthermore, the terms “comprising” and “having”, and any variations thereof, are intended to cover non-exclusive inclusion, such that a process, method, system, product, or server that includes a series of steps or units is not necessarily limited to those steps or units that are explicitly listed, but may include other steps or units that are not explicitly listed or that are inherent to such processes, methods, products, or devices.
[0048] The embodiments of this application are applied to the field of artificial intelligence technology.
[0049] Artificial intelligence (AI) refers to the theories, methods, technologies, and application systems that utilize digital computers or computers-controlled machines to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that attempts to understand the essence of intelligence and produce new intelligent machines that can react in a way similar to human intelligence. AI studies the design principles and implementation methods of various intelligent machines, enabling them to possess perception, reasoning, and decision-making capabilities.
[0050] Artificial intelligence (AI) is a comprehensive discipline encompassing a wide range of fields, including both hardware and software technologies. Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies primarily include computer vision, speech processing, natural language processing, and machine learning / deep learning.
[0051] With the research and advancement of artificial intelligence (AI) technology, AI is being studied and applied in various fields, such as smart homes, smart wearable devices, virtual assistants, smart speakers, smart marketing, autonomous driving, drones, robots, smart healthcare, and smart customer service. It is believed that with the development of technology, AI will be applied in more fields and play an increasingly important role.
[0052] This application's embodiments may relate to Computer Vision (CV) technology within artificial intelligence. Computer vision is a science that studies how to enable machines to "see." More specifically, it refers to machine vision, which uses cameras and computers to replace human eyes in recognizing and measuring targets, and further performs image processing to transform the computer-processed images into those more suitable for human observation or transmission to instruments for detection. As a scientific discipline, computer vision researches related theories and technologies, attempting to establish artificial intelligence systems capable of extracting information from images or multidimensional data. Computer vision technologies typically include image processing, image recognition, image semantic understanding, image retrieval, OCR, video processing, video semantic understanding, video content / behavior recognition, 3D object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous localization and mapping (SLAM), and other technologies, as well as common biometric recognition technologies such as facial recognition and fingerprint recognition.
[0053] This application's embodiments may also relate to Machine Learning (ML) in artificial intelligence technology. ML is a multidisciplinary field involving probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory, and many other disciplines. It specifically studies how computers can simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to endow computers with intelligence; its applications span all areas of artificial intelligence. Machine learning and deep learning typically include techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and instructional learning.
[0054] The relevant terms used in the embodiments of this application are described below.
[0055] Pre-training includes two methods: supervised pre-training and self-supervised pre-training. In supervised pre-training, a classification task is used as the upstream task. A supervised classification model is pre-trained on a large-scale labeled classification dataset. Then, the last layer of the fully connected classification neurons is removed, and the remaining parts are used as a pre-trained feature encoder, which is then transferred to the downstream task. Self-supervised pre-training uses different self-supervised tasks as upstream tasks. A self-supervised model is pre-trained on a large-scale unlabeled dataset. The backbone network of this model is then retained as a pre-trained feature encoder and transferred to the downstream task.
[0056] Feature encoder: A neural network model used to map a high-dimensional input image to a low-dimensional feature vector. For example, this neural network model can be a convolutional neural network (CNN), which is a computational network composed of multiple convolution operations.
[0057] A visual encoder, as a type of feature encoder, is primarily used to extract features from video frame data to obtain visual features. Visual encoders can be obtained through pre-training.
[0058] Figure 1 This is a schematic diagram of a system architecture involved in an embodiment of this application. For example... Figure 1 As shown, the system architecture may include user equipment 101, data acquisition equipment 102, training equipment 103, execution equipment 104, database 105, and content library 106.
[0059] The data acquisition device 102 is used to read training data from the content library 106 and store the read training data in the database 105. The training data involved in this embodiment includes unlabeled sample data. For example, the sample data may include video frames extracted from videos, without limitation.
[0060] Training device 103 trains a machine learning model based on training data maintained in database 105. The model obtained by training device 103 can effectively extract features from samples (e.g., video frames). Furthermore, this model can be further connected to other downstream models. The model obtained by training device 103 (e.g., a visual encoder) can be applied to different systems or devices.
[0061] Additionally, refer to Figure 1The execution device 104 is equipped with an I / O interface 107 for data interaction with external devices. For example, it receives data to be predicted, such as image data, sent by the user device 101 through the I / O interface. The computing module 109 in the execution device 104 processes the input data using a trained machine learning model, outputs the prediction results of the data, and sends the corresponding results back to the user device 101 through the I / O interface.
[0062] User equipment 101 may include mobile phones, tablets, laptops, handheld computers, vehicle terminals, mobile internet devices (MIDs), or other terminal devices with browser installation capabilities.
[0063] The execution device 104 can be a server. For example, the server can be a rack server, blade server, tower server, or cabinet server, etc. The server can be a standalone server or a server cluster composed of multiple servers.
[0064] In this embodiment, the execution device 104 is connected to the user equipment 101 via a network. The network can be an intranet, the Internet, the Global System for Mobile Communication (GSM), Wideband Code Division Multiple Access (WCDMA), 4G network, 5G network, Bluetooth, Wi-Fi, voice communication network, or other wireless or wired networks.
[0065] It should be noted that, Figure 1 This is merely a schematic diagram of a system architecture provided in an embodiment of this application. The positional relationships between the devices, components, modules, etc., shown in the figure do not constitute any limitation. In some embodiments, the data acquisition device 102, user device 101, training device 103, and execution device 104 may be the same device. The database 105 may be distributed across one server or multiple servers, and the content library 106 may be distributed across one server or multiple servers.
[0066] In related technologies, visual encoders are used to acquire visual features from video frames to represent semantic information in videos. Traditional visual encoders typically utilize labeled video datasets for supervised pre-training. While training with labeled video datasets can extract semantic information to some extent, the high human cost of manually labeling the dataset limits the availability of large-scale labeled datasets. Conversely, using a limited number of labeled datasets for pre-training negatively impacts model performance.
[0067] Based on this, embodiments of this application provide a model training method and apparatus. By using at least two video frames extracted from video data as unlabeled training samples, a first encoding network and a second encoding network are used to extract visual features of different video frames in the at least two video frames. By comparing and learning the visual features of the different video frames, the semantic information of the video data is mined without relying on manually labeled tags. This allows for the expansion of the training dataset with lower overhead to achieve large-scale self-supervised pre-training to obtain a visual encoder, thereby enhancing the feature extraction capability of the visual encoder.
[0068] This application can be applied to any application scenario requiring visual semantics, including but not limited to video search services on video platforms, and news search in image or video format. For example, in video search, the pre-trained visual encoder obtained by this application can be used to quickly and accurately extract visual features from video frames, obtain the semantic information contained in the video, and then combine the semantic information of the user-input query information with the semantic information of the video frames for video filtering. Similarly, in news search, the pre-trained visual encoder obtained by this application can be used to quickly and accurately extract visual features from images or video frames, obtain the semantic information contained in the images or video frames, and while matching the semantic information of the news text content with the semantic information of the news keywords input by the user, further news filtering can be performed by combining the semantic information of the news images or video frames.
[0069] Figure 2 This diagram illustrates the interface for video searching on the Video Channel. (For example...) Figure 2 As shown, (a) the application interface in the figure includes the main entry point 210 of the video account, (b) the figure shows the main interface after clicking on the entry point 210, including the search entry point 220, and (c) the figure shows the videos that may be of interest after entering the query "basketball teaching" in the search entry point 220.
[0070] For search-based products, the relevance of search results is a crucial metric. Relevance technology uses user-input query text to match video titles or descriptions. This method relies on the user-uploaded title or description being highly relevant to the video content or adequately describing it. For example, if a user inputs the query "basketball tutorial," but the returned video title is indeed "basketball tutorial," but the video content is an advertisement for basketballs, the relevance between the video content and the title or description is weak. This significantly impacts the accuracy of the video search and results in a poor user experience.
[0071] In this embodiment, at least two video frames extracted from video data are used as unlabeled training samples. This allows for the expansion of the training dataset with lower overhead, enabling large-scale self-supervised pre-training to obtain a visual encoder. This visual encoder can fully extract the semantic information of the video data without relying on manually labeled tags, resulting in richer and more accurate semantic representations of the extracted visual features. Therefore, when using the semantic information of the user-input query text to match the semantic information of the video frames obtained by the visual encoder, it no longer depends on the video title or description, thus helping to avoid retrieving irrelevant video content, achieving more accurate text-video retrieval, and ultimately improving the user's product experience.
[0072] The technical solutions of the embodiments of this application will be described in detail below through some examples. The following embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments.
[0073] First, combine Figure 3 The training process of the model involved in the embodiments of this application is described.
[0074] Figure 3 This is a schematic flowchart illustrating a method 300 for training a model according to an embodiment of this application. This method 300 can be executed by any electronic device with data processing capabilities. For example, the electronic device can be implemented as a server, or, for example, as a... Figure 1 The training device 103 in this application is not limited thereto.
[0075] like Figure 3 As shown, method 300 includes steps 310 to 350.
[0076] 310. Acquire the first video data and extract at least two video frames from the first video data.
[0077] For example, m video frames can be extracted at equal intervals from the first video data to be processed, where m is a positive integer.
[0078] It should be noted that the m video frames extracted from the first video data are semantically related. Specifically, video files contain rich semantic information. For example, in a short video of a basketball game, although the visual perspective, player positions, and camera angles may change during playback, the semantics between the video frames are generally quite related, all being scenes related to the basketball game, such as a player dribbling and shooting a layup, a player being stripped of the ball, or the cheering crowd after a goal. As a concrete example, the semantic connections between temporally adjacent video frames are even closer; for example, adjacent video frames exhibit semantic continuity.
[0079] Therefore, this embodiment of the application extracts at least two video frames from the video data as unlabeled training samples, enabling the visual encoder to learn from the semantic information inherent in the video data, resulting in richer and more diverse samples compared to those generated using data augmentation.
[0080] 320. The first video frame from the at least two video frames is input into a first encoding network to extract the first visual features of the first video frame. The first encoding network includes a visual encoder. The parameters of the first encoding network are updated through gradient backpropagation.
[0081] 330. The second video frame from the at least two video frames is input into the second encoding network to extract the second visual features of the second video frame. The first and second encoding networks have the same network structure, and the parameters of the second encoding network are updated by the parameters of the first encoding network based on the momentum shift value. For example, the parameters of the second encoding network can be slowly updated by the parameters of the first encoding network based on the moving average of momentum.
[0082] In some embodiments, the first video frame and the second video frame can be any of the at least two video frames extracted from the first video data, and this application does not limit this. Since both the first video frame and the second video frame are video frames extracted from the first video data, their semantics are relatively related. Therefore, the semantic correlation between the first video frame and the second video frame can be used for self-supervised pre-training, enabling the visual encoder to have better visual semantic understanding capabilities.
[0083] In some embodiments, the first video frame and the second video frame can be two adjacent video frames from at least two video frames extracted from the first video data. Since the first video frame and the second video frame are adjacent video frames extracted from the first video data, the semantic connection between them is closer. Therefore, the semantic continuity between the first video frame and the second video frame can be used for self-supervised pre-training to further improve the visual encoder's visual semantic understanding ability.
[0084] In some embodiments, the first video frame and the second video frame can be clusters of adjacent video frames extracted from at least two video frames in the first video data. The video frame clustering is obtained based on at least two adjacent video frames. Since adjacent video frame clusters also have semantic relevance or semantic continuity, using adjacent video frame clustering for self-supervised training can also enable the visual encoder to have better semantic understanding capabilities.
[0085] In some embodiments, the electronic device may include (e.g., deploy) a network architecture for training a model, and a method 300 for performing self-supervised training of the training model based on unlabeled video data.
[0086] Figure 4 A schematic diagram of the network architecture for training the model is shown, which can be used to execute method 300. For example... Figure 4 As shown, the network architecture includes two main modules: a query network 410 and a key network 420. The query network 410 can be used as an example of a first encoding network, and the key network 420 can be used as an example of a second encoding network. In some embodiments, the query network 410 may further include a first visual encoder 411, a first projector 412, and a predictor 413, and the key network 420 may further include a second visual encoder 421 and a second projector 422.
[0087] It should be understood that Figure 4 An example of a network architecture for model training is shown. This example is merely intended to help those skilled in the art understand and implement the embodiments of this application, and is not intended to limit the scope of the embodiments of this application. Those skilled in the art can make equivalent transformations or modifications based on the examples given herein, and such transformations or modifications should still fall within the protection scope of the embodiments of this application.
[0088] In some embodiments, the first visual encoder 411 and the first mapper 412 in the Query network 410 have the same network structure as the second visual encoder 421 and the second mapper 422 in the Key network 420, but the network parameters are updated in different ways.
[0089] Specifically, the first visual encoder 411 and the first mapper 412 are updated through gradient backpropagation, and the second visual encoder 421 and the second mapper 422 are updated with momentum based on the stopgrad update of the corresponding first visual encoder 411 and the first mapper 412. For example, the parameters of the second visual encoder 421 are slowly updated by the parameters of the first visual encoder 411 based on the moving average of momentum, and the parameters of the second mapper 422 are slowly updated by the parameters of the first mapper 412 based on the moving average of momentum.
[0090] Momentum, originally a concept in physics, is equal to the mass of an object multiplied by its current velocity, used to represent the object's state of motion, P. t From the object's previous state P t-1 It consists of two parts: the change in momentum ΔP caused by the force acting on the object, and the change in momentum of the object.
[0091] P t =ξP t-1 +(1-ξ)ΔP
[0092] Where ξ∈[0,1] are momentum update coefficients, used to control the previous time step P. t-1 The proportion it accounts for.
[0093] Momentum, a concept from physics, can be applied to machine learning. Specifically, the parameters of the Query network 410 can be represented as θ. q The parameters of the Key network 420 can be expressed as θ k The parameters θ of the Key network 420 can be obtained. k Analogous to momentum, the parameters θ of the Key Network 420 k Perform momentum update, i.e., update the parameters θ of the current Key network 420. k Depends on the parameters θ of the Key network 420 at the previous time step k And the parameter θ of the current Query network 410 q The weighted sum, i.e., θ k The parameter θ of the corresponding part of the Query network 410 q The update is based on the moving average of momentum, as shown in the following formula (1):
[0094] θ k ←ξθ k +(1-ξ)θ q (1)
[0095] Where ξ is the momentum update coefficient, used to control θ at the previous moment. kThe proportion it accounts for is usually between 0.99 and 1. As a specific example, it can be set to 0.99-0.9999.
[0096] It should be noted that when updating network parameters via backpropagation, the model parameters must be loaded into the graphics processing unit (GPU) memory for fast parallel computation. Furthermore, the GPU memory needs to store data such as gradients corresponding to backpropagation, which often involves a large amount of data. However, this embodiment updates the parameters of the second encoding network (e.g., the Key network) using momentum updates. This eliminates the need to store gradient data and perform fast parallel computation; the network parameters only need to be updated according to the momentum update formula. Therefore, this embodiment helps reduce the large memory overhead required for pre-training and saves computational resources, allowing training to be performed using fewer GPUs.
[0097] In some embodiments, the first visual encoder 411 and the second visual encoder 421 may each include a visual Transformer with the same network structure. The visual Transformer is mainly used to extract features from video frames to obtain visual features. For example, the visual Transformer includes modules such as a multi-head attention mechanism, skip connections, layer normalization, and a feedforward neural network.
[0098] In other embodiments, the first visual encoder 411 and the second visual encoder 421 may each include a CNN with the same network structure, i.e., a computational network composed of multiple convolutional operations. CNNs are mainly used to extract features from video frames to obtain visual features.
[0099] It should be noted that the visual encoder in the embodiments of this application can be any network structure capable of extracting image or video features. This application does not limit its specific form. For example, the visual encoder can also include a combination of visual Transformer and CNN with the same network structure.
[0100] For example, the encoding process of a visual encoder can be expressed as the following formula (2):
[0101] v fi =visual_enc(v i (2)
[0102] Among them, v i The input video frames are visual encoders, where visual_enc represents the visual encoder, and v fi These are the visual features extracted from video frames by the visual encoder. As a concrete example, vfi The dimension is 512.
[0103] Optionally, the first mapper 412 and the second mapper 422 may each include a multilayer perceptron (MLP), such as a two-layer MLP. Introducing mappers into the encoding network can help reduce the impact of upstream and downstream tasks on the encoding network, improve the transfer capability of the visual encoder, and alleviate overfitting during pre-training.
[0104] Optionally, the predictor 413 may include an MLP, such as a two-layer MLP. Introducing a predictor into the encoding network allows the Query network 410 and the Key network 420 to be asymmetric, preventing learning collapse.
[0105] See also Figure 4 The encoding process of Query Network 410 and Key Network 420 is as follows: Given a video, m frames are sampled from the video, i.e., v = {v1, v2, ..., v...} m Then, the data of the i-th video frame v... i The input Query network 410 is sequentially encoded by the first visual encoder 411, the first mapper 412, and the predictor to obtain vq. i vq i This is an example of a first visual feature. Simultaneously, the data v of the (i+1)th video frame... i+1 The input is sequentially fed into the Key network 420, and then sequentially encoded by the second visual encoder 421 and the second mapper 422 to obtain vk. i+1 vk i+1 This is an example of a second visual feature.
[0106] As an example, VQ i and VK i+1 It can be shown in the following formulas (3) and (4).
[0107] vq i =Pred(Proj(visual_enc(v i ))) (3)
[0108] vk i+1 =Proj_k(visual_enc_k(v i+1 (4)
[0109] Where visual_enc, Proj, and Pred represent the first visual encoder 411, the first mapper 412, and the predictor 413 of the Query network 410, respectively, and visual_enc_k and Proj_k represent the second visual encoder 421 and the second mapper 422 of the Key network 420, respectively.
[0110] As a concrete example, the input and output dimensions of the first mapper 412, the predictor 413, and the second mapper are all 512, vq i and VK i+1 All are 512-dimensional feature vectors.
[0111] 440. Calculate the loss based on the first visual feature and the second visual feature.
[0112] In some embodiments, the first and second video frames can be used as positive sample pairs to shorten the distance between positive samples and their corresponding counterparts during model training. In this case, the loss can be calculated based on the minimum distance between the first and second visual features.
[0113] In some embodiments, the first and second video frames described above can be used as positive sample pairs, and the third video frame in the second video data (i.e., other video data different from the first video data in step 310 above) can be used as negative samples. During model training, the distance between positive samples and their corresponding positive samples is reduced while the distance between samples and negative samples is increased. At this time, the loss can be calculated based on the minimum distance between the first visual feature and the second visual feature, and the maximum distance between the first visual feature and the third visual feature obtained from the third video frame.
[0114] As one possible implementation, in method 300, second video data can be acquired, a third video frame can be extracted from the second video data, and the third video frame can be input into the second coding network to extract the third visual features of the third video frame. At this time, a loss can be calculated based on the first visual features, the second visual features, and the third visual features. The first and second visual features are positive sample features, and the third visual feature is a negative sample feature.
[0115] It should be noted that the number of third video frames is not limited in the embodiments of this application. For example, the third video frame may include multiple video frames or multiple video frames clustered together.
[0116] Optionally, a negative sample queue can be maintained to store multiple negative sample features. Optionally, visual features output by the second encoding network can be added to this negative sample queue as negative sample features, thus adding negative sample features to the queue. For example, a third visual feature can be added to this negative sample queue.
[0117] For example, see Figure 4 The features vk output by the Key network 420 can be used to... i+1 Add it to the negative sample queue 430. When the video input to the Query network 410 and Key network 420 is another video, the feature vk can be obtained from the negative sample queue 430. i+1 As a feature of negative samples.
[0118] Therefore, when updating the parameters of the second encoding network through momentum updates in this embodiment, the visual features generated by the second encoding network can be stored in the negative sample queue instead of GPU memory. This reduces the large memory overhead required for pre-training while maintaining a relatively large negative sample queue of visual features. Introducing more negative samples for comparative learning can help improve model performance.
[0119] Furthermore, the embodiments of this application use momentum updates to slowly update the parameters of the second encoding network, which helps to ensure the consistency of features in the negative sample queue. If the second encoding network is updated too quickly, the features entering the negative sample queue may differ too much, making learning impossible.
[0120] In some embodiments, when the first visual feature and the second visual feature are positive sample features, and the third visual feature is a negative sample feature, a contrastive learning method can be used to calculate the contrastive loss for model training. As an example, for Figure 4 In the network architecture, the noise-contrastive estimation (NCE) loss can be determined according to the following formula (5), i.e., the contrast loss.
[0121]
[0122] Where τ is the temperature hyperparameter, q represents the feature representation output by the Query network, and k + The key network outputs a feature representation of the positive sample that matches q, {k}. -} represents the set of feature representations of the negative samples corresponding to q output by the Key network.
[0123] In some embodiments, a second video frame may be input into a first encoding network to extract a fourth visual feature of the second video frame, and a first video frame may be input into a second encoding network to extract a fifth visual feature of the first video frame. Then, based on the first, second, fourth, and fifth visual features, a symmetric loss is calculated, and this symmetric loss is determined as the loss for model training.
[0124] As one possible implementation, after calculating the contrast loss #1 based on the first, second, and third visual features, a contrast loss #2 can also be calculated based on the fourth, fifth, and third visual features. The sum of this contrast loss #1 and contrast loss #2 is then used as the symmetric loss, i.e., the loss during model training.
[0125] Among them, the fourth and fifth visual features are used as positive sample features, and the third visual feature is used as negative sample features. The specific calculation process is similar to the process of calculating the contrast loss #1 based on the first, second, and third visual features. Please refer to the description above, which will not be repeated here.
[0126] As an example, for Figure 4 The network architecture in the code uses the visual features vq of the i-th video frame encoded by the Query network 410 when calculating the loss for adjacent frame matching. i Visual features vk of the (i+1)th video frame encoded by the Key network 420 i+1 As a positive sample pair, the visual feature vq of the (i+1)th video frame encoded by the Query network 410. i+1 Visual features vk of the i-th video frame encoded by the Key network 420 i As positive sample pairs, video frames from other videos encoded by the Key network are used as negative sample queues. The Frame Adjacency Matching (FAM) loss, i.e., the symmetric loss, is calculated according to the following formula (6).
[0127]
[0128] Where m is the number of video frames sampled in each video; vq i and VQ i+1 These are the visual features of the i-th and (i+1)-th video frames in the same video encoded by the Query network; vk i and VK i+1 These are the visual features of the i-th and (i+1)-th video frames in the same video encoded by the Key network; V - It is a negative sample queue composed of visual features of video frames from other videos encoded by the Key network.
[0129] 350. Based on the loss, the parameters of the first encoding network are updated to obtain the trained visual encoder.
[0130] Specifically, the parameters of each network module in the first encoding network can be updated using the gradient backpropagation update algorithm based on the loss until the training stop condition is met. The output of the visual encoder in the first encoding network that meets the training stop condition is then used as the trained visual encoder.
[0131] For example, for Figure 4 In the network architecture, after the model training is completed, the first visual encoder 411 in the Query network 420 can be retained for subsequent extraction of visual features, while the first mapper 412 and predictor 413 in the Query network 420, as well as the entire Key network 420, can be discarded.
[0132] Therefore, this embodiment of the application extracts at least two video frames from video data as unlabeled training samples, uses a first encoding network and a second encoding network to extract visual features of different video frames in the at least two video frames, and learns by comparing the visual features of the different video frames to mine the semantic information of the video data without relying on manually labeled tags. This expands the training dataset with low overhead to achieve large-scale self-supervised pre-training to obtain a visual encoder, thereby enhancing the feature extraction capability of the visual encoder.
[0133] Furthermore, in this embodiment of the application, the first video frame and the second video frame are specifically implemented as adjacent video frames, and the semantic continuity between the first video frame and the second video frame is used for self-supervised pre-training, which enables the visual encoder to have better visual semantic understanding capabilities.
[0134] Furthermore, by using the momentum contrastive learning method to maintain the negative sample queue, the embodiments of this application can reduce the large memory overhead required for pre-training, while introducing more negative samples for contrastive learning, thus helping to improve model performance.
[0135] The visual encoder trained in this application embodiment can be used for video queries based on visual semantics. Figure 5 A schematic flowchart of a video query method 500 provided in an embodiment of this application is shown. Figure 5 As shown, method 500 includes steps 510 to 540.
[0136] 510. Obtain the query text and extract its semantic features to obtain the query semantic features.
[0137] The query text can be generated from the query data, which can be in text, audio, or image format, without limitation. When the query data is in text format, text cleaning can be performed to obtain the query text; when the query data is in audio format, speech recognition can be performed to obtain the query text; and when the query data is in image format, image text recognition can be performed to obtain the query text.
[0138] In some embodiments, after obtaining the query text, a semantic representation model can be used to extract the semantic features of the query text to obtain the query semantic features. For example, the semantic representation model can be a Bidirectional Encoder Representations from Transformer (BERT) model, which is not limited in this application.
[0139] 520, obtain at least one candidate video.
[0140] For example, at least one candidate video can be selected from a candidate video library. Here, the candidate video library can be a pre-defined video library that contains all the videos corresponding to the video application on the video application server.
[0141] Optionally, step 520 can be implemented by obtaining at least one candidate video corresponding to the query text from the candidate video library. Specifically, a preliminary screening of a large number of videos in the candidate video library can be performed based on the query text to obtain the at least one candidate video.
[0142] 530. Visual features of video frames from candidate videos are extracted using a visual encoder, wherein the visual encoder is trained using at least two video frames extracted from the video data as unlabeled training samples. Specifically, the visual encoder can be trained according to the model training method 300 provided in the embodiments of the above application.
[0143] For example, m video frames can be extracted from each of the at least one candidate video in step 520, and then visual features can be extracted from each video frame of each video using a visual encoder to obtain the semantic information contained in the video. That is, the visual features of the video frames contain the semantic information of the video. Here, the visual encoder is trained according to the model training method provided in the embodiments of this application. The visual encoder can mine the semantic information of video data without relying on manually labeled tags, and the semantic representation of the obtained visual features is richer and more appropriate.
[0144] 540. Based on the above visual features and query semantic features, determine the target video corresponding to the query text in at least one candidate video.
[0145] In some embodiments, the target video corresponding to the query text can be determined from at least one candidate video based on the similarity score between the visual features of the video frames of each video and the query semantic features. For example, the visual features of the video frames of at least one candidate video can be sorted from high to low according to the similarity score between the visual features and the query semantic features to obtain at least one sorted candidate video, and the top few videos in the sorted candidate video can be recommended to the user as the target video.
[0146] Therefore, the visual encoder obtained by using at least two video frames extracted from video data as unlabeled training samples for self-supervised pre-training in this embodiment can fully mine the semantic information of video data without relying on manually labeled tags. Thus, even when the description or tags of the video data are not closely related to the actual content of the video, the visual encoder can match the query semantic features of the user-input query text with the visual features of the video frames extracted by the visual encoder, without relying on the title or description of the video. This helps to avoid retrieving irrelevant video content, achieve more accurate text-video retrieval, and thus improve the user's product experience.
[0147] The specific embodiments of this application have been described in detail above with reference to the accompanying drawings. However, this application is not limited to the specific details of the above embodiments. Within the scope of the technical concept of this application, various simple modifications can be made to the technical solutions of this application, and these simple modifications all fall within the protection scope of this application. For example, the various specific technical features described in the above embodiments can be combined in any suitable manner without contradiction. To avoid unnecessary repetition, this application will not describe the various possible combinations separately. Furthermore, various different embodiments of this application can also be arbitrarily combined, as long as they do not violate the spirit of this application, they should also be considered as the content disclosed in this application.
[0148] It should also be understood that, in the various method embodiments of this application, the sequence numbers of the above processes do not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application. It should be understood that these sequence numbers can be interchanged where appropriate so that the embodiments of this application described can be implemented in a sequence other than those illustrated or described.
[0149] The method embodiments of this application have been described in detail above. The following description, in conjunction with... Figures 6 to 8 The following describes in detail the device embodiments of this application.
[0150] Figure 6 This is a schematic block diagram of a model training apparatus 600 according to an embodiment of this application. Figure 6As shown, the model training device 600 may include an acquisition unit 610, a first encoding network 620, a second encoding network 630, a calculation unit 640, and an update unit 650.
[0151] The acquisition unit 610 is used to acquire first video data and extract at least two video frames from the first video data.
[0152] A first encoding network 620 is used to take a first video frame from the at least two video frames as input and extract a first visual feature of the first video frame, wherein the first encoding network includes a visual encoder.
[0153] The second encoding network 630 is used to take the second video frame from the at least two video frames as input and extract the second visual features of the second video frame, wherein the first encoding network and the second encoding network include the same network structure, and the parameters of the second encoding network are updated by the parameters of the first encoding network based on the momentum shift value.
[0154] The calculation unit 640 is used to calculate the loss based on the first visual feature and the second visual feature;
[0155] The update unit 650 is used to update the parameters of the first encoding network according to the loss to obtain the trained visual encoder.
[0156] In some embodiments, the first video frame and the second video frame are two adjacent video frames.
[0157] In some embodiments, the first video frame and the second video frame are clusters of two adjacent video frames, wherein the video frame clustering is obtained based on at least two adjacent video frames.
[0158] In some embodiments, the acquisition unit 610 is further configured to: acquire second video data and extract a third video frame from the second video data;
[0159] The second coding network 630 is also used to: input the third video frame into the second coding network and extract the third visual features of the third video frame;
[0160] The calculation unit 640 is specifically used to: calculate the loss based on the first visual feature, the second visual feature and the third visual feature, wherein the first visual feature and the second visual feature are positive sample features and the third visual feature is a negative sample feature.
[0161] In some embodiments, the device 600 further includes a negative sample queue for storing second visual features.
[0162] In some embodiments, the first encoding network 620 is further configured to take the second video frame as input and extract a fourth visual feature of the second video frame;
[0163] The second coding network 630 is also used to take the first video frame as input and extract the fifth visual feature of the first video frame;
[0164] Specifically, the calculation unit 640 is used to: calculate the symmetry loss based on the first visual feature, the second visual feature, the fourth visual feature, and the fifth visual feature, and to determine the symmetry loss as the loss.
[0165] In some embodiments, the video encoder includes a visual Transformer or a convolutional neural network (CNN).
[0166] It should be understood that the device embodiments and method embodiments can correspond to each other, and similar descriptions can be referred to the method embodiments. To avoid repetition, further details will not be provided here. Specifically, Figure 6 The apparatus 600 shown can execute the above-described method embodiments, and the foregoing and other operations and / or functions of each module in the apparatus 600 are respectively for implementing the corresponding processes in the above-described method 300. For the sake of brevity, they will not be described in detail here.
[0167] Figure 7 This is a schematic block diagram of a video query device 700 according to an embodiment of this application. Figure 7 As shown, the device 700 may include an acquisition unit 710, a visual encoder 720, and a determination unit 730.
[0168] The acquisition unit 710 is used to acquire the query text and extract semantic features from the query text to obtain query semantic features;
[0169] The acquisition unit 710 is also used to acquire at least one candidate video;
[0170] A visual encoder 720 is used to extract visual features of video frames of the candidate video, wherein the visual encoder is trained according to the training model method 200 provided in the embodiments of this application;
[0171] The determining unit 730 is configured to determine the target video corresponding to the query text among the at least one candidate video based on the visual features and the query semantic features.
[0172] It should be understood that the device embodiments and method embodiments can correspond to each other, and similar descriptions can be referred to the method embodiments. To avoid repetition, further details will not be provided here. Specifically, Figure 7The apparatus 700 shown can execute the above-described method embodiments, and the aforementioned and other operations and / or functions of each module in the apparatus 700 are respectively for implementing the corresponding processes in the above-described method 500. For the sake of brevity, they will not be described in detail here.
[0173] The apparatus of this application embodiment has been described above from the perspective of functional modules in conjunction with the accompanying drawings. It should be understood that this functional module can be implemented in hardware, in software instructions, or in a combination of hardware and software modules. Specifically, the steps of the method embodiments in this application can be completed by integrated logic circuits in the processor's hardware and / or by software instructions. The steps of the method disclosed in this application embodiment can be directly embodied as being executed by a hardware decoding processor, or by a combination of hardware and software modules in the decoding processor. Optionally, the software module can reside in a mature storage medium in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, etc. This storage medium is located in memory, and the processor reads information from the memory and, in conjunction with its hardware, completes the steps in the above method embodiments.
[0174] Figure 8 This is a schematic block diagram of the electronic device 800 provided in the embodiments of this application.
[0175] like Figure 8 As shown, the electronic device 800 may include:
[0176] The system includes a memory 810 and a processor 820. The memory 810 stores computer programs and transfers the program code to the processor 820. In other words, the processor 820 can retrieve and run the computer program from the memory 810 to implement the methods described in the embodiments of this application.
[0177] For example, the processor 820 can be used to execute the above-described method embodiments according to instructions in the computer program.
[0178] In some embodiments of this application, the processor 820 may include, but is not limited to:
[0179] General-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.
[0180] In some embodiments of this application, the memory 810 includes, but is not limited to:
[0181] Volatile memory and / or non-volatile memory. Non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as Static RAM (SRAM), Dynamic RAM (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDR SDRAM), Enhanced Synchronous DRAM (ESDRAM), Synchronous Link DRAM (SLDRAM), and Direct Rambus RAM (DR RAM).
[0182] In some embodiments of this application, the computer program may be divided into one or more modules, which are stored in the memory 810 and executed by the processor 820 to perform the method provided in this application. The one or more modules may be a series of computer program instruction segments capable of performing a specific function, which describe the execution process of the computer program in the electronic device.
[0183] like Figure 8 As shown, the electronic device 800 may further include:
[0184] Transceiver 830, which can be connected to processor 820 or memory 810.
[0185] The processor 820 can control the transceiver 830 to communicate with other devices; specifically, it can send information or data to other devices or receive information or data sent by other devices. The transceiver 830 may include a transmitter and a receiver. The transceiver 830 may further include antennas, and the number of antennas may be one or more.
[0186] It should be understood that the various components in the electronic device are connected through a bus system, which includes a data bus, a power bus, a control bus, and a status signal bus.
[0187] This application also provides a computer storage medium storing a computer program thereon, which, when executed by a computer, enables the computer to perform the methods of the above-described method embodiments. Alternatively, embodiments of this application also provide a computer program product containing instructions that, when executed by a computer, cause the computer to perform the methods of the above-described method embodiments.
[0188] When implemented using software, it can be implemented entirely or partially as a computer program product. This computer program product includes one or more computer instructions. When these computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium accessible to a computer or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., digital video disc (DVD)), or a semiconductor medium (e.g., solid-state disk (SSD)).
[0189] It is understood that in the specific implementation of this application, when the above embodiments of this application are applied to specific products or technologies and involve user information and other related data, user permission or consent is required, and the collection, use and processing of related data must comply with relevant laws, regulations and standards.
[0190] Those skilled in the art will recognize that the modules and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.
[0191] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or modules may be electrical, mechanical, or other forms.
[0192] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical modules; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. For example, the functional modules in the various embodiments of this application may be integrated into one processing module, or each module may exist physically separately, or two or more modules may be integrated into one module.
[0193] The above are merely specific embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.
Claims
1. A model training method, characterized in that, include: Acquire first video data and extract at least two video frames from the first video data; The first video frame of the at least two video frames is input into the first encoding network to extract the first visual features of the first video frame, wherein the first encoding network includes a visual encoder. The second video frame of the at least two video frames is input into the second coding network to extract the second visual features of the second video frame. The first coding network and the second coding network have the same network structure, and the parameters of the second coding network are updated by the parameters of the first coding network based on the momentum shift value. The loss is calculated based on the first visual feature and the second visual feature; wherein the first visual feature and the second visual feature are positive sample features; Based on the loss, the parameters of the first encoding network are updated to obtain the trained visual encoder; Wherein, the first video frame and the second video frame are two adjacent video frames; or The first video frame and the second video frame are clusters of two adjacent video frames, wherein the video frame clustering is obtained based on at least two adjacent video frames.
2. The method according to claim 1, characterized in that, The method further includes: Acquire the second video data and extract the third video frame from the second video data; The third video frame is input into the second encoding network to extract the third visual features of the third video frame; The step of calculating the loss based on the first visual feature and the second visual feature includes: The loss is calculated based on the first visual feature, the second visual feature, and the third visual feature, wherein the third visual feature is a negative sample feature.
3. The method according to claim 1 or 2, characterized in that, Also includes: The second visual feature is added to the negative sample queue, wherein the negative sample queue is used to store negative sample features.
4. The method according to claim 1 or 2, characterized in that, Also includes: The second video frame is input into the first encoding network to extract the fourth visual features of the second video frame; The first video frame is input into the second encoding network to extract the fifth visual feature of the first video frame; The step of calculating the loss based on the first visual feature and the second visual feature includes: Calculate the symmetry loss based on the first visual feature, the second visual feature, the fourth visual feature, and the fifth visual feature; The symmetry loss is defined as the loss.
5. The method according to claim 1 or 2, characterized in that, The video encoder includes a visual Transformer or a convolutional neural network (CNN).
6. A video query method, characterized in that, include: Obtain the query text and extract its semantic features to obtain the query semantic features; Obtain at least one candidate video; Visual features of video frames of the candidate video are extracted using a visual encoder, wherein the visual encoder is trained according to the method described in any one of claims 1-5; Based on the visual features and the query semantic features, the target video corresponding to the query text is determined from the at least one candidate video.
7. A model training device, characterized in that, include: An acquisition unit is used to acquire first video data and extract at least two video frames from the first video data. A first encoding network is used to take a first video frame from the at least two video frames as input and extract a first visual feature of the first video frame, wherein the first encoding network includes a visual encoder; A second encoding network is used to take a second video frame from the at least two video frames as input and extract a second visual feature of the second video frame. The first encoding network and the second encoding network have the same network structure, and the parameters of the second encoding network are updated by the parameters of the first encoding network based on the momentum shift value. The calculation unit is used to calculate the loss based on the first visual feature and the second visual feature; wherein the first visual feature and the second visual feature are positive sample features; An update unit is used to update the parameters of the first encoding network according to the loss, so as to obtain the trained visual encoder. Wherein, the first video frame and the second video frame are two adjacent video frames; or The first video frame and the second video frame are clusters of two adjacent video frames, wherein the video frame clustering is obtained based on at least two adjacent video frames.
8. A video query device, characterized in that, include: The acquisition unit is used to acquire the query text and extract semantic features from the query text to obtain query semantic features; The acquisition unit is also used to acquire at least one candidate video; A visual encoder for extracting visual features from video frames of the candidate video, wherein the visual encoder is trained according to the method described in any one of claims 1-5; A determining unit is configured to determine the target video corresponding to the query text from the at least one candidate video based on the visual features and the query semantic features.
9. An electronic device, characterized in that, The method includes a processor and a memory, wherein the memory stores instructions, and when the processor executes the instructions, it causes the processor to perform the method according to any one of claims 1-6.
10. A computer storage medium, characterized in that, Used to store a computer program, the computer program including a method for performing any one of claims 1-6.
11. A computer program product, characterized in that, It includes computer program code that, when executed by an electronic device, causes the electronic device to perform the method of any one of claims 1-6.