A feature extraction model training method and device, equipment and storage medium

Abstract

The application discloses a feature extraction model training method based on artificial intelligence technology, comprising the following steps: obtaining T groups of training samples, wherein each group of training samples comprises text information samples and video segment samples with a corresponding relationship; for each group of training samples, obtaining a text semantic vector by using a to-be-trained text feature extraction model; for each group of training samples, obtaining a visual semantic vector by using a to-be-trained visual feature extraction model; determining positive samples and negative samples according to the text semantic vector and the visual semantic vector corresponding to each group of training samples; and training the to-be-trained text feature extraction model and the to-be-trained visual feature extraction model according to the positive samples and the negative samples. The application also provides related devices, equipment and storage media. The application can realize the extraction of self-supervised multi-modal video features by comparing the positive samples and the negative samples for model training, so that the training set does not need to be manually labeled, thereby reducing the model training cost.

Description

Technical Field

[0001] This application relates to artificial intelligence technology, and more particularly to a method, apparatus, device, and storage medium for training a feature extraction model. Background Technology

[0002] With the development of multimedia and network technologies, a large amount of multimedia information has been generated, including images, audio, and video, with video showing particularly rapid growth. Videos present a wide variety of rich and diverse content, thus carrying a wealth of information and being more easily perceived and understood.

[0003] Video processing technology is an important technology with a wide range of applications, such as video classification and retrieval. These applications often require feature extraction from the video. In traditional methods, video frames are first extracted, then text and image features are extracted from the video frames, and finally, the text and image features are fused together to obtain the multimodal features of the video frame.

[0004] The inventors discovered that traditional solutions suffer from at least the following problems: Videos often exhibit a mismatch between image and text information. For example, in a video frame, a building might appear on screen, but the subtitles might be unrelated to the content. In such cases, significant manpower is required to verify the image and text information during model training, leading to high costs associated with manual data annotation. Summary of the Invention

[0005] This application provides a training method, apparatus, device, and storage medium for a feature extraction model. By comparing positive and negative samples during model training, this application enables self-supervised multimodal video feature extraction, thus eliminating the need for manual annotation of the training set and reducing model training costs.

[0006] In view of this, this application provides a method for model training, including:

[0007] Obtain T sets of training samples, where each set of training samples includes text information samples and video clip samples with corresponding relationships, and T is an integer greater than 1;

[0008] For each set of training samples, the text semantic vector corresponding to the text information sample is obtained through the text feature extraction model to be trained;

[0009] For each set of training samples, the visual semantic vector corresponding to the video segment sample is obtained through the visual feature extraction model to be trained;

[0010] Based on the text semantic vector and visual semantic vector corresponding to each group of training samples, at least one group of positive samples and at least one group of negative samples are determined. Each group of positive samples includes the text semantic vector and visual semantic vector corresponding to the same group of training samples, and each group of negative samples includes the text semantic vector and visual semantic vector corresponding to different groups of training samples.

[0011] Based on at least one set of positive samples and at least one set of negative samples, update the model parameters of the text feature extraction model and the visual feature extraction model to be trained until the model training conditions are met, and obtain the text feature extraction model and the visual feature extraction model.

[0012] This application also provides a method for model training, including:

[0013] Obtain the text information to be trained and the video segment to be trained, wherein the text information to be trained and the video segment to be trained have a corresponding relationship, and the video segment to be trained has a labeled classification label;

[0014] The semantic vector of the text to be trained is obtained by means of the text feature extraction model, wherein the text feature extraction model is trained by any one of the methods 1 to 11 above;

[0015] The visual semantic vector corresponding to the video segment to be trained is obtained by a visual feature extraction model, wherein the visual feature extraction model is trained using any one of the methods 1 to 11 above.

[0016] The semantic vectors of the text to be trained and the visual semantic vectors to be trained are concatenated to obtain the multimodal feature vectors to be trained.

[0017] Based on the multimodal feature vector to be trained, the classification probability distribution is obtained through the video classification model to be trained;

[0018] Based on the classification probability distribution and classification labels, the model parameters of the video classification model to be trained are updated until the model training conditions are met, thus obtaining the video classification model.

[0019] This application also provides a model training apparatus, comprising:

[0020] The acquisition module is used to acquire T sets of training samples, where each set of training samples includes text information samples and video clip samples with corresponding relationships, and T is an integer greater than 1;

[0021] The acquisition module is also used to obtain the text semantic vector corresponding to the text information sample through the text feature extraction model to be trained for each group of training samples.

[0022] The acquisition module is also used to obtain the visual semantic vector corresponding to the video segment sample through the visual feature extraction model to be trained for each group of training samples.

[0023] The determination module is used to determine at least one set of positive samples and at least one set of negative samples based on the text semantic vector and visual semantic vector corresponding to each set of training samples. Each set of positive samples includes the text semantic vector and visual semantic vector corresponding to the same set of training samples, and each set of negative samples includes the text semantic vector and visual semantic vector corresponding to different sets of training samples.

[0024] The training module is used to update the model parameters of the text feature extraction model and the visual feature extraction model to be trained based on at least one set of positive samples and at least one set of negative samples, until the model training conditions are met, and thus obtain the text feature extraction model and the visual feature extraction model.

[0025] In one possible design, in another implementation of another aspect of the embodiments of this application,

[0026] The acquisition module is specifically used to acquire video samples;

[0027] Perform optical character recognition (OCR) processing on each video frame in the video sample to obtain the text recognition result for each video frame;

[0028] If there are consecutive video frames in the video samples that belong to the same text recognition result, then the consecutive video frames are regarded as video segment samples included in a set of training samples, and the same text recognition result is regarded as text information samples included in a set of training samples.

[0029] In one possible design, in another implementation of another aspect of the embodiments of this application,

[0030] The acquisition module is specifically used to acquire video samples and the corresponding subtitle files of the video samples. The subtitle files include subtitle information, the start frame corresponding to the subtitle information, and the end frame corresponding to the subtitle information.

[0031] Based on the subtitle file, the subtitle information is used as a set of text information samples included in the training samples, and the continuous video frames from the start frame to the end frame are used as a set of video segment samples included in the training samples.

[0032] In one possible design, in another implementation of another aspect of the embodiments of this application,

[0033] The acquisition module is specifically used to acquire text sequence samples based on text information samples;

[0034] Based on text sequence samples, text semantic vectors are obtained through a text feature extraction model to be trained.

[0035] The acquisition module is specifically used to acquire video sequence samples based on video clip samples;

[0036] Based on video sequence samples, visual semantic vectors are obtained through a visual feature extraction model to be trained.

[0037] In one possible design, in another implementation of another aspect of the embodiments of this application,

[0038] The acquisition module is specifically used to perform word segmentation on the text information sample to obtain at least one text unit of the text information sample;

[0039] Encode each text unit in at least one text unit to obtain at least one text unit feature, wherein the text unit feature has a one-to-one correspondence with the text unit;

[0040] Generate text sequence samples based on at least one text unit feature.

[0041] In one possible design, in another implementation of another aspect of the embodiments of this application,

[0042] The acquisition module is specifically used to acquire the number of text units corresponding to the text information sample;

[0043] Based on the number of text units, a set of video frames is obtained from the video clip samples, wherein the number of video frames in the set of video frames is equal to the number of text units;

[0044] Each video frame in the video frame set is subjected to feature extraction processing to obtain at least one three-dimensional video frame feature, wherein the three-dimensional video frame feature has a one-to-one correspondence with the video frame;

[0045] For each of the at least one three-dimensional video frame features, feature transformation is performed to obtain at least one two-dimensional video frame feature, wherein the two-dimensional video frame feature and the three-dimensional video frame feature have a one-to-one correspondence.

[0046] Position encoding is performed on each of the at least one two-dimensional video frame features to obtain at least one video frame feature, wherein there is a one-to-one correspondence between the video frame features and the two-dimensional video frame features.

[0047] Generate video sequence samples based on at least one video frame feature.

[0048] In one possible design, in another implementation of another aspect of the embodiments of this application,

[0049] The acquisition module is specifically used to perform image analysis on each video frame in the video clip sample to obtain the image evaluation value corresponding to each video frame. The image evaluation value is used to evaluate the quality of the video frame.

[0050] The video frames in the video clip samples are sorted in descending order of image evaluation value, and the number of video frames equal to the number of text units is selected as the video frame set.

[0051] In one possible design, in another implementation of another aspect of the embodiments of this application,

[0052] The acquisition module is specifically used to acquire the first feature vector based on text sequence samples through the first temporal network included in the text feature extraction model to be trained;

[0053] Based on the first feature vector, the text semantic vector is obtained through the first multilayer perceptron network included in the text feature extraction model to be trained;

[0054] The acquisition module is specifically used to acquire a second feature vector based on video sequence samples through the second temporal network included in the visual feature extraction model to be trained, wherein the second temporal network has the same network structure as the first temporal network;

[0055] Based on the second feature vector, the visual semantic vector is obtained through the second multilayer perceptron network included in the visual feature extraction model to be trained.

[0056] In one possible design, in another implementation of another aspect of the embodiments of this application,

[0057] The acquisition module is specifically used to acquire the first feature vector based on text sequence samples through the first temporal network included in the text feature extraction model to be trained;

[0058] Based on the first feature vector, the text semantic vector is obtained through the first multilayer perceptron network included in the text feature extraction model to be trained;

[0059] The acquisition module is specifically used to acquire a second feature vector based on video sequence samples through the second temporal network included in the visual feature extraction model to be trained, wherein the second temporal network has a different network structure from the first temporal network;

[0060] Based on the second feature vector, the visual semantic vector is obtained through the second multilayer perceptron network included in the visual feature extraction model to be trained.

[0061] In one possible design, in another implementation of another aspect of the embodiments of this application, the T training samples include a first training sample and a second training sample, wherein the first training sample includes a first text information sample and a first video segment sample with a corresponding relationship, and the second training sample includes a second text information sample and a second video segment sample with a corresponding relationship.

[0062] The acquisition module is specifically used to acquire the first text semantic vector corresponding to the first text information sample through the text feature extraction model to be trained, and to acquire the second text semantic vector corresponding to the second text information sample through the text feature extraction model to be trained.

[0063] The acquisition module is specifically used to acquire the first visual semantic vector corresponding to the first video segment sample through the visual feature extraction model to be trained, and to acquire the second visual semantic vector corresponding to the second video segment sample through the visual feature extraction model to be trained.

[0064] The determination module is specifically used to take the first visual semantic vector and the first text semantic vector as a set of positive samples, and the second visual semantic vector and the second text semantic vector as another set of positive samples;

[0065] The first visual semantic vector and the second text semantic vector are used as one set of negative samples, and the second visual semantic vector and the first text semantic vector are used as another set of negative samples.

[0066] In one possible design, in another implementation of another aspect of the embodiments of this application, the model training device further includes a generation module and a processing module;

[0067] The acquisition module is also used to acquire target text information and target video segments after obtaining the text feature extraction model and the visual feature extraction model until the model training conditions are met, where the target text information and the target video segments have a corresponding relationship.

[0068] The generation module is used to generate a target text sequence based on the target text information. The target text sequence includes L text unit features, and the text unit features have a one-to-one correspondence with the text units in the target text information. L is an integer greater than or equal to 1.

[0069] The acquisition module is further configured to acquire L video frames from the target text information based on the number of text units included in the target text sequence, and generate a target video sequence based on the L video frames. The target video sequence includes L video frame features, and the video frame features have a one-to-one correspondence with the video frames acquired from the target video segment.

[0070] The acquisition module is also used to obtain the semantic vector of the target text based on the target text sequence through a text feature extraction model;

[0071] The acquisition module is also used to obtain the target visual semantic vector based on the target video sequence through a visual feature extraction model;

[0072] The processing module is used to concatenate the target text semantic vector and the target visual semantic vector to obtain a multimodal feature vector.

[0073] This application also provides a model training apparatus, comprising:

[0074] The acquisition module is used to acquire the text information to be trained and the video segments to be trained. The text information to be trained and the video segments to be trained have a corresponding relationship, and the video segments to be trained have labeled classification tags.

[0075] The acquisition module is also used to acquire the semantic vector of the text to be trained corresponding to the text information to be trained through the text feature extraction model, wherein the text feature extraction model is obtained by performing the above-mentioned methods;

[0076] The acquisition module is also used to acquire the visual semantic vector to be trained corresponding to the video segment to be trained through a visual feature extraction model, wherein the visual feature extraction model is obtained by performing the above-mentioned methods;

[0077] The processing module is used to concatenate the text semantic vector to be trained and the visual semantic vector to be trained to obtain the multimodal feature vector to be trained.

[0078] The acquisition module is also used to obtain the classification probability distribution based on the multimodal feature vector to be trained through the video classification model to be trained;

[0079] The training module is used to update the model parameters of the video classification model to be trained based on the classification probability distribution and classification labels until the model training conditions are met, thus obtaining the video classification model.

[0080] This application also provides a computer device, including: a memory, a processor, and a bus system;

[0081] The memory is used to store programs;

[0082] The processor is used to execute programs in memory, and the processor is used to execute the methods mentioned above according to the instructions in the program code;

[0083] Bus systems are used to connect memory and processor to enable communication between them.

[0084] Another aspect of this application provides a computer-readable storage medium storing instructions that, when executed on a computer, cause the computer to perform the methods described above.

[0085] Another aspect of this application provides a computer program product or computer program including computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the methods provided in the above aspects.

[0086] As can be seen from the above technical solutions, the embodiments of this application have the following advantages:

[0087] This application provides a method for model training. First, T sets of training samples are obtained, each set including text information samples and video segment samples with corresponding relationships. For each set of training samples, text semantic vectors are obtained through a text feature extraction model to be trained, and visual semantic vectors are obtained through a visual feature extraction model. Based on this, at least one set of positive samples and at least one set of negative samples are determined. Finally, the model parameters of the text feature extraction model and the visual feature extraction model to be trained are updated using these positive and negative samples until the model training conditions are met, thus obtaining the trained text feature extraction model and visual feature extraction model. By collecting text semantic vectors and visual semantic vectors from different video segments in the above manner, multiple sets of positive and negative samples are constructed. By comparing positive and negative samples for model training, self-supervised multimodal video feature extraction can be achieved. Therefore, manual annotation of the training set is not required, thereby reducing the model training cost. Attached Figure Description

[0088] Figure 1 This is a schematic diagram of the architecture of the model training system in an embodiment of this application;

[0089] Figure 2 This is another schematic diagram of the model training system in the embodiments of this application;

[0090] Figure 3 This is a flowchart illustrating the feature extraction model training method in an embodiment of this application.

[0091] Figure 4 This is a schematic diagram of a training sample in an embodiment of this application;

[0092] Figure 5 This is a schematic diagram illustrating the extraction of text sequence samples in an embodiment of this application;

[0093] Figure 6 This is a schematic diagram illustrating the extraction of video sequence samples in an embodiment of this application;

[0094] Figure 7 This is a schematic diagram of training a feature extraction model in an embodiment of this application;

[0095] Figure 8 This is another flowchart illustrating the feature extraction model training method in the embodiments of this application;

[0096] Figure 9 This is a schematic diagram illustrating the implementation of a video classification task in an embodiment of this application;

[0097] Figure 10 This is a schematic diagram of a model training device in an embodiment of this application;

[0098] Figure 11 This is another schematic diagram of the model training device in the embodiments of this application;

[0099] Figure 12 This is a schematic diagram of the server structure in an embodiment of this application;

[0100] Figure 13 This is a schematic diagram of the structure of a terminal device in an embodiment of this application. Detailed Implementation

[0101] This application provides a training method, apparatus, device, and storage medium for a feature extraction model. By comparing positive and negative samples during model training, this application enables self-supervised multimodal video feature extraction, thus eliminating the need for manual annotation of the training set and reducing model training costs.

[0102] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a particular order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented, for example, in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “corresponding to,” and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0103] To enable computers to acquire knowledge that humans can understand, a multi-layered network of simple concepts needs to be constructed to define complex objects. Through iterative computation and training on this network, the computer can grasp the characteristics of the object. With the rapid development of video technology, the extraction and application of video features have received increasing attention. The following section will introduce the application methods of video features in specific scenarios.

[0104] I. Video Inspection;

[0105] For example, the video application backend inspects user-uploaded videos. Specifically, for each uploaded video, it extracts the visual semantic vector and the text semantic vector corresponding to the subtitles as video features. These video features are then used as input to a video classification model, which outputs the video's category. Alternatively, it can detect whether the video contains non-compliant content. Videos are automatically categorized and uploaded based on their category. Videos containing non-compliant content can be automatically removed or submitted for manual review.

[0106] II. Facial recognition;

[0107] For example, when a user activates a card using facial recognition, they need to face the camera and say the content specified by the subtitles (e.g., "Hello"), thus recording video A. Visual semantic vectors and text semantic vectors corresponding to the subtitles are extracted from video A to form the user's user feature A. If the user needs to cancel or report a lost card, they again face the camera and say the content specified by the subtitles (e.g., "Hello"), thus recording video B. Visual semantic vectors and text semantic vectors corresponding to the subtitles are extracted from video B to form the user's user feature B. If user feature A and user feature B match successfully, the corresponding service can be performed (e.g., canceling or reporting a lost card).

[0108] Typically, video feature extraction relies on a feature extraction model. Therefore, this application proposes a training method for a feature extraction model that enables self-supervised learning and eliminates the need for manually labeled training sets. The feature extraction model training method proposed in this application can be applied to… Figure 1 Please refer to the model training system shown. Figure 1 , Figure 1This is a schematic diagram of the model training system architecture in an embodiment of this application. As shown in the figure, the model training system includes a terminal device, and a client is deployed on the terminal device. The client can run on the terminal device via a browser or as a standalone application (APP), etc. The specific presentation of the client is not limited here. The terminal device can train the feature extraction model locally and store the trained feature extraction model locally for subsequent business calls. The feature extraction model includes two parts: a text feature extraction model and a visual feature extraction model.

[0109] The feature extraction model training method adopted in this application can also be applied to… Figure 2 Please refer to the model training system shown. Figure 2 , Figure 2 This is another architectural diagram of the model training system in this application embodiment. As shown in the figure, the model training system includes a server and terminal devices. The server involved in this application can be an independent physical server, a server cluster composed of multiple physical servers, or a distributed system. It can also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms. The terminal devices can be smartphones, tablets, laptops, PDAs, personal computers, smart TVs, smartwatches, in-vehicle devices, wearable devices, etc., but are not limited to these. The terminal devices and the server can be directly or indirectly connected through wired or wireless communication, which is not limited in this application. The number of servers and terminal devices is also not limited. The solution provided in this application can be completed independently by the terminal device, independently by the server, or jointly by the terminal device and the server, which is not specifically limited in this application. Terminal device A can train a feature extraction model locally and upload the trained feature extraction model to the server for other terminal devices (e.g., terminal device B) to download and use.

[0110] The technical solutions provided in this application specifically involve machine learning (ML), computer vision (CV), and natural language processing (NLP) technologies within the field of artificial intelligence (AI). AI utilizes digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceiving the environment, acquiring knowledge, and using that knowledge to obtain optimal results—theories, methods, technologies, and application systems. In other words, AI is a comprehensive technology within computer science that attempts to understand the essence of intelligence and produce a new type of intelligent machine capable of reacting in a manner similar to human intelligence. AI studies the design principles and implementation methods of various intelligent machines, enabling them to possess perception, reasoning, and decision-making capabilities. AI technology is a comprehensive discipline involving a wide range of fields, encompassing both hardware and software technologies. Basic AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technologies, operating / interactive systems, and mechatronics. AI software technologies mainly include CV technology, speech processing technology, NLP technology, as well as machine learning / deep learning, autonomous driving, and intelligent transportation.

[0111] Computer vision (CV) is a science that studies how to enable machines to "see." More specifically, it refers to using cameras and computers to replace human eyes in recognizing, tracking, and measuring targets, and then performing image processing to create images more suitable for human observation or transmission to instruments. As a scientific discipline, CV studies related theories and technologies, attempting to build AI systems capable of extracting information from images or multidimensional data. CV technologies typically include image processing, image recognition, image semantic understanding, image retrieval, optical character recognition (OCR), video processing, video semantic understanding, video content / behavior recognition, 3D object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous localization and mapping (SLAM), autonomous driving, intelligent transportation, and other technologies, as well as common biometric recognition technologies such as facial recognition and fingerprint recognition.

[0112] Natural Language Processing (NLP) is an important area within computer science and AI. It studies the theories and methods for enabling effective communication between humans and computers using natural language. NLP is a science that integrates linguistics, computer science, and mathematics. Therefore, research in this field involves natural language—the language people use in daily life—and thus it has a close connection with linguistic research. NLP technologies typically include text processing, semantic understanding, machine translation, question answering, and knowledge graphs.

[0113] Machine Learning (ML) is a multidisciplinary field involving probability theory, statistics, approximation theory, convex analysis, and algorithm complexity theory. It specifically studies how computers can simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. ML is the core of AI and the fundamental way to endow computers with intelligence; its applications span all areas of AI. ML and deep learning typically include techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and instructional learning.

[0114] Based on the above introduction, the training method of the feature extraction model in this application will be described below. Please refer to [link / reference needed]. Figure 3 One embodiment of the feature extraction model training method in this application includes:

[0115] 110. Obtain T sets of training samples, where each set of training samples includes text information samples and video clip samples with corresponding relationships, and T is an integer greater than 1;

[0116] In one or more embodiments, the model training device segments video segments of varying lengths from the video, using individual text information samples (i.e., single-sentence subtitles) as units, thereby obtaining T sets of training samples, where T is an integer greater than 1.

[0117] Specifically, for ease of understanding, please refer to Figure 4 , Figure 4 This is a schematic diagram of training samples in an embodiment of this application. As shown in the figure, taking two sets of training samples as examples, training sample 1 includes the text information sample "We play games in the sunshine", and the video clip sample included in training sample 1 has P video frames. Training sample 2 includes the text information sample "The plants here are growing very lushly", and the video clip sample included in training sample 2 has Q video frames.

[0118] It should be noted that the model training device can be deployed on a server or terminal device, or on a system consisting of a server and a terminal device; no limitation is made here.

[0119] 120. For each set of training samples, obtain the text semantic vector corresponding to the text information sample through the text feature extraction model to be trained;

[0120] In one or more embodiments, the model training device acquires text information samples from the training samples, and then preprocesses the text information samples, such as performing word segmentation, text cleaning, and feature processing, removing punctuation marks, and deleting useless words. The preprocessed text information samples are used as input to the text feature extraction model to be trained, and the text feature extraction model outputs a text semantic vector.

[0121] It is understandable that for T sets of training samples, T text semantic vectors can be obtained.

[0122] 130. For each set of training samples, obtain the visual semantic vector corresponding to the video segment sample through the visual feature extraction model to be trained;

[0123] In one or more embodiments, the model training device acquires video segment samples from the training samples, and then preprocesses the video segment samples, for example, converting each frame of the video segment sample into two-dimensional features and then performing position encoding. The preprocessed video segment samples are used as input to the visual feature extraction model to be trained, and the visual feature extraction model outputs a visual semantic vector.

[0124] It is understandable that for T sets of training samples, T visual semantic vectors can be obtained.

[0125] 140. Based on the text semantic vector and visual semantic vector corresponding to each group of training samples, determine at least one group of positive samples and at least one group of negative samples. Each group of positive samples includes the text semantic vector and visual semantic vector corresponding to the same group of training samples, and each group of negative samples includes the text semantic vector and visual semantic vector corresponding to different groups of training samples.

[0126] In one or more embodiments, the model training device forms at least one set of positive samples and at least one set of negative samples based on the text semantic vector and visual semantic vector corresponding to each set of training samples.

[0127] Specifically, positive samples are positive examples. A set of positive samples includes the text semantic vectors and visual semantic vectors corresponding to the same set of training samples. For example, the text semantic vectors and visual semantic vectors generated based on training sample 1 belong to a set of positive samples. Negative samples are negative examples. A set of negative samples includes the text semantic vectors and visual semantic vectors corresponding to different sets of training samples. For example, the text semantic vectors generated based on training sample 1 and the visual semantic vectors generated based on training sample 2 belong to a set of negative samples.

[0128] It is understood that the ratio of positive sample groups to negative sample groups can be 1 to 64. This is only an illustration and should not be construed as a limitation of this application.

[0129] 150. Based on at least one set of positive samples and at least one set of negative samples, update the model parameters of the text feature extraction model and the visual feature extraction model to be trained until the model training conditions are met, and obtain the text feature extraction model and the visual feature extraction model.

[0130] In one or more embodiments, the model training device calculates a loss result based on at least one set of positive samples and at least one set of negative samples using a contrastive loss function. The loss result is then used for backpropagation to update the model parameters of the text feature extraction model and the visual feature extraction model. This process continues until the model training conditions are met, resulting in a trained text feature extraction model and a visual feature extraction model.

[0131] Specifically, the loss result can be calculated using the following contrastive learning loss function:

[0132]

[0133] Among them, L q This represents the loss result. q represents the visual semantic vector. k represents the textual semantic vector. + Let k represent a positive sample, that is, the text semantic vector k corresponding to the visual semantic vector q. - τ represents a negative sample, i.e., a text semantic vector k that does not correspond to the visual semantic vector q. τ represents the temperature coefficient, which adjusts the degree of attention given to difficult samples; a smaller temperature coefficient indicates a greater focus on separating the current sample from the most similar samples.

[0134] To facilitate calculation, the dot product result can be sharpened by multiplying corresponding bits and then adding the results. During training, the result can be adjusted to the interval (0,1).

[0135] It should be noted that, in one scenario, an exhaustion criterion can be used to determine whether the model training conditions are met. For example, an iteration threshold can be set; when the number of iterations reaches the threshold, it indicates that the model training conditions have been met. In another scenario, an observational criterion can be used to determine whether the model training conditions are met. For example, when the loss result has converged, it indicates that the model training conditions have been met.

[0136] This application provides a method for model training. By collecting textual and visual semantic vectors from different video segments as described above, multiple sets of positive and negative samples are constructed. Model training is performed by comparing the positive and negative samples, enabling self-supervised extraction of multimodal video features. Therefore, manual annotation of the training set is unnecessary, thus reducing model training costs.

[0137] Optionally, in the above Figure 3 Based on the corresponding embodiments, in another optional embodiment provided by this application, obtaining at least two sets of training samples may specifically include:

[0138] Obtain video samples;

[0139] Perform optical character recognition (OCR) processing on each video frame in the video sample to obtain the text recognition result for each video frame;

[0140] If there are consecutive video frames in the video samples that belong to the same text recognition result, then the consecutive video frames are regarded as video segment samples included in a set of training samples, and the same text recognition result is regarded as text information samples included in a set of training samples.

[0141] In one or more embodiments, a method for generating video samples based on OCR recognition is described. As can be seen from the foregoing embodiments, each set of training samples includes text information samples and video segment samples with corresponding relationships. Therefore, it is necessary to obtain each set of training samples before model training. It is understood that since the extraction method of each set of training samples is similar, this explanation only takes the extraction of one set of training samples as an example.

[0142] Specifically, a video sample is first acquired, and then OCR technology is used to recognize each video frame in the sample, typically focusing on recognizing subtitle information. Based on this, the text recognition result for each video frame can be obtained. For easier understanding, please refer to Table 1, which provides an illustration of the text recognition results corresponding to each video frame.

[0143] Table 1

[0144] Video frame number Text recognition results 0 none 1 Today is a good day. 2 Today is a good day. 3 Today is a good day. 4 Today is a good day. 5 Today is a good day. 6 Today is a good day. 7 Today is a good day. 8 Today is a good day. 9 none 10 none 11 Let's go hiking this weekend! … …

[0145] As can be seen, no text was recognized in video frame 0. Video frames 1 through 8 all recognized the same text, "Today is a good day," while video frame 9 also failed to recognize any text. Therefore, video frames 1 through 8 are consecutive video frames with the same text recognition result. Thus, these 8 consecutive video frames are taken as a video segment sample, and the corresponding text information sample is "Today is a good day."

[0146] Secondly, this application provides a method for generating video samples based on OCR recognition. This method allows for the identification of text information samples from video samples, and the extraction of corresponding video segment samples. Therefore, compared to manually extracting training samples, this method not only reduces model training costs but also improves the efficiency of sample components.

[0147] Optionally, in the above Figure 3 Based on the corresponding embodiments, in another optional embodiment provided by this application, obtaining at least two sets of training samples may specifically include:

[0148] Obtain video samples and corresponding subtitle files. The subtitle files include subtitle information, the start frame corresponding to the subtitle information, and the end frame corresponding to the subtitle information.

[0149] Based on the subtitle file, the subtitle information is used as a set of text information samples included in the training samples, and the continuous video frames from the start frame to the end frame are used as a set of video segment samples included in the training samples.

[0150] In one or more embodiments, a method for generating video samples based on subtitle files is described. As can be seen from the foregoing embodiments, each set of training samples includes text information samples and video clip samples with corresponding relationships. Therefore, it is necessary to obtain each set of training samples before model training. It is understood that since the extraction method of each set of training samples is similar, this explanation only takes the extraction of one set of training samples as an example.

[0151] Specifically, first, a video sample and its corresponding subtitle file are obtained. The subtitle file includes subtitle information, the start frame corresponding to the subtitle information, and the end frame corresponding to the subtitle information. For easier understanding, please refer to Table 2, which provides an illustration of the information contained in the subtitle file.

[0152] Table 2

[0153] Subtitle information start frame End frame Today is a good day. Video frame 1 Video frame number 8 Let's go hiking this weekend! Frame 11 Video frame number 25 There are many beautiful mushrooms on the mountain. Video frame number 66 Video frame number 75 It's almost afternoon before we know it. Video frame number 122 Video frame 145 … … …

[0154] As can be seen, taking the subtitle information "Today is a good day" as the text information sample, the corresponding video frames 1 to 8 are consecutive video frames with the same text recognition result. Therefore, these 8 consecutive video frames are taken as a video segment sample, and the corresponding text information sample is "Today is a good day".

[0155] Secondly, this application provides a method for generating video samples based on subtitle files. Using this method, corresponding video segment samples can be directly extracted from the video samples based on the subtitle files, and the subtitle information in the subtitle text can be used as the corresponding text information samples. Therefore, compared to manually extracting training samples, this method not only reduces model training costs but also improves the efficiency of the sample components.

[0156] Optionally, in the above Figure 3 Based on the corresponding embodiments, in another optional embodiment provided by this application, the text semantic vector corresponding to the text information sample is obtained through the text feature extraction model to be trained, which may specifically include...

[0157] Obtain text sequence samples based on text information samples;

[0158] Based on text sequence samples, text semantic vectors are obtained through a text feature extraction model to be trained.

[0159] The visual semantic vectors corresponding to video segment samples are obtained through a visual feature extraction model to be trained. Specifically, this can include:

[0160] Obtain video sequence samples from video clip samples;

[0161] Based on video sequence samples, visual semantic vectors are obtained through a visual feature extraction model to be trained.

[0162] In one or more embodiments, a method for generating text sequence samples is described. As can be seen from the foregoing embodiments, after feature processing, text information samples and video clip samples can be used as inputs to a text feature extraction model (or, a text feature extraction model) and a visual feature extraction model (or, a visual feature extraction model) to be trained, thereby enabling the model to recognize the content of the information.

[0163] Specifically, text information samples can be Chinese, English, or other types of text. These are segmented according to the conventions of the corresponding language, and then relevant text features are extracted to obtain the corresponding text sequence samples. Video clip samples are video segments extracted at a certain frame rate. For example, extracting a 2-second video clip at a frame rate of 24 frames per second constitutes a video clip sample, which includes 48 video frames. Based on this, a portion of video frames can be extracted from the video clip samples to form video sequence samples. There are various ways to select video frames from video sequence samples, which are not limited here.

[0164] Secondly, this application provides a method for feature extraction from text information samples and video clip samples. By using the above method, the text information samples and video clip samples need to be preprocessed before model training, which helps to improve the model's learning ability and enable the model to learn more valuable features.

[0165] Optionally, in the above Figure 3 Based on the corresponding embodiments, in another optional embodiment provided by this application, obtaining text sequence samples based on text information samples may specifically include:

[0166] The text information sample is segmented into words to obtain at least one text unit of the text information sample;

[0167] Encode each text unit in at least one text unit to obtain at least one text unit feature, wherein the text unit feature has a one-to-one correspondence with the text unit;

[0168] Generate text sequence samples based on at least one text unit feature.

[0169] In one or more embodiments, a method for generating text sequence samples is described. As can be seen from the foregoing embodiments, in the text modality part, the text information sample needs to be segmented according to language conventions. Assuming the text information sample is in English, it is segmented by word, thereby obtaining at least one text unit, where one text unit represents one word, for example, "nice". Assuming the text information sample is in Chinese, it can be segmented according to a dictionary, thereby obtaining at least one text unit, where one text unit represents one word, for example, "game".

[0170] Specifically, for ease of understanding, please refer to Figure 5 , Figure 5 This is a schematic diagram illustrating the extraction of text sequence samples in an embodiment of this application. As shown in the figure, frames "X+1" to "X+10" correspond to the same text information sample, namely "We play games in the sunlight". Based on this, six text units are obtained, namely "we", "in", "sunlight", "under", "play", and "game". Each text unit can then be encoded to obtain the text unit features corresponding to each text unit.

[0171] For example, word-to-vector (Word2Vec), the Continuous Bag-Of-Words (CBOW) model, or the Bidirectional Encoder Representation from Transformers (BERT) can be used to encode text units, obtaining D-dimensional text unit features, where D can be 300 dimensions. Based on this, assuming a text information sample includes L text units, a text sequence sample can be represented as:

[0172] {T i |T i ∈R D×L}, i∈{1,2,3,...,L};

[0173] Among them, T i Let represent the i-th text unit in the text information sample. D represents the feature dimension of a text unit. L represents the number of text units included in the text information sample.

[0174] Furthermore, this application provides a method for generating text sequence samples. Using this method, text information samples can be segmented into words, and then a pre-trained language representation model is used to encode the segmented text units, thereby obtaining word embedding representations (i.e., text unit features). Based on this, the neural network model can model more complex contexts and include richer semantic information in the text unit features.

[0175] Optionally, in the above Figure 3 Based on the corresponding embodiments, in another optional embodiment provided by this application, obtaining video sequence samples based on video segment samples may specifically include:

[0176] Obtain the number of text units corresponding to the text information sample;

[0177] Based on the number of text units, a set of video frames is obtained from the video clip samples, wherein the number of video frames in the set of video frames is equal to the number of text units;

[0178] Each video frame in the video frame set is subjected to feature extraction processing to obtain at least one three-dimensional video frame feature, wherein the three-dimensional video frame feature has a one-to-one correspondence with the video frame;

[0179] For each of the at least one three-dimensional video frame features, feature transformation is performed to obtain at least one two-dimensional video frame feature, wherein the two-dimensional video frame feature and the three-dimensional video frame feature have a one-to-one correspondence.

[0180] Position encoding is performed on each of the at least one two-dimensional video frame features to obtain at least one video frame feature, wherein there is a one-to-one correspondence between the video frame features and the two-dimensional video frame features.

[0181] Generate video sequence samples based on at least one video frame feature.

[0182] In one or more embodiments, a method for generating video sequence samples is described. As described in the foregoing embodiments, in the visual modality part, firstly, based on the number of text units, a corresponding number of video frames are obtained from the video segment samples to form a video frame set. Then, feature extraction processing is performed on each video frame in the video frame set to obtain a three-dimensional video frame feature sequence {X}. i |X i ∈R c×h×w}, where i∈{1,2,3,...L}. Based on this, feature transformation is performed on the three-dimensional video frame features in the three-dimensional video frame feature sequence to obtain the two-dimensional video frame feature sequence {X}. i |X i ∈R c×n}, where n = h × w, i.e., x i ∈R c×n For two-dimensional video frame features, since the positions change after tiling, positional encoding is required, for example, using cos-sin two-dimensional positional encoding. Ultimately, the corresponding video frame features are obtained, and the sequence of these video frame features constitutes a video sequence sample.

[0183] Specifically, for ease of understanding, please refer to Figure 6 , Figure 6 This is a schematic diagram illustrating the extraction of video sequence samples in an embodiment of this application. As shown in the figure, frames "X+1" to "X+10" correspond to the same text information sample, namely, "We are playing games in the sunlight." Thus, six text units are obtained. Based on this, six video frames can be extracted from "X+1" to "X+10" to form a video frame set.

[0184] Furthermore, in this embodiment of the application, a method for generating video sequence samples is provided. Using the above method, based on the number of text units obtained after segmenting the text information samples, the same number of video frames are extracted. Thus, training objects with mapping relationships can be formed, which facilitates model training.

[0185] Optionally, in the above Figure 3 Based on the corresponding embodiments, in another optional embodiment provided by this application, obtaining a set of video frames from the video segment samples according to the number of text units may specifically include:

[0186] Image analysis is performed on each video frame in the video clip sample to obtain the image evaluation value corresponding to each video frame. The image evaluation value is used to evaluate the quality of the video frame.

[0187] The video frames in the video clip samples are sorted in descending order of image evaluation value, and the number of video frames equal to the number of text units is selected as the video frame set.

[0188] In one or more embodiments, a method for selecting a set of video frames is described. As can be seen from the foregoing embodiments, it is necessary to extract a certain number of video frames from the video segment samples. In one implementation, a corresponding number of video frames can be randomly selected from the video segment samples; in another implementation, video frames with better image quality can be selected preferentially.

[0189] Specifically, image analysis is performed on each video frame in the video clip sample. The analysis includes the overall brightness of the image, the sharpness of the picture, and the proportion of the target object (e.g., people, buildings, or animals) in the picture. Different scores are assigned to different analysis contents. For ease of understanding, please refer to Table 3, which is a schematic diagram of the image evaluation criteria.

[0190] Table 3

[0191]

[0192] Therefore, the image evaluation value of each video frame can be calculated based on the table above. The higher the image evaluation value, the better the quality of the video frame. Based on this, the video frames in the video segment sample are sorted in descending order of image evaluation value, and the number of video frames equal to the number of text units is selected as the video frame set.

[0193] For easier understanding, please refer to Table 4, which lists the video frames sorted from highest to lowest image evaluation value.

[0194] Table 4

[0195] Video frame number Image evaluation value 5 8 6 8 7 7 3 7 4 5 2 5 8 4 1 1 0 1

[0196] Based on Table 4, assuming that 6 video frames need to be extracted, then “video frame 5”, “video frame 6”, “video frame 7”, “video frame 3”, “video frame 4” and “video frame 2” should be extracted as a set of video frames.

[0197] Furthermore, in this embodiment of the application, a method for selecting a set of video frames is provided. By means of the above method, after performing image quality analysis on the video frames, video frames with better image quality are selected first for model training, which is beneficial to improving the model training effect.

[0198] Optionally, in the above Figure 3 Based on the corresponding embodiments, in another optional embodiment provided by this application, the text semantic vector is obtained based on the text sequence sample through a text feature extraction model to be trained, which may specifically include:

[0199] Based on text sequence samples, the first feature vector is obtained through the first temporal network included in the text feature extraction model to be trained;

[0200] Based on the first feature vector, the text semantic vector is obtained through the first multilayer perceptron network included in the text feature extraction model to be trained;

[0201] Based on video sequence samples, visual semantic vectors are obtained through a visual feature extraction model to be trained, which may include:

[0202] Based on video sequence samples, a second feature vector is obtained through the second temporal network included in the visual feature extraction model to be trained, wherein the second temporal network has the same network structure as the first temporal network;

[0203] Based on the second feature vector, the visual semantic vector is obtained through the second multilayer perceptron network included in the visual feature extraction model to be trained.

[0204] In one or more embodiments, a method for prediction using Siamese networks is introduced. As can be seen from the foregoing embodiments, the feature extraction model to be trained includes two parts: a text feature extraction model to be trained and a visual feature extraction model to be trained. The feature extraction model can adopt a Siamese network structure, that is, it consists of two encoders of temporal networks with the same structure.

[0205] Specifically, for ease of understanding, please refer to Figure 7 , Figure 7 This is a schematic diagram of training a feature extraction model in an embodiment of this application. As shown in the figure, text information samples (e.g., "We play games in the sun") and video clip samples are extracted. Based on this, text sequence samples are generated from the text information samples, and video sequence samples are generated from the video clip samples. The features of each text unit in the text sequence sample (i.e., T) are then used to extract the text information samples. i ∈R D×L The first feature vector is taken as input to the first temporal network in the text feature extraction model to be trained, thereby outputting the first feature vector. This first feature vector is then taken as input to the first multilayer perceptron (MLP) network in the text feature extraction model to be trained, thereby outputting the text semantic vector. Similarly, the features of each video frame in the video sequence sample (i.e., x) are taken as input. i ∈R c×nThe second feature vector is used as input to the second temporal network in the visual feature extraction model to be trained, thereby outputting the second feature vector. The second feature vector is then used as input to the second MLP network in the visual feature extraction model to be trained, thereby outputting the visual semantic vector.

[0206] It should be noted that both the first and second time-series networks can be Transformer networks, or Recurrent Neural Networks (RNNs), or Long Short-Term Memory (LSTM) networks, etc., without any restrictions here.

[0207] Furthermore, this application provides a method for prediction using Siamese networks. This method introduces the concept of self-supervised learning into multimodal deep learning, and utilizes the contrastive learning approach of Siamese networks to eliminate the dependence of model training on labeled data, thereby achieving the modeling of high-level semantic relationships in the temporal dimension of video frames. Based on this, a good representation of video images and text information can be learned, which is beneficial for its application in downstream tasks.

[0208] Optionally, in the above Figure 3 Based on the corresponding embodiments, in another optional embodiment provided by this application, the text semantic vector is obtained based on the text sequence sample through a text feature extraction model to be trained, which may specifically include:

[0209] Based on text sequence samples, the first feature vector is obtained through the first temporal network included in the text feature extraction model to be trained;

[0210] Based on the first feature vector, the text semantic vector is obtained through the first multilayer perceptron network included in the text feature extraction model to be trained;

[0211] Based on video sequence samples, visual semantic vectors are obtained through a visual feature extraction model to be trained, which may include:

[0212] Based on video sequence samples, a second feature vector is obtained through the second temporal network included in the visual feature extraction model to be trained, wherein the second temporal network has a different network structure from the first temporal network;

[0213] Based on the second feature vector, the visual semantic vector is obtained through the second multilayer perceptron network included in the visual feature extraction model to be trained.

[0214] In one or more embodiments, a method for prediction using a pseudo-Siamese network is introduced. As can be seen from the foregoing embodiments, the feature extraction model to be trained includes two parts: a text feature extraction model to be trained and a visual feature extraction model to be trained. The feature extraction model can adopt a pseudo-Siamese network structure, that is, it consists of encoders of two temporal networks with different structures.

[0215] Specifically, for better understanding, please refer to [the relevant documentation / reference]. Figure 7 The features of each text unit in the text sequence sample (i.e., T) i ∈R D×L The first feature vector is taken as input to the first temporal network in the text feature extraction model to be trained, thereby outputting the first feature vector. The first feature vector is then taken as input to the first MLP network in the text feature extraction model to be trained, thereby outputting the text semantic vector. Similarly, the features of each video frame in the video sequence sample (i.e., x) are taken as input. i ∈R c×n The second feature vector is used as input to the second temporal network in the visual feature extraction model to be trained, thereby outputting the second feature vector. The second feature vector is then used as input to the second MLP network in the visual feature extraction model to be trained, thereby outputting the visual semantic vector.

[0216] It should be noted that the first time-series network can be a Transformer network, and the second time-series network can be an LSTM network. Alternatively, the first time-series network can be an LSTM network, and the second time-series network can be a Transformer network. No restrictions are imposed here.

[0217] Furthermore, this application provides a method for prediction using a pseudo-Siamese network. This method introduces the concept of self-supervised learning into multimodal deep learning. By utilizing the contrastive learning approach with a pseudo-Siamese network, not only is the model training no longer dependent on labeled data, enabling the modeling of high-level semantic relationships in the temporal dimension of video frames, but it also improves the flexibility of the model structure compared to a Siamese network. Based on this, a good representation of video images and text information can be learned, which is beneficial for its application in downstream tasks.

[0218] Optionally, in the above Figure 3 Based on the corresponding embodiments, in another optional embodiment provided by the present application, the T training samples include a first training sample and a second training sample, wherein the first training sample includes a first text information sample and a first video segment sample with a corresponding relationship, and the second training sample includes a second text information sample and a second video segment sample with a corresponding relationship.

[0219] For each set of training samples, the text semantic vector corresponding to the text information sample is obtained through the text feature extraction model to be trained. Specifically, this may include:

[0220] The first text semantic vector corresponding to the first text information sample is obtained through the text feature extraction model to be trained, and the second text semantic vector corresponding to the second text information sample is obtained through the text feature extraction model to be trained.

[0221] For each set of training samples, the visual semantic vector corresponding to the video segment sample is obtained through the visual feature extraction model to be trained. Specifically, this may include:

[0222] The first visual semantic vector corresponding to the first video segment sample is obtained through the visual feature extraction model to be trained, and the second visual semantic vector corresponding to the second video segment sample is obtained through the visual feature extraction model to be trained.

[0223] Based on the text semantic vector and visual semantic vector corresponding to each set of training samples, determine at least one set of positive samples and at least one set of negative samples, which may include:

[0224] The first visual semantic vector and the first text semantic vector are used as one set of positive samples, and the second visual semantic vector and the second text semantic vector are used as another set of positive samples;

[0225] The first visual semantic vector and the second text semantic vector are used as one set of negative samples, and the second visual semantic vector and the first text semantic vector are used as another set of negative samples.

[0226] In one or more embodiments, a method for model training using positive and negative samples composed of different training samples is described. As can be seen from the foregoing embodiments, positive and negative samples can be recombined based on the text semantic vector and visual semantic vector corresponding to each group of training samples. For ease of explanation, the following description will use a first training sample and a second training sample as examples, wherein the first training sample includes a first text information sample and a first video segment sample, and the second training sample includes a second text information sample and a second video segment sample.

[0227] Specifically, after preprocessing the first and second text information samples, the preprocessed first text information sample is used as input to the text feature extraction model to be trained, thereby outputting a first text semantic vector. Similarly, after preprocessing the first and second video segment samples, the preprocessed first video segment sample is used as input to the visual feature extraction model to be trained, thereby outputting a first visual semantic vector. Similarly, the preprocessed second video segment sample is used as input to the visual feature extraction model to be trained, thereby outputting a second visual semantic vector.

[0228] Based on this, the first visual semantic vector and the first text semantic vector are considered as one group of positive samples, and the second visual semantic vector and the second text semantic vector are considered as another group of positive samples. Conversely, the first visual semantic vector and the second text semantic vector are considered as one group of negative samples, and the second visual semantic vector and the first text semantic vector are considered as another group of negative samples. It is understandable that, generally, the number of positive sample groups exceeds the number of negative sample groups.

[0229] Secondly, in this embodiment of the application, a method is provided to train a model by using different training samples to form positive and negative samples. Through the above method, based on the formed positive and negative samples, the feature extraction model can be trained by using a contrastive learning method. This not only realizes the extraction of self-supervised multimodal video features, but also reduces the cost of model training.

[0230] Optionally, in the above Figure 3 Based on the corresponding embodiments, in another optional embodiment provided by this application, after the model training conditions are met and the text feature extraction model and the visual feature extraction model are obtained, it may further include:

[0231] Obtain target text information and target video clips, wherein the target text information and target video clips have a corresponding relationship;

[0232] Generate a target text sequence based on the target text information, wherein the target text sequence includes L text unit features, and the text unit features have a one-to-one correspondence with the text units in the target text information, where L is an integer greater than or equal to 1;

[0233] Based on the number of text units included in the target text sequence, L video frames are extracted from the target text information, and a target video sequence is generated based on the L video frames. The target video sequence includes L video frame features, and each video frame feature has a one-to-one correspondence with the video frames extracted from the target video segment.

[0234] Based on the target text sequence, the semantic vector of the target text is obtained through a text feature extraction model;

[0235] Based on the target video sequence, the target visual semantic vector is obtained through a visual feature extraction model;

[0236] The target text semantic vector and the target visual semantic vector are concatenated to obtain a multimodal feature vector.

[0237] In one or more embodiments, a method for feature extraction using a multimodal feature extraction model is described. As can be seen from the foregoing embodiments, after training the text feature extraction model and the visual feature extraction model, features from the video can be extracted as the video's multimodal feature vector.

[0238] Specifically, taking the extraction of multimodal feature vectors from a video as an example, firstly, the video needs to be segmented according to text information (e.g., single-sentence subtitles), and then each text information and its corresponding video segment are obtained. Taking any text information (i.e., target text information) and its corresponding video segment (i.e., target video segment) as an example, the target text information is preprocessed, and then the features of each text unit are extracted to obtain the target text sequence. The video frames in the target video segment are preprocessed, and then the features of the video frames are extracted to obtain the target video sequence, wherein the number of video frames contained in the target video sequence is equal to the number of text units contained in the target text sequence (e.g., both are L).

[0239] Based on this, the target text sequence is used as input to the text feature extraction model, which outputs a target text semantic vector. Similarly, the target video sequence is used as input to the visual feature extraction model, which outputs a target visual semantic vector. By concatenating the target text semantic vector and the target visual semantic vector, a multimodal feature vector for a video segment can be obtained.

[0240] Assuming the video is divided into 100 video segments, in one implementation, the multimodal feature vectors corresponding to the 100 video segments can be averaged, and the averaged feature vector can be used as the multimodal feature vector of the entire video.

[0241] Secondly, this application provides a method for feature extraction using a multimodal feature extraction model. Through this method, after the text feature extraction model and the visual feature extraction model are trained, the visual information and subtitle information of the video can be directly input into the Siamese network in the form of images and text, respectively. The output obtained is a representation of the information in the video (i.e., a multimodal feature vector). Therefore, the generated feature extraction model can be flexibly applied to various downstream tasks or speech pre-training, saving the cost of training the feature extractor with a large amount of data. The pre-trained model can perform transfer learning on the adaptive and softmax layers using only a small dataset, further lowering the barrier to entry for using deep learning models and reducing the cost of using deep learning models for multimodal video feature extraction.

[0242] Based on the above introduction, the training method of the feature extraction model in this application will be described below. Please refer to [link / reference needed]. Figure 8 Another embodiment of the feature extraction model training method in this application includes:

[0243] 210. Obtain the text information to be trained and the video segment to be trained, wherein the text information to be trained and the video segment to be trained have a corresponding relationship, and the video segment to be trained has a labeled classification label;

[0244] In one or more embodiments, the model training device acquires video samples for training a video classification model. For ease of explanation, an example is taken of any text information (i.e., the text information to be trained) extracted from the video sample and its corresponding video segment (i.e., the video segment to be trained).

[0245] Specifically, assuming the video classification model is designed for emotion classification, it can distinguish whether a video belongs to the "happy emotion type," "normal emotion type," or "sad emotion type." Therefore, the video segments to be trained also need to be labeled; that is, the video segments to be trained must have pre-labeled classification tags.

[0246] It should be noted that the model training device can be deployed on a server or terminal device, or on a system consisting of a server and a terminal device; no limitation is made here.

[0247] 220. Obtain the semantic vector of the text to be trained corresponding to the text information to be trained through a text feature extraction model, wherein the text feature extraction model is trained using the model training method provided in any of the above embodiments;

[0248] In one or more embodiments, the model training device preprocesses the text information to be trained, such as performing word segmentation, text cleaning, and feature processing, removing punctuation marks, and deleting useless words. The preprocessed text information to be trained is then used as input to the text feature extraction model, which outputs a semantic vector of the text to be trained.

[0249] It should be noted that the training method of the text feature extraction model has been introduced in the foregoing embodiments, so it will not be repeated here.

[0250] 230. Obtain the visual semantic vector corresponding to the video segment to be trained through a visual feature extraction model, wherein the visual feature extraction model is trained using the model training method provided in any of the above embodiments;

[0251] In one or more embodiments, the model training device preprocesses the video segments to be trained, for example, by converting each frame of the video segment sample into two-dimensional features and then performing positional encoding. The preprocessed video segments to be trained are used as input to the visual feature extraction model, which outputs the visual semantic vector to be trained.

[0252] It should be noted that the training method of the visual feature extraction model has been introduced in the foregoing embodiments, so it will not be repeated here.

[0253] 240. The text semantic vector to be trained and the visual semantic vector to be trained are concatenated to obtain the multimodal feature vector to be trained.

[0254] In one or more embodiments, the model training device can concatenate the text semantic vector to be trained and the visual semantic vector to be trained to obtain the multimodal feature vector to be trained.

[0255] Specifically, assume the feature extraction model is G∈{G_i|i=v,t}, where G v G is a visual feature extraction model. t This is a text feature extraction model. Classification task data {x} i ,y i In the dataset}∈X×Y, X represents the training dataset (i.e., including the text information and video segments to be trained), and Y represents the training label set (i.e., the labeled classification labels). Let x... i It is divided into a visual component (i.e., the video segment to be trained). and the text portion (i.e., the text information to be trained). Enter them into G respectively v and G t In this process, the visual semantic vector to be trained is obtained. and the semantic vector of the text to be trained Visual semantic vectors to be trained and the semantic vector of the text to be trained By concatenating these components, we obtain the multimodal feature vector to be trained corresponding to the training data.

[0256] 250. Based on the multimodal feature vector to be trained, obtain the classification probability distribution through the video classification model to be trained;

[0257] In one or more embodiments, after obtaining the multimodal feature vector to be trained corresponding to the training data, the model training device inputs the multimodal feature vector to be trained into the video classification model to be trained, thereby obtaining the prediction result of the emotion classification task, that is, obtaining the classification probability distribution. The video classification model to be trained includes an adaptive layer and a softmax layer.

[0258] 260. Based on the classification probability distribution and classification labels, update the model parameters of the video classification model to be trained until the model training conditions are met, and obtain the video classification model.

[0259] In one or more embodiments, the model training device uses a loss function to calculate the classification probability distribution. Compared with the already labeled category tag y i The difference lies in the following: During gradient backpropagation, the parameters of the feature extraction model are fixed, meaning the model parameters of the feature extraction model are not updated; only the model parameters of the video classification model to be trained are updated, that is, only the parameters of the adaptive layer and the softmax layer are updated.

[0260] Specifically, for ease of understanding, please refer to Figure 9 , Figure 9 This is a schematic diagram illustrating the video classification task in an embodiment of this application. As shown in the figure, the number of adaptive layers must be determined based on the actual task and the size of the model parameters, and typically does not exceed three layers. Adaptive layers are essentially fully connected layers, which can perform non-linear transformations on the input vector, i.e.:

[0261] Y = f(WX + b);

[0262] Where Y represents the output, X represents the input, f(·) represents the activation function, W represents the weight matrix, and b represents the bias constant.

[0263] The softmax layer transforms the output of the adaptive layer into probabilities for various classification lists (e.g., sentiment categories). The calculation method is as follows:

[0264]

[0265] Among them, z j =Wxj x in +b j This is the output of the fully connected layer. W and b are the model parameters to be trained in this layer.

[0266] It should be noted that, in one scenario, an exhaustion criterion can be used to determine whether the model training conditions are met. For example, an iteration threshold can be set; when the number of iterations reaches the threshold, it indicates that the model training conditions have been met. In another scenario, an observational criterion can be used to determine whether the model training conditions are met. For example, when the loss result has converged, it indicates that the model training conditions have been met.

[0267] This application provides a method for model training. After the text feature extraction model and the visual feature extraction model are trained using the above method, the model can be applied to different downstream tasks as needed. For example, in video classification tasks, a video classification model (i.e., a classifier) ​​with good performance can be trained using the concatenated multimodal feature vectors and classification labels, and then applied in video classification scenarios.

[0268] The model training apparatus in this application is described in detail below. Please refer to [link / reference]. Figure 10 , Figure 10 This is a schematic diagram of one embodiment of the model training device in this application. The model training device 30 includes:

[0269] The acquisition module 310 is used to acquire T sets of training samples, wherein each set of training samples includes text information samples and video clip samples with corresponding relationships, and T is an integer greater than 1;

[0270] The acquisition module 310 is also used to acquire the text semantic vector corresponding to the text information sample through the text feature extraction model to be trained for each group of training samples.

[0271] The acquisition module 310 is also used to acquire the visual semantic vector corresponding to the video segment sample through the visual feature extraction model to be trained for each group of training samples.

[0272] The determination module 320 is used to determine at least one set of positive samples and at least one set of negative samples based on the text semantic vector and visual semantic vector corresponding to each set of training samples. Each set of positive samples includes the text semantic vector and visual semantic vector corresponding to the same set of training samples, and each set of negative samples includes the text semantic vector and visual semantic vector corresponding to different sets of training samples.

[0273] The training module 330 is used to update the model parameters of the text feature extraction model and the visual feature extraction model to be trained based on at least one set of positive samples and at least one set of negative samples, until the model training conditions are met, and thus obtain the text feature extraction model and the visual feature extraction model.

[0274] This application provides a model training apparatus. Using this apparatus, textual semantic vectors and visual semantic vectors from different video segments are collected to construct multiple sets of positive and negative samples. By comparing the positive and negative samples for model training, self-supervised extraction of multimodal video features can be achieved. Therefore, manual annotation of the training set is unnecessary, thereby reducing the model training cost.

[0275] Optionally, in the above Figure 10 Based on the corresponding embodiments, in another embodiment of the model training device 30 provided in this application,

[0276] The acquisition module 310 is specifically used to acquire video samples;

[0277] Perform optical character recognition (OCR) processing on each video frame in the video sample to obtain the text recognition result for each video frame;

[0278] If there are consecutive video frames in the video samples that belong to the same text recognition result, then the consecutive video frames are regarded as video segment samples included in a set of training samples, and the same text recognition result is regarded as text information samples included in a set of training samples.

[0279] This application provides a model training apparatus. Using this apparatus, text information samples can be identified from video samples, and corresponding video segment samples can be extracted accordingly. Therefore, compared to manually extracting training samples, this not only reduces model training costs but also improves the efficiency of sample components.

[0280] Optionally, in the above Figure 10 Based on the corresponding embodiments, in another embodiment of the model training device 30 provided in this application,

[0281] The acquisition module 310 is specifically used to acquire video samples and subtitle files corresponding to the video samples. The subtitle files include subtitle information, the start frame corresponding to the subtitle information, and the end frame corresponding to the subtitle information.

[0282] Based on the subtitle file, the subtitle information is used as a set of text information samples included in the training samples, and the continuous video frames from the start frame to the end frame are used as a set of video segment samples included in the training samples.

[0283] This application provides a model training apparatus. Using this apparatus, corresponding video segment samples can be directly extracted from video samples based on subtitle files, and subtitle information in the subtitle text can be used as corresponding text information samples. Therefore, compared to manually extracting training samples, this not only reduces model training costs but also improves the efficiency of sample components.

[0284] Optionally, in the above Figure 10 Based on the corresponding embodiments, in another embodiment of the model training device 30 provided in this application,

[0285] The acquisition module 310 is specifically used to acquire text sequence samples based on text information samples;

[0286] Based on text sequence samples, text semantic vectors are obtained through a text feature extraction model to be trained.

[0287] The acquisition module 310 is specifically used to acquire video sequence samples based on video clip samples;

[0288] Based on video sequence samples, visual semantic vectors are obtained through a visual feature extraction model to be trained.

[0289] This application provides a model training apparatus. Using this apparatus, text information samples and video clip samples need to be preprocessed before model training, which helps improve the model's learning ability and enables the model to learn more valuable features.

[0290] Optionally, in the above Figure 10 Based on the corresponding embodiments, in another embodiment of the model training device 30 provided in this application,

[0291] The acquisition module 310 is specifically used to perform word segmentation on the text information sample to obtain at least one text unit of the text information sample.

[0292] Encode each text unit in at least one text unit to obtain at least one text unit feature, wherein the text unit feature has a one-to-one correspondence with the text unit;

[0293] Generate text sequence samples based on at least one text unit feature.

[0294] In this embodiment, a model training apparatus is provided. Using this apparatus, text information samples can be segmented into words, and then a pre-trained language representation model can be used to encode the segmented text units, thereby obtaining word embedding representations (i.e., text unit features). Based on this, the neural network model can model more complex contexts and include richer semantic information in the text unit features.

[0295] Optionally, in the above Figure 10 Based on the corresponding embodiments, in another embodiment of the model training device 30 provided in this application,

[0296] The acquisition module 310 is specifically used to acquire the number of text units corresponding to the text information sample;

[0297] Based on the number of text units, a set of video frames is obtained from the video clip samples, wherein the number of video frames in the set of video frames is equal to the number of text units;

[0298] Each video frame in the video frame set is subjected to feature extraction processing to obtain at least one three-dimensional video frame feature, wherein the three-dimensional video frame feature has a one-to-one correspondence with the video frame;

[0299] For each of the at least one three-dimensional video frame features, feature transformation is performed to obtain at least one two-dimensional video frame feature, wherein the two-dimensional video frame feature and the three-dimensional video frame feature have a one-to-one correspondence.

[0300] Position encoding is performed on each of the at least one two-dimensional video frame features to obtain at least one video frame feature, wherein there is a one-to-one correspondence between the video frame features and the two-dimensional video frame features.

[0301] Generate video sequence samples based on at least one video frame feature.

[0302] In this embodiment, a model training apparatus is provided. Using the above apparatus, based on the number of text units obtained after segmenting text information samples, the same number of video frames are extracted. Thus, training objects with mapping relationships can be formed, thereby facilitating model training.

[0303] Optionally, in the above Figure 10 Based on the corresponding embodiments, in another embodiment of the model training device 30 provided in this application,

[0304] The acquisition module 310 is specifically used to perform image analysis on each video frame in the video segment sample to obtain the image evaluation value corresponding to each video frame, wherein the image evaluation value is used to evaluate the quality of the video frame;

[0305] The video frames in the video clip samples are sorted in descending order of image evaluation value, and the number of video frames equal to the number of text units is selected as the video frame set.

[0306] This application provides a model training apparatus. Using this apparatus, after analyzing the image quality of video frames, video frames with better image quality are preferentially selected for model training, thereby improving the model training effect.

[0307] Optionally, in the above Figure 10 Based on the corresponding embodiments, in another embodiment of the model training device 30 provided in this application,

[0308] The acquisition module 310 is specifically used to acquire the first feature vector based on the text sequence sample through the first temporal network included in the text feature extraction model to be trained;

[0309] Based on the first feature vector, the text semantic vector is obtained through the first multilayer perceptron network included in the text feature extraction model to be trained;

[0310] The acquisition module 310 is specifically used to acquire a second feature vector based on video sequence samples through a second temporal network included in the visual feature extraction model to be trained, wherein the second temporal network has the same network structure as the first temporal network.

[0311] Based on the second feature vector, the visual semantic vector is obtained through the second multilayer perceptron network included in the visual feature extraction model to be trained.

[0312] This application provides a model training apparatus. Using this apparatus, the concept of self-supervised learning is introduced into multimodal deep learning. By utilizing a Siamese network contrastive learning method, the dependence of model training on labeled data is eliminated, enabling the modeling of high-level semantic relationships in the temporal dimension of video frames. Based on this, a good representation of video images and text information can be learned, which is beneficial for its application in downstream tasks.

[0313] Optionally, in the above Figure 10 Based on the corresponding embodiments, in another embodiment of the model training device 30 provided in this application,

[0314] The acquisition module 310 is specifically used to acquire the first feature vector based on the text sequence sample through the first temporal network included in the text feature extraction model to be trained;

[0315] Based on the first feature vector, the text semantic vector is obtained through the first multilayer perceptron network included in the text feature extraction model to be trained;

[0316] The acquisition module 310 is specifically used to acquire a second feature vector based on video sequence samples through a second temporal network included in the visual feature extraction model to be trained, wherein the second temporal network has a different network structure from the first temporal network.

[0317] Based on the second feature vector, the visual semantic vector is obtained through the second multilayer perceptron network included in the visual feature extraction model to be trained.

[0318] This application provides a model training apparatus. Using this apparatus, the idea of ​​self-supervised learning is introduced into multimodal deep learning. By utilizing a pseudo-Siamese network contrastive learning method, not only is the model training freed from the dependence on labeled data, enabling the modeling of high-level semantic relationships in the temporal dimension of video frames, but the flexibility of the model structure is also improved compared to Siamese networks. Based on this, a good representation of video images and text information can be learned, which is beneficial for its application in downstream tasks.

[0319] Optionally, in the above Figure 10 Based on the corresponding embodiments, in another embodiment of the model training device 30 provided in this application, the T groups of training samples include a first training sample and a second training sample, wherein the first training sample includes a first text information sample and a first video segment sample with a corresponding relationship, and the second training sample includes a second text information sample and a second video segment sample with a corresponding relationship.

[0320] The acquisition module 310 is specifically used to acquire the first text semantic vector corresponding to the first text information sample through the text feature extraction model to be trained, and to acquire the second text semantic vector corresponding to the second text information sample through the text feature extraction model to be trained.

[0321] The acquisition module 310 is specifically used to acquire the first visual semantic vector corresponding to the first video segment sample through the visual feature extraction model to be trained, and to acquire the second visual semantic vector corresponding to the second video segment sample through the visual feature extraction model to be trained.

[0322] The determination module 320 is specifically used to take the first visual semantic vector and the first text semantic vector as a set of positive samples, and the second visual semantic vector and the second text semantic vector as another set of positive samples;

[0323] The first visual semantic vector and the second text semantic vector are used as one set of negative samples, and the second visual semantic vector and the first text semantic vector are used as another set of negative samples.

[0324] This application provides a model training apparatus. Using this apparatus, a feature extraction model can be trained based on composed positive and negative samples using a contrastive learning method. This not only achieves self-supervised multimodal video feature extraction but also reduces the cost of model training.

[0325] Optionally, in the above Figure 10Based on the corresponding embodiments, in another embodiment of the model training device 30 provided in this application, the model training device 30 further includes a generation module 340 and a processing module 350;

[0326] The acquisition module 310 is also used to acquire target text information and target video segments after obtaining the text feature extraction model and the visual feature extraction model until the model training conditions are met, wherein the target text information and the target video segments have a corresponding relationship.

[0327] The generation module 340 is used to generate a target text sequence based on the target text information. The target text sequence includes L text unit features, and the text unit features have a one-to-one correspondence with the text units in the target text information. L is an integer greater than or equal to 1.

[0328] The acquisition module 310 is further configured to acquire L video frames from the target text information based on the number of text units included in the target text sequence, and generate a target video sequence based on the L video frames. The target video sequence includes L video frame features, and the video frame features have a one-to-one correspondence with the video frames acquired from the target video segment.

[0329] The acquisition module 310 is also used to acquire the semantic vector of the target text based on the target text sequence through a text feature extraction model;

[0330] The acquisition module 310 is also used to acquire the target visual semantic vector based on the target video sequence through a visual feature extraction model;

[0331] The processing module 350 is used to concatenate the target text semantic vector and the target visual semantic vector to obtain a multimodal feature vector.

[0332] This application provides a model training apparatus. Using this apparatus, after the text feature extraction model and the visual feature extraction model are trained, the visual information and subtitle information of the video can be directly input into the Siamese network in the form of images and text, respectively. The resulting output represents the information in the video (i.e., a multimodal feature vector is obtained). Therefore, the generated feature extraction model can be flexibly applied to various downstream tasks or speech pre-training, saving the cost of training the feature extractor with a large amount of data. The pre-trained model can perform transfer learning on the adaptive and softmax layers using only a small dataset, further lowering the barrier to entry for using deep learning models and reducing the cost of using deep learning models for multimodal video feature extraction.

[0333] The model training apparatus in this application is described in detail below. Please refer to [link / reference]. Figure 11 , Figure 11This is a schematic diagram of another embodiment of the model training device in this application. The model training device 40 includes:

[0334] The acquisition module 410 is used to acquire the text information to be trained and the video segment to be trained, wherein the text information to be trained and the video segment to be trained have a corresponding relationship, and the video segment to be trained has a labeled classification label.

[0335] The acquisition module 410 is also used to acquire the semantic vector of the text to be trained corresponding to the text information to be trained through the text feature extraction model, wherein the text feature extraction model is obtained by performing the above-mentioned methods;

[0336] The acquisition module 410 is also used to acquire the visual semantic vector to be trained corresponding to the video segment to be trained through the visual feature extraction model, wherein the visual feature extraction model is obtained by performing the above-mentioned methods;

[0337] The processing module 420 is used to concatenate the text semantic vector to be trained and the visual semantic vector to be trained to obtain the multimodal feature vector to be trained.

[0338] The acquisition module 410 is also used to obtain the classification probability distribution based on the multimodal feature vector to be trained through the video classification model to be trained;

[0339] The training module 430 is used to update the model parameters of the video classification model to be trained based on the classification probability distribution and classification labels until the model training conditions are met, thus obtaining the video classification model.

[0340] This application provides a model training apparatus. Using this apparatus, after the text feature extraction model and the visual feature extraction model have been trained, the model can be applied to different downstream tasks as needed. For example, in video classification tasks, a high-performance video classification model (i.e., a classifier) ​​can be trained using the concatenated multimodal feature vectors and classification labels, and then applied in video classification scenarios.

[0341] This application also provides another model training device that can be deployed on a server. Figure 12This is a schematic diagram of a server structure provided in an embodiment of this application. The server 500 can vary significantly due to different configurations or performance. It may include one or more central processing units (CPUs) 522 (e.g., one or more processors) and memory 532, and one or more storage media 530 (e.g., one or more mass storage devices) for storing application programs 542 or data 544. The memory 532 and storage media 530 can be temporary or persistent storage. The program stored in the storage media 530 may include one or more modules (not shown in the diagram), each module may include a series of instruction operations on the server. Furthermore, the CPU 522 may be configured to communicate with the storage media 530 and execute the series of instruction operations in the storage media 530 on the server 500.

[0342] Server 500 may also include one or more power supplies 526, one or more wired or wireless network interfaces 550, one or more input / output interfaces 558, and / or one or more operating systems 541, such as Windows Server. TM Mac OS X TM Unix TM Linux TM FreeBSD TM etc.

[0343] The steps performed by the server in the above embodiments can be based on this Figure 12 The server structure shown.

[0344] This application also provides another model training device that can be deployed on a terminal device, such as... Figure 13 As shown, for ease of explanation, only the parts related to the embodiments of this application are shown. For specific technical details not disclosed, please refer to the method section of the embodiments of this application. In the embodiments of this application, a smartphone is used as an example for illustration:

[0345] Figure 13 This is a block diagram illustrating a portion of the structure of a smartphone related to the terminal device provided in the embodiments of this application. (Reference) Figure 13 The smartphone includes components such as a radio frequency (RF) circuit 610, a memory 620, an input unit 630, a display unit 640, a sensor 650, an audio circuit 660, a wireless fidelity (WiFi) module 670, a processor 680, and a power supply 690. Those skilled in the art will understand that... Figure 13The smartphone structure shown does not constitute a limitation on smartphones and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0346] The following is combined Figure 13 A detailed introduction to the various components of a smartphone:

[0347] RF circuit 610 can be used for receiving and transmitting signals during information transmission or calls. Specifically, it receives downlink information from the base station and processes it with processor 680; additionally, it transmits uplink data to the base station. Typically, RF circuit 610 includes, but is not limited to, an antenna, at least one amplifier, a transceiver, a coupler, a low-noise amplifier (LNA), a duplexer, etc. Furthermore, RF circuit 610 can also communicate wirelessly with networks and other devices. The aforementioned wireless communication can use any communication standard or protocol, including but not limited to Global System for Mobile Communications (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Long Term Evolution (LTE), email, Short Message Service (SMS), etc.

[0348] The memory 620 can be used to store software programs and modules. The processor 680 executes various functions and data processing of the smartphone by running the software programs and modules stored in the memory 620. The memory 620 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, applications required for at least one function (such as sound playback function, image playback function, etc.), etc.; the data storage area may store data created based on the use of the smartphone (such as audio data, phonebook, etc.). In addition, the memory 620 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device.

[0349] The input unit 630 can be used to receive input numerical or character information, and to generate key signal inputs related to user settings and function control of the smartphone. Specifically, the input unit 630 may include a touch panel 631 and other input devices 632. The touch panel 631, also known as a touch screen, can collect touch operations performed by the user on or near it (such as operations performed by the user using a finger, stylus, or any suitable object or accessory on or near the touch panel 631), and drive the corresponding connected devices according to a pre-set program. Optionally, the touch panel 631 may include two parts: a touch detection device and a touch controller. The touch detection device detects the user's touch position and the signal generated by the touch operation, and transmits the signal to the touch controller; the touch controller receives touch information from the touch detection device, converts it into touch point coordinates, and sends it to the processor 680, and can also receive and execute commands sent by the processor 680. In addition, the touch panel 631 can be implemented using various types such as resistive, capacitive, infrared, and surface acoustic wave. In addition to the touch panel 631, the input unit 630 may also include other input devices 632. Specifically, other input devices 632 may include, but are not limited to, one or more of the following: physical keyboard, function keys (such as volume control buttons, power buttons, etc.), trackball, mouse, joystick, etc.

[0350] The display unit 640 can be used to display information input by the user or information provided to the user, as well as various menus of the smartphone. The display unit 640 may include a display panel 641, optionally configured as a liquid crystal display (LCD), organic light-emitting diode (OLED), or similar form. Further, a touch panel 631 may cover the display panel 641. When the touch panel 631 detects a touch operation on or near it, it transmits the information to the processor 680 to determine the type of touch event. Subsequently, the processor 680 provides corresponding visual output on the display panel 641 based on the type of touch event. Although in Figure 13 In this embodiment, the touch panel 631 and the display panel 641 are two separate components to realize the input and output functions of the smartphone. However, in some embodiments, the touch panel 631 and the display panel 641 can be integrated to realize the input and output functions of the smartphone.

[0351] The smartphone may also include at least one sensor 650, such as a light sensor, a motion sensor, and other sensors. Specifically, the light sensor may include an ambient light sensor and a proximity sensor, wherein the ambient light sensor can adjust the brightness of the display panel 641 according to the ambient light level, and the proximity sensor can turn off the display panel 641 and / or backlight when the smartphone is moved to the ear. As a type of motion sensor, an accelerometer sensor can detect the magnitude of acceleration in various directions (generally three axes), and can detect the magnitude and direction of gravity when stationary. It can be used for applications that recognize the smartphone's posture (such as landscape / portrait switching, related games, magnetometer posture calibration), vibration recognition-related functions (such as pedometer, tapping), etc. Other sensors that may be configured in the smartphone, such as gyroscopes, barometers, hygrometers, thermometers, and infrared sensors, will not be described in detail here.

[0352] Audio circuit 660, speaker 661, and microphone 662 provide an audio interface between the user and the smartphone. Audio circuit 660 converts received audio data into electrical signals and transmits them to speaker 661, where speaker 661 converts them into sound signals for output. On the other hand, microphone 662 converts collected sound signals into electrical signals, which are received by audio circuit 660, converted into audio data, and then processed by processor 680 before being transmitted via RF circuit 610 to, for example, another smartphone, or the audio data can be output to memory 620 for further processing.

[0353] WiFi is a short-range wireless transmission technology. Smartphones, through their WiFi module 670, can help users send and receive emails, browse web pages, and access streaming media, providing wireless broadband internet access. Although Figure 13 WiFi module 670 is shown, but it is understood that it is not an essential component of a smartphone and can be omitted as needed without changing the nature of the invention.

[0354] The processor 680 is the control center of the smartphone, connecting various parts of the smartphone through various interfaces and lines. It performs various functions and processes data by running or executing software programs and / or modules stored in the memory 620, and by calling data stored in the memory 620, thereby providing overall monitoring of the smartphone. Optionally, the processor 680 may include one or more processing units; optionally, the processor 680 may integrate an application processor and a modem processor, wherein the application processor mainly handles the operating system, user interface, and applications, and the modem processor mainly handles wireless communication. It is understood that the aforementioned modem processor may also not be integrated into the processor 680.

[0355] The smartphone also includes a power supply 690 (such as a battery) that powers various components. Optionally, the power supply can be logically connected to the processor 680 through a power management system, thereby enabling functions such as charging, discharging, and power consumption management through the power management system.

[0356] Although not shown, smartphones may also include a camera, Bluetooth module, etc., which will not be described in detail here.

[0357] The steps performed by the terminal device in the above embodiments can be based on this Figure 13 The terminal device structure is shown.

[0358] This application also provides a computer-readable storage medium storing a computer program that, when run on a computer, causes the computer to perform the methods described in the foregoing embodiments.

[0359] This application also provides a computer program product including a program, which, when run on a computer, causes the computer to perform the methods described in the foregoing embodiments.

[0360] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0361] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection between apparatuses or units through some interfaces, and may be electrical, mechanical, or other forms.

[0362] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0363] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0364] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0365] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A training method for a feature extraction model, characterized in that, include: Obtain T sets of training samples, where each set of training samples includes text information samples and video segment samples with corresponding relationships, where T is an integer greater than 1; including: obtaining video samples; performing optical character recognition (OCR) processing on each video frame in the video samples to obtain the text recognition result of each video frame; if there are consecutive video frames in the video samples that belong to the same text recognition result, then the consecutive video frames are used as video segment samples included in a set of training samples, and the same text recognition result is used as text information samples included in the set of training samples; or obtaining video samples and subtitle files corresponding to the video samples, wherein the subtitle file includes subtitle information, the start frame corresponding to the subtitle information, and the end frame corresponding to the subtitle information; according to the subtitle file, the subtitle information is used as text information samples included in a set of training samples, and the consecutive video frames formed from the start frame to the end frame are used as video segment samples included in a set of training samples. For each set of training samples, the text semantic vector corresponding to the text information sample is obtained through the text feature extraction model to be trained; For each set of training samples, the visual semantic vector corresponding to the video segment sample is obtained through the visual feature extraction model to be trained; Based on the text semantic vector and visual semantic vector corresponding to each group of training samples, at least one group of positive samples and at least one group of negative samples are determined. Each group of positive samples includes the text semantic vector and visual semantic vector corresponding to the same group of training samples, and each group of negative samples includes the text semantic vector and visual semantic vector corresponding to different groups of training samples. Based on the at least one set of positive samples and the at least one set of negative samples, a loss result is calculated using a contrastive learning loss function. The loss result is then used for backpropagation to update the model parameters of the text feature extraction model to be trained and the model parameters of the visual feature extraction model until the model training conditions are met, thereby obtaining the text feature extraction model and the visual feature extraction model. The step of obtaining the text semantic vector corresponding to the text information sample through the text feature extraction model to be trained includes: Obtain a text sequence sample based on the text information sample; Based on the text sequence samples, the text semantic vector is obtained through the text feature extraction model to be trained; The visual semantic vectors corresponding to the video segment samples are obtained through a visual feature extraction model to be trained, including: Obtain the number of text units corresponding to the text information sample; Based on the number of text units, a set of video frames is obtained from the video segment sample, wherein the number of video frames included in the set of video frames is equal to the number of text units; Each video frame in the video frame set is subjected to feature extraction processing to obtain at least one three-dimensional video frame feature, wherein the three-dimensional video frame feature has a one-to-one correspondence with the video frame; Each of the at least one three-dimensional video frame features is subjected to feature transformation to obtain at least one two-dimensional video frame feature, wherein the two-dimensional video frame feature and the three-dimensional video frame feature have a one-to-one correspondence. Each of the at least one two-dimensional video frame features is subjected to position encoding processing to obtain at least one video frame feature, wherein the video frame feature and the two-dimensional video frame feature have a one-to-one correspondence. Generate video sequence samples based on the features of at least one video frame; Based on the video sequence samples, the visual semantic vector is obtained through the visual feature extraction model to be trained.

2. The training method according to claim 1, characterized in that, The step of obtaining a text sequence sample based on the text information sample includes: The text information sample is segmented into words to obtain at least one text unit of the text information sample; Encoding is performed on each of the at least one text unit to obtain at least one text unit feature, wherein the text unit feature has a one-to-one correspondence with the text unit; The text sequence sample is generated based on the features of at least one text unit.

3. The training method according to claim 1, characterized in that, The step of obtaining a set of video frames from the video segment samples based on the number of text units includes: Image analysis is performed on each video frame in the video segment sample to obtain an image evaluation value corresponding to each video frame, wherein the image evaluation value is used to evaluate the quality of the video frame; The video frames in the video segment sample are sorted in descending order of image evaluation value, and the number of video frames equal to the number of text units is selected as the video frame set.

4. The training method according to claim 1, characterized in that, The step of obtaining the text semantic vector based on the text sequence sample and through the text feature extraction model to be trained includes: Based on the text sequence sample, a first feature vector is obtained through the first temporal network included in the text feature extraction model to be trained; Based on the first feature vector, the text semantic vector is obtained through the first multilayer perceptron network included in the text feature extraction model to be trained; The step of obtaining the visual semantic vector based on the video sequence samples and through the visual feature extraction model to be trained includes: Based on the video sequence samples, a second feature vector is obtained through the second temporal network included in the visual feature extraction model to be trained, wherein the second temporal network has the same network structure as the first temporal network; Based on the second feature vector, the visual semantic vector is obtained through the second multilayer perceptron network included in the visual feature extraction model to be trained.

5. The training method according to claim 1, characterized in that, The step of obtaining the text semantic vector based on the text sequence sample and through the text feature extraction model to be trained includes: Based on the text sequence sample, a first feature vector is obtained through the first temporal network included in the text feature extraction model to be trained; Based on the first feature vector, the text semantic vector is obtained through the first multilayer perceptron network included in the text feature extraction model to be trained; The step of obtaining the visual semantic vector based on the video sequence samples and through the visual feature extraction model to be trained includes: Based on the video sequence samples, a second feature vector is obtained through the second temporal network included in the visual feature extraction model to be trained, wherein the second temporal network has a different network structure from the first temporal network; Based on the second feature vector, the visual semantic vector is obtained through the second multilayer perceptron network included in the visual feature extraction model to be trained.

6. The training method according to claim 1, characterized in that, The T training samples include a first training sample and a second training sample, wherein the first training sample includes a first text information sample and a first video clip sample with a corresponding relationship, and the second training sample includes a second text information sample and a second video clip sample with a corresponding relationship. For each set of training samples, obtaining the text semantic vector corresponding to the text information sample through the text feature extraction model to be trained includes: The first text semantic vector corresponding to the first text information sample is obtained through the text feature extraction model to be trained, and the second text semantic vector corresponding to the second text information sample is obtained through the text feature extraction model to be trained. For each set of training samples, the step of obtaining the visual semantic vector corresponding to the video segment sample through the visual feature extraction model to be trained includes: The first visual semantic vector corresponding to the first video segment sample is obtained through the visual feature extraction model to be trained, and the second visual semantic vector corresponding to the second video segment sample is obtained through the visual feature extraction model to be trained. The step of determining at least one set of positive samples and at least one set of negative samples based on the text semantic vector and visual semantic vector corresponding to each set of training samples includes: The first visual semantic vector and the first text semantic vector are used as one set of positive samples, and the second visual semantic vector and the second text semantic vector are used as another set of positive samples; The first visual semantic vector and the second text semantic vector are used as one set of negative samples, and the second visual semantic vector and the first text semantic vector are used as another set of negative samples.

7. The training method according to any one of claims 1 to 6, characterized in that, After obtaining the text feature extraction model and the visual feature extraction model by satisfying the model training conditions, the method further includes: Obtain target text information and target video clips, wherein the target text information and the target video clips have a corresponding relationship; A target text sequence is generated based on the target text information, wherein the target text sequence includes L text unit features, and the text unit features have a one-to-one correspondence with the text units in the target text information, and L is an integer greater than or equal to 1; Based on the number of text units included in the target text sequence, L video frames are obtained from the target text information, and a target video sequence is generated based on the L video frames. The target video sequence includes L video frame features, and the video frame features have a one-to-one correspondence with the video frames obtained from the target video segment. Based on the target text sequence, the target text semantic vector is obtained through a text feature extraction model; Based on the target video sequence, a target visual semantic vector is obtained through a visual feature extraction model; The target text semantic vector and the target visual semantic vector are concatenated to obtain a multimodal feature vector.

8. A method for training a model, characterized in that, include: Obtain training text information and training video segments, wherein the training text information and the training video segments have a corresponding relationship, and the training video segments have labeled classification tags; The semantic vector of the text to be trained is obtained by means of the text feature extraction model, wherein the text feature extraction model is trained by any one of the methods 1 to 7 above; The visual semantic vector corresponding to the video segment to be trained is obtained by a visual feature extraction model, wherein the visual feature extraction model is trained using any one of the methods 1 to 7 above. The text semantic vector to be trained and the visual semantic vector to be trained are concatenated to obtain the multimodal feature vector to be trained. Based on the multimodal feature vector to be trained, the classification probability distribution is obtained through the video classification model to be trained; Based on the classification probability distribution and the classification label, the model parameters of the video classification model to be trained are updated until the model training conditions are met, thus obtaining the video classification model.

9. A model training device, characterized in that, include: The acquisition module is used to acquire T sets of training samples, wherein each set of training samples includes text information samples and video clip samples with corresponding relationships, and T is an integer greater than 1; The acquisition module is further configured to acquire the text semantic vector corresponding to the text information sample through the text feature extraction model to be trained for each group of training samples; The acquisition module is further configured to acquire the visual semantic vector corresponding to the video segment sample through the visual feature extraction model to be trained for each group of training samples. The determining module is used to determine at least one set of positive samples and at least one set of negative samples based on the text semantic vector and visual semantic vector corresponding to each set of training samples, wherein each set of positive samples includes the text semantic vector and visual semantic vector corresponding to the same set of training samples, and each set of negative samples includes the text semantic vector and visual semantic vector corresponding to different sets of training samples. The training module is used to calculate the loss result using the contrastive learning loss function based on the at least one set of positive samples and the at least one set of negative samples, and to use the loss result to perform back gradient propagation to update the model parameters of the text feature extraction model to be trained and the model parameters of the visual feature extraction model until the model training conditions are met, so as to obtain the text feature extraction model and the visual feature extraction model. The acquisition module is specifically used to acquire video samples; Perform optical character recognition (OCR) processing on each video frame in the video sample to obtain the text recognition result of each video frame; If there are consecutive video frames in the video samples that belong to the same text recognition result, then the consecutive video frames are used as video segment samples included in a set of training samples, and the same text recognition result is used as text information samples included in the set of training samples. The acquisition module is specifically used to acquire video samples and subtitle files corresponding to the video samples, wherein the subtitle file includes subtitle information, a start frame corresponding to the subtitle information, and an end frame corresponding to the subtitle information; According to the subtitle file, the subtitle information is used as a set of text information samples included in a training sample, and the continuous video frames from the start frame to the end frame are used as a set of video segment samples included in a training sample. The acquisition module is specifically used to acquire a text sequence sample based on the text information sample; Based on the text sequence samples, the text semantic vector is obtained through the text feature extraction model to be trained; The acquisition module is further specifically used to acquire video sequence samples based on the video segment samples; Based on the video sequence samples, the visual semantic vector is obtained through the visual feature extraction model to be trained; The acquisition module is specifically used to acquire the number of text units corresponding to the text information sample; Based on the number of text units, a set of video frames is obtained from the video segment sample, wherein the number of video frames included in the set of video frames is equal to the number of text units; Each video frame in the video frame set is subjected to feature extraction processing to obtain at least one three-dimensional video frame feature, wherein the three-dimensional video frame feature has a one-to-one correspondence with the video frame; Each of the at least one three-dimensional video frame features is subjected to feature transformation to obtain at least one two-dimensional video frame feature, wherein the two-dimensional video frame feature and the three-dimensional video frame feature have a one-to-one correspondence. Each of the at least one two-dimensional video frame features is subjected to position encoding processing to obtain at least one video frame feature, wherein the video frame feature and the two-dimensional video frame feature have a one-to-one correspondence. The video sequence sample is generated based on the features of at least one video frame.

10. The apparatus according to claim 9, characterized in that, The acquisition module is specifically used to perform word segmentation on the text information sample to obtain at least one text unit of the text information sample; Encoding is performed on each of the at least one text unit to obtain at least one text unit feature, wherein the text unit feature has a one-to-one correspondence with the text unit; The text sequence sample is generated based on the features of at least one text unit.

11. The apparatus according to claim 9, characterized in that, The acquisition module is specifically used to perform image analysis on each video frame in the video segment sample to obtain the image evaluation value corresponding to each video frame, wherein the image evaluation value is used to evaluate the quality of the video frame; The video frames in the video segment sample are sorted in descending order of image evaluation value, and the number of video frames equal to the number of text units is selected as the video frame set.

12. The apparatus according to claim 9, characterized in that, The acquisition module is specifically used to acquire a first feature vector based on the text sequence sample through the first temporal network included in the text feature extraction model to be trained; Based on the first feature vector, the text semantic vector is obtained through the first multilayer perceptron network included in the text feature extraction model to be trained; The acquisition module is specifically used to acquire a second feature vector based on the video sequence samples through the second temporal network included in the visual feature extraction model to be trained, wherein the second temporal network has the same network structure as the first temporal network; Based on the second feature vector, the visual semantic vector is obtained through the second multilayer perceptron network included in the visual feature extraction model to be trained.

13. The apparatus according to claim 9, characterized in that, The acquisition module is specifically used to acquire a first feature vector based on the text sequence sample through the first temporal network included in the text feature extraction model to be trained; Based on the first feature vector, the text semantic vector is obtained through the first multilayer perceptron network included in the text feature extraction model to be trained; The acquisition module is specifically used to acquire a second feature vector based on the video sequence samples through the second temporal network included in the visual feature extraction model to be trained, wherein the second temporal network has a different network structure from the first temporal network; Based on the second feature vector, the visual semantic vector is obtained through the second multilayer perceptron network included in the visual feature extraction model to be trained.

14. The apparatus according to claim 9, characterized in that, The T training samples include a first training sample and a second training sample, wherein the first training sample includes a first text information sample and a first video clip sample with a corresponding relationship, and the second training sample includes a second text information sample and a second video clip sample with a corresponding relationship. The acquisition module is specifically used to acquire the first text semantic vector corresponding to the first text information sample through the text feature extraction model to be trained, and to acquire the second text semantic vector corresponding to the second text information sample through the text feature extraction model to be trained. The acquisition module is specifically used to acquire the first visual semantic vector corresponding to the first video segment sample through the visual feature extraction model to be trained, and to acquire the second visual semantic vector corresponding to the second video segment sample through the visual feature extraction model to be trained. The determining module is specifically used to take the first visual semantic vector and the first text semantic vector as a set of positive samples, and the second visual semantic vector and the second text semantic vector as another set of positive samples; The first visual semantic vector and the second text semantic vector are used as one set of negative samples, and the second visual semantic vector and the first text semantic vector are used as another set of negative samples.

15. The apparatus according to any one of claims 9-14, characterized in that, The model training device also includes a generation module and a processing module; The acquisition module is used to acquire target text information and target video segments, wherein the target text information and the target video segments have a corresponding relationship; The generation module is used to generate a target text sequence based on the target text information, wherein the target text sequence includes L text unit features, the text unit features have a one-to-one correspondence with the text units in the target text information, and L is an integer greater than or equal to 1; The acquisition module is further configured to acquire L video frames from the target text information based on the number of text units included in the target text sequence, and generate a target video sequence based on the L video frames, wherein the target video sequence includes L video frame features, and the video frame features have a one-to-one correspondence with the video frames acquired from the target video segment. The acquisition module is further configured to acquire the target text semantic vector based on the target text sequence using a text feature extraction model; The acquisition module is further configured to acquire a target visual semantic vector based on the target video sequence using a visual feature extraction model; The processing module is used to concatenate the target text semantic vector and the target visual semantic vector to obtain a multimodal feature vector.

16. A model training device, characterized in that, include: The acquisition module is used to acquire the text information to be trained and the video segment to be trained, wherein the text information to be trained and the video segment to be trained have a corresponding relationship, and the video segment to be trained has a labeled classification label; The acquisition module is further configured to acquire the semantic vector of the text to be trained corresponding to the text information to be trained through a text feature extraction model, wherein the text feature extraction model is trained using any one of the methods 1 to 7 above; The acquisition module is further configured to acquire the visual semantic vector to be trained corresponding to the video segment to be trained through a visual feature extraction model, wherein the visual feature extraction model is trained using any one of the methods 1 to 7 above. The processing module is used to concatenate the text semantic vector to be trained and the visual semantic vector to be trained to obtain the multimodal feature vector to be trained. The acquisition module is also used to obtain the classification probability distribution based on the multimodal feature vector to be trained through the video classification model to be trained; The training module is used to update the model parameters of the video classification model to be trained according to the classification probability distribution and the classification label until the model training conditions are met, thereby obtaining the video classification model.

17. A computer device, characterized in that, include: Memory, processor, and bus system; The memory is used to store programs; The processor is configured to execute a program in the memory, and the processor is configured to execute the method as described in any one of claims 1 to 7 according to instructions in the program code, or to execute the method as described in claim 8; The bus system is used to connect the memory and the processor to enable communication between the memory and the processor.

18. A computer-readable storage medium comprising instructions, when executed on a computer, causing the computer to perform the method of any one of claims 1 to 7, or to perform the method of claim 8.

19. A computer program product, comprising a computer program and instructions, characterized in that, When the computer program / instruction is executed by the processor, it implements the method as described in any one of claims 1 to 7, or performs the method as described in claim 8.

Images