Video feature extraction method, device, equipment and computer storage medium

Abstract

The present application discloses a video feature extraction method, device, equipment and computer storage medium, belonging to the field of video processing technology. In the training of the video feature extraction model, the method uses the topic tags added when the sample video is released as the real video tags, so there is no need for manual labeling, avoiding the high error rate caused by manual labeling, improving the accuracy of the trained model, and the model training efficiency is higher. In addition, the topic tags added when the sample video is released are usually closely related to the content of the video, so they can more accurately express the semantics of the video and are more in line with the actual video application scenario, so that not only can the trained model accurately express the semantic content of the video, but also can be more in line with the actual video application scenario.

Description

Technical Field

[0001] The present application relates to the field of computer technology, in particular to the field of video processing technology, and provides a video feature extraction method, device, equipment and computer storage medium. Background Art

[0002] Video is an important way to obtain information on the Internet. Taking short videos as an example, short videos are extremely important content carriers in current information flow products. Comprehensive and accurate understanding of the content of short videos is a prerequisite for realizing functions such as retrieval, clustering, recommendation or search for short videos. In addition to common basic features such as video classification and tags, video content understanding also includes converting videos into a string of digital features with semantic information. How to comprehensively and accurately characterize video content features is the most challenging problem in content understanding.

[0003] In the video feature extraction scheme in the related art, supervised learning is usually used to learn the model, and then the obtained model is used for feature extraction. However, the supervised learning scheme requires manual labeling of each sample in advance, and the number of samples is generally large. Manual labeling is not only time-consuming but also prone to errors, resulting in a low accuracy rate of the trained model. Summary of the invention

[0004] The embodiments of the present application provide a video feature extraction method, apparatus, device and computer storage medium, which trains a model by using topic tag lines added when a video is released, without the need for manual labeling, thus avoiding the high error rate caused by manual labeling and improving the accuracy of the trained model.

[0005] On the one hand, a video feature extraction method is provided, the method comprising:

[0006] For a target video to be processed, obtaining target image feature information of a plurality of video frames included in the target video, and obtaining target text information associated with the target video;

[0007] Inputting the text information and the image feature information into a trained video feature extraction model to obtain a target video feature vector of the target video;

[0008] Among them, the trained video feature extraction model is trained based on the predicted video labels obtained based on the sample video feature vectors corresponding to each sample video, and the corresponding real video labels; each sample video feature vector is obtained based on the sample text information and sample image feature information of the corresponding sample video, and the real video label of each sample video is obtained based on the topic label added when the sample video is released.

[0009] On the one hand, a video feature extraction device is provided, the device comprising:

[0010] A data preprocessing unit, configured to obtain target image feature information of a plurality of video frames included in a target video to be processed, and obtain target text information associated with the target video;

[0011] A feature extraction unit, used for inputting the target text information and the target image feature information into a trained video feature extraction model to obtain a target video feature vector of the target video;

[0012] Among them, the trained video feature extraction model is trained based on the predicted video labels obtained based on the sample video feature vectors corresponding to each sample video, and the corresponding real video labels; each sample video feature vector is obtained based on the sample text information and sample image feature information of the corresponding sample video, and the real video label of each sample video is obtained based on the topic label added when the sample video is released.

[0013] Optionally, the data preprocessing unit is specifically used to:

[0014] Extracting image features from the multiple video frames respectively to obtain target image feature information corresponding to each of the multiple video frames; and

[0015] The target text information is obtained by using one or more of the following methods:

[0016] Extracting target text information contained in the title from the title of the target video;

[0017] Using an image text recognition method, extracting target text information contained in each video frame included in the target video;

[0018] A speech-to-text recognition method is adopted to extract target text information contained in the audio data included in the target video.

[0019] Optionally, the feature extraction unit is specifically used to:

[0020] Performing vectorization processing on each character included in the target text information to obtain a character feature vector corresponding to each character;

[0021] Based on the target image feature information corresponding to each of the multiple video frames, obtaining the image feature vector corresponding to each of the multiple video frames;

[0022] Perform cascade processing on the obtained character feature vectors and image feature vectors to obtain a target feature vector sequence;

[0023] A serialized feature extraction method is adopted to extract features from the target feature vector sequence to obtain the target video feature vector.

[0024] Optionally, the feature extraction unit is specifically used to:

[0025] For each text sentence included in the target text information, perform the following operations respectively:

[0026] For a plurality of characters included in a text sentence, obtaining content feature vectors corresponding to each of the plurality of characters from a preset character encoding dictionary;

[0027] Based on the positions of the multiple characters in the text sentence, respectively, obtaining position feature vectors corresponding to the multiple characters;

[0028] Based on the data type of the text sentence, obtaining the type feature vectors corresponding to each of the multiple characters; wherein the data type includes the title text, image text and voice text;

[0029] According to the content feature vectors, the position feature vectors and the type feature vectors corresponding to the multiple characters, character feature vectors corresponding to the multiple characters are obtained.

[0030] Optionally, the feature extraction unit is specifically used to:

[0031] Cascading the character feature vectors of the multiple characters included in each text sentence in the characters to obtain a sentence feature vector of each text sentence; wherein the sentence feature vector includes a sentence start marker for indicating the starting position of the text sentence;

[0032] The sentence feature vectors of each text sentence are cascaded to obtain a sentence feature vector sequence; wherein type separation marks are added between text sentences of different text types in the sentence feature vector sequence;

[0033] The sentence feature vector sequence and each image feature vector are cascaded to obtain the target feature vector sequence.

[0034] Optionally, the feature extraction unit is specifically used to:

[0035] For the multiple video frames, perform the following operations respectively:

[0036] For a video frame, obtaining a content feature vector of the video frame according to target image feature information of the video frame;

[0037] Based on the position of the one video frame in the target video, obtaining a position feature vector of the one video frame;

[0038] Based on the data type of the video frame, obtaining a type feature vector of the video frame; wherein the data type includes a video image type;

[0039] An image feature vector of the video frame is obtained according to the content feature vector, the position feature vector and the type feature vector of the video frame.

[0040] Optionally, the real video tag of each sample video includes a topic tag and a sequence number tag, and one sequence number tag uniquely corresponds to one sample video, and the device further includes a model training unit for:

[0041] The trained video feature extraction model is obtained by training in a manner of multiple iterative training. In each iterative training, the following operations are included:

[0042] Based on the sample text information and sample image feature information corresponding to each sample video input in this round, respectively, a sample video feature vector corresponding to each sample video is obtained;

[0043] For each sample video feature vector obtained, perform the following operations respectively:

[0044] For a sample video feature vector, multi-label classification processing is performed based on the sample video feature vector to obtain a topic tag feature vector of the sample video corresponding to the sample video feature vector; wherein the topic tag feature vector includes multiple first indicator bits, one first indicator bit corresponds to one of all topic tags, and the value on each first indicator bit is used for

[0045] Characterizing whether the sample video has a corresponding topic tag;

[0046] Performing feature mapping based on the feature vector of the sample video to obtain a sequence number label feature vector of the sample video; wherein the sequence number label feature vector includes a plurality of second indicator bits, one second indicator bit corresponds to one sequence number label, and the value of each second indicator bit is used to characterize the probability that the predicted sequence number of the sample video is the corresponding sequence number label;

[0047] Based on the obtained feature vectors of each topic tag and the topic tag of the corresponding sample video, a topic classification loss value is obtained, and based on the obtained feature vectors of each sequence number tag and the sequence number tag of the corresponding sample video, a sequence number classification loss value is obtained;

[0048] Based on the topic classification loss value and the sequence number classification loss value, the model loss value of this round of iterative training is determined, and the model parameters are adjusted based on the model loss value.

[0049] Optionally, the model training unit is specifically used to:

[0050] For each obtained topic tag feature vector, perform the following operations respectively:

[0051] For a topic tag feature vector, based on the values ​​of each first indicator bit included in the topic tag feature vector, the topic tag of the corresponding sample video, and the set weight parameter and modulation parameter, a sub-loss value of the corresponding sample video is obtained; wherein the weight parameter is used to balance the influence of positive samples and negative samples on the topic classification loss value, and the modulation parameter is used to balance the influence of simple samples and difficult samples on the topic classification loss value;

[0052] Based on the sub-loss values ​​corresponding to each sample video, the topic classification loss value is obtained.

[0053] Optionally, the model training unit is specifically used to:

[0054] For each obtained serial number label feature vector, perform the following operations respectively:

[0055] For a sequence number label feature vector, obtain a first value of a target indicator bit corresponding to the sequence number label of the corresponding sample video in the sequence number label feature vector, and obtain the first value of the remaining second indicator bits in the sequence number label feature vector except the target indicator bit.

[0056] Two values;

[0057] Based on the set space margin, the first value is spatially transformed in the set space.

[0058] Change, and obtain the third value;

[0059] Based on each of the obtained second values ​​and the third value, obtaining a sub-loss value of a corresponding sample video;

[0060] Based on the sub-loss values ​​corresponding to each sample video, the serial number classification loss value is obtained.

[0061] Optionally, the device further includes an execution unit, configured to:

[0062] Obtaining target video feature vectors corresponding to each of the multiple videos to be clustered based on target image feature information and target text information corresponding to each of the multiple videos to be clustered, using the trained video feature extraction model;

[0063] Performing clustering processing based on the target video feature vectors of the multiple videos to be clustered to obtain multiple video sets, each video set containing at least two videos to be clustered whose similarity between each other is greater than a set similarity threshold;

[0064] In response to a video data request made by a target object for a first video, determining a target video set in which the first video is located from the multiple video sets;

[0065] Based on the target video set, the video data of the first video and the video data of the remaining videos in the target video set are returned to the target object.

[0066] On the one hand, a computer device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of any one of the above methods when executing the computer program.

[0067] In one aspect, a computer storage medium is provided, on which computer program instructions are stored, and when the computer program instructions are executed by a processor, the steps of any of the above methods are implemented.

[0068] In one aspect, a computer program product or a computer program is provided, the computer program product or the computer program comprising computer instructions, the computer instructions being stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device performs the steps of any of the above methods.

[0069] In the embodiment of the present application, when training the video feature extraction model, the topic tags added when the sample video is released are used as real video tags, so there is no need for manual labeling, which avoids the high error rate caused by manual labeling, improves the accuracy of the trained model, and the efficiency of model training is higher. In addition, the topic tags added when the sample video is released are usually closely related to the video content, so they can more accurately express the video semantics and are more in line with the actual video application scenario, so that not only can the trained model accurately express the semantic content of the video, but also can be more in line with the actual video application scenario. In addition, when training, the embodiment of the present application performs feature extraction based on the multimodal information of the video, that is, text information and image feature information, so that the extracted video feature vector can comprehensively express the video content and improve the accuracy of video content expression. BRIEF DESCRIPTION OF THE DRAWINGS

[0070] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the related technologies, the drawings required for use in the embodiments or the related technical descriptions are briefly introduced below. Obviously, the drawings described below are only the embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0071] Figure 1 A schematic diagram of an application scenario provided for an embodiment of the present application;

[0072] Figure 2 A flowchart of a model training process for a video feature extraction model provided in an embodiment of the present application;

[0073] Figure 3 A schematic diagram of the structure of a video feature extraction model provided in an embodiment of the present application;

[0074] Figure 4 A schematic diagram of the structure of a topic tag feature vector provided in an embodiment of the present application;

[0075] Figure 5 A schematic diagram of the structure of a sequence number tag feature vector provided in an embodiment of the present application;

[0076] Figure 6 A schematic diagram of a flow chart of a video feature extraction method provided in an embodiment of the present application;

[0077] Figure 7 A schematic diagram of a process for obtaining a video feature vector of a target video provided in an embodiment of the present application;

[0078] Figure 8 A schematic diagram of a process for obtaining a character vector provided in an embodiment of the present application;

[0079] Fig. 9 A schematic diagram of obtaining an image feature vector of a video frame provided in an embodiment of the present application;

[0080] Fig.10 A schematic diagram of an interface for displaying similar videos in aggregate provided in an embodiment of the present application;

[0081] Fig.11 A schematic diagram of the structure of a video feature extraction device provided in an embodiment of the present application;

[0082] Fig.12 A schematic diagram of the structure of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0083] In order to make the purpose, technical scheme and advantages of the present application clearer, the technical scheme in the embodiment of the present application will be clearly and completely described below in conjunction with the drawings in the embodiment of the present application. Obviously, the described embodiment is only a part of the embodiment of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application. In the absence of conflict, the embodiments in the present application and the features in the embodiments can be combined with each other arbitrarily. In addition, although the logical order is shown in the flow chart, in some cases, the steps shown or described can be performed in an order different from that here.

[0084] To facilitate understanding of the technical solutions provided in the embodiments of the present application, some key terms used in the embodiments of the present application are explained here:

[0085] Modality: In the embodiments of the present application, each form of information can be called a modality. For video, the media of video information includes audio, video (referring to video images) and text, so each media form can be called a modality of the video, namely video modality (or image modality) and text modality, etc.

[0086] Hashtag: refers to the label added by the user when the video is released, which usually indicates the content related to the video. For example, if a user uploads a video of a living room after decoration, the user may add hashtags such as "#decoration" and "#livingroomdecoration" to it.

[0087] Video feature extraction model: refers to a model that extracts features from multimodal information of a video to obtain a video feature vector. The video feature extraction model is obtained based on the machine learning (ML) process. It aims to achieve the ability to process and understand multi-source modal information through machine learning methods, such as multimodal learning between video, audio, and semantics (text). By utilizing the complementarity between multiple modalities and eliminating the redundancy between modalities, better feature representation can be learned.

[0088] The embodiments of the present application relate to artificial intelligence and machine learning technologies, and are mainly designed based on machine learning in artificial intelligence.

[0089] Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that machines have the functions of perception, reasoning and decision-making.

[0090] Artificial intelligence technology is a comprehensive discipline that covers a wide range of fields, including both hardware-level and software-level technologies. Basic artificial intelligence technologies generally include sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operating / interactive systems, mechatronics and other technologies. Artificial intelligence software technologies mainly include computer vision technology, speech processing technology, natural language processing technology, as well as machine learning / deep learning, autonomous driving, smart transportation and other major directions.

[0091] Computer vision technology (CV) is a science that studies how to make machines "see". To put it more specifically, it refers to the use of cameras and computers to replace human eyes to identify, track and measure targets, and further perform image processing to make computer processing into images that are more suitable for human eye observation or transmission to instruments for detection. As a scientific discipline, computer vision studies related theories and technologies, and attempts to establish an artificial intelligence system that can obtain information from images or multi-dimensional data. Computer vision technology usually includes image processing, image recognition, image semantic understanding, image retrieval, optical character recognition (OCR), video processing, video semantic understanding, video content / behavior recognition, three-dimensional object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous positioning and mapping, autonomous driving, smart transportation and other technologies, as well as common biometric recognition technologies such as face recognition and fingerprint recognition.

[0092] The key technologies of speech technology include automatic speech recognition technology, speech synthesis technology and voiceprint recognition technology. Enabling computers to listen, see, speak and feel is the development direction of human-computer interaction in the future, among which speech has become one of the most promising human-computer interaction methods in the future.

[0093] Natural language processing (NLP) is an important direction in the fields of computer science and artificial intelligence. It studies various theories and methods that can achieve effective communication between people and computers using natural language. Natural language processing is a science that integrates linguistics, computer science, and mathematics. Therefore, research in this field will involve natural language, that is, the language people use in daily life, so it is closely related to the study of linguistics. Natural language processing technology usually includes text processing, semantic understanding, machine translation, robot question answering, knowledge graph and other technologies.

[0094] Machine learning is a multi-disciplinary subject that involves probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory and other disciplines. It specializes in studying how computers simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications are spread across all areas of artificial intelligence. Machine learning and deep learning usually include artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and self-learning.

[0095] Machine learning is the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications are spread across all areas of artificial intelligence. Machine learning and deep learning usually include artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning and other technologies. Artificial neural networks (ANN) abstract the human brain neural network from the perspective of information processing, establish a simple model, and form different networks according to different connection methods. A neural network is a computational model composed of a large number of nodes (or neurons) connected to each other. Each node represents a specific output function, called an activation function. The connection between each two nodes represents a weighted value for the signal passing through the connection, called a weight, which is equivalent to the memory of the artificial neural network. The output of the network varies depending on the network connection method, weight value and activation function. The network itself is usually an approximation of a certain algorithm or function in nature, or it may be an expression of a logical strategy.

[0096] With the research and advancement of artificial intelligence technology, artificial intelligence technology has been studied and applied in many fields, such as common smart homes, smart wearable devices, virtual assistants, smart speakers, smart marketing, unmanned driving, automatic driving, drones, robots, smart medical care, smart customer service, Internet of Vehicles, automatic driving, smart transportation, etc. I believe that with the development of technology, artificial intelligence technology will be applied in more fields and play an increasingly important role.

[0097] The solution provided in the embodiment of the present application involves artificial intelligence machine learning, speech technology, computer vision technology and other technologies. The embodiment of the present application uses speech technology and computer vision technology to extract multimodal information from the video, and uses the feature extraction model learned by machine learning technology to perform feature extraction, thereby extracting a video feature vector that can fully express the video content.

[0098] Specifically, in the embodiment of the present application, when extracting features from a video, an artificial neural network model based on deep learning is used. The video feature extraction in the embodiment of the present application can be divided into two parts, including a training part and an application part. Among them, the training part involves the technical field of machine learning. In the training part, the artificial neural network model (i.e., the video feature extraction model mentioned later) is trained by the machine learning technology, so that the artificial neural network model is trained based on the multimodal information of each video given in the embodiment of the present application and the video label, and the model parameters are continuously adjusted by the optimization algorithm until the model converges; the application part is used to extract features from the video using the feature extraction part in the artificial neural network model trained in the training part, and the obtained video feature vector can be used for downstream feature applications, such as video retrieval, clustering, recommendation or search applications. In addition, it should be noted that the artificial neural network model in the embodiment of the present application can be either online training or offline training, which is not specifically limited here. In this article, offline training is used as an example for illustration.

[0099] The following is a brief introduction to the design concept of the embodiments of the present application.

[0100] At present, videos are widely used, and comprehensive and accurate understanding of the content of videos is very important for subsequent video-related applications.

[0101] The video feature extraction solutions in related technologies still have the following problems:

[0102] (1) When using supervised learning to learn models, a large number of video labels need to be annotated in advance, which consumes a lot of manpower and material resources, is time-consuming and costly. At the same time, the traditional convolutional neural network (CNN) model training requires large amounts of computation and is difficult to support rapid training of tens of millions of videos and billions of frames.

[0103] (2) When learning a model based on user behavior, a large amount of user behavior data is required. However, video user behavior has a very obvious head effect, resulting in a large number of videos having no user behavior data, and thus it still cannot be applied to all videos.

[0104] (3) Pure self-supervised learning methods generally only transform images or videos multiple times to construct similar samples. They do not adjust the similarity granularity of features according to the application scenario. They only learn in a single dimension and lack information learning in other dimensions.

[0105] Therefore, the current video feature extraction scheme is not perfect.

[0106] In view of this, an embodiment of the present application provides a method based on video feature extraction. In this method, when training a video feature extraction model, the topic tags added when the sample video is released are used as real video tags, so there is no need for manual labeling, avoiding the high error rate caused by manual labeling, improving the accuracy of the trained model, and reducing the labor cost of model training and the large amount of time consumed by manual labeling, thereby making the model training more efficient.

[0107] Moreover, the topic tags added when the sample video is released are usually closely related to the content of the video, and can therefore more accurately express the semantics of the video and are more in line with the actual video application scenarios. This not only enables the trained model to accurately express the semantic content of the video, but also to be more in line with the actual video application scenarios.

[0108] In addition, during training, the embodiments of the present application perform feature extraction based on the multimodal information of the video, namely, text information and image feature information, so that the extracted video feature vector can fully express the video content and improve the accuracy of video content expression.

[0109] In addition, considering the large amount of computation required for training traditional CNN models, the embodiment of the present application adopts a method based on a visual-linguistic model to fuse the multimodal information of the video, so that the model can be efficiently trained and iterated. At the same time, the idea of ​​the face recognition algorithm ArcFace (Additive AngularMargin Loss for Deep Face Recognition) is introduced during the training process, and the video feature space is constrained based on the loss function (additive angular margin loss) used for face recognition, so that the same class is compressed into a more compact space, which is denser, so that the features learned by the network have more obvious angular distribution characteristics.

[0110] After introducing the design ideas of the embodiments of the present application, the following briefly introduces the application scenarios to which the technical solutions of the embodiments of the present application can be applied. It should be noted that the application scenarios introduced below are only used to illustrate the embodiments of the present application and are not limited. In the specific implementation process, the technical solutions provided by the embodiments of the present application can be flexibly applied according to actual needs.

[0111] The solution provided in the embodiment of the present application can be applied to most video application scenarios, especially to short video application scenarios, such as clustering, recommendation or search of similar short videos. Figure 1 As shown, it is a schematic diagram of an application scenario provided by an embodiment of the present application. In this scenario, a terminal device 101 and a server 102 may be included.

[0112] The terminal device 101 may be, for example, a mobile phone, a tablet computer (PAD), a laptop computer, a desktop computer, a smart TV, and a smart wearable device. The terminal device 101 may be installed with an application that can perform video services, such as a browser or a video client, and the user may perform corresponding video services on the application. The application involved in the embodiment of the present application may be a software client, or a client such as a web page or a small program, and the server is a background server corresponding to the software or web page, small program, etc., without limiting the specific type of the client.

[0113] The server 102 may be a background server corresponding to the client installed on the terminal device 101, for example, an independent physical server, or a server cluster or distributed system composed of multiple physical servers. It may also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, i.e., content delivery networks (CDNs), as well as big data and artificial intelligence platforms, but is not limited thereto.

[0114] The server 102 may include one or more processors 1021, a memory 1022, and an I / O interface 1023 for interacting with a terminal. In addition, the server 102 may also be configured with a database 1024, which may be used to store data information of each modality of each video and trained model parameters. The memory 1022 of the server 102 may also store program instructions of the video feature extraction method provided in the embodiment of the present application, which, when executed by the processor 1021, can be used to implement the steps of the video feature extraction method provided in the embodiment of the present application to obtain the video feature vector of the video.

[0115] In actual application, subsequent video service applications may be performed based on the video feature vector obtained by the server 102 .

[0116] In a possible implementation, the terminal device 101 may initiate a video search to the server 102 , and the server 102 may match the video feature vectors of each video based on the search information input by the user, and then select one or more videos with a higher matching degree and return them to the terminal device 101 .

[0117] In another possible implementation, the server 102 may perform clustering based on the video feature vectors of each video to group similar videos together. In this way, when the terminal device 101 initiates a video playback request, the server 102 may obtain a second video similar to the first video being played by the user based on the clustering results and recommend it to the user.

[0118] For example, in a short video application, videos with the same highlights can be aggregated together and recommended to users, making it convenient for users to browse a large number of similar videos; or, when users browse short videos, based on the clustering results of short videos, corresponding similar short videos can be obtained for the videos that the user has recently browsed, and then the user's subsequent short video playback sequence can be arranged based on these short videos. For example, some similar short videos can be arranged, and some videos with less similarity can be interspersed in the sequence to increase the diversity of video information browsed by users.

[0119] Of course, in addition to the application of the video feature vector involved above, the extracted video feature vector can also be applied to other video-related scenarios, such as video recommendation or video retrieval, etc., and the embodiments of the present application do not limit this.

[0120] The terminal device 101 and the server 102 may be directly or indirectly connected to each other through one or more networks 103. The network 103 may be a wired network or a wireless network, for example, a mobile cellular network or a wireless fidelity (Wireless-Fidelity, WIFI) network, or other possible networks, which are not limited in the embodiment of the present invention.

[0121] It should be noted that in the embodiment of the present application, the number of terminal devices 101 can be one or more, and similarly, the number of servers 102 can be one or more, that is, there is no limitation on the number of terminal devices 101 or servers 102.

[0122] In a possible application scenario, the video-related data (such as video text information, image feature information, and corresponding video feature vectors, etc.) and model parameters involved in the embodiments of the present application can be stored using cloud storage technology. Cloud storage is a new concept that extends and develops from the concept of cloud computing. A distributed cloud storage system refers to a storage system that uses cluster applications, grid technology, and distributed storage file systems to bring together a large number of different types of storage devices (or storage nodes) in the network through application software or application interfaces to work together and provide external data storage and business access functions.

[0123] In a possible application scenario, in order to reduce communication latency, servers 102 can be deployed in various regions, or different servers 102 can be used to serve the regions corresponding to each terminal device 101 for load balancing. Multiple servers 102 can share data through blockchain, and multiple servers 102 are equivalent to a data sharing system composed of multiple servers 102. For example, terminal device 101 is located at location a and communicates with server 102, and terminal device 101 is located at location b and communicates with other servers 102.

[0124] Each server 102 in the data sharing system has a node identifier corresponding to the server 102. Each server 102 in the data sharing system can store the node identifiers of other servers 102 in the data sharing system, so that the generated blocks can be broadcast to other servers 102 in the data sharing system according to the node identifiers of other servers 102. Each server 102 can maintain a node identifier list as shown in the following table, and store the server 102 name and the node identifier in the node identifier list. The node identifier can be an Internet Protocol (IP) address for interconnection between networks and any other information that can be used to identify the node.

[0125] Of course, the method provided in the embodiment of the present application is not limited to Figure 1 The application scenarios shown can also be used in other possible application scenarios, and the embodiments of the present application are not limited thereto. Figure 1 The functions that can be implemented by each device in the application scenario shown will be described in the subsequent method embodiments, and will not be described in detail here.

[0126] The method flow provided in each embodiment of the present application can be Figure 1 The process may be executed by the server 102 or the terminal device 101, or may be executed by the server 102 and the terminal device 101 together. Here, the process is mainly introduced by taking the server 102 as an example.

[0127] Since the video feature extraction process is based on a trained video feature extraction model, before introducing the process of video feature extraction, the training process of the video feature extraction model is introduced first. Figure 2 , which is a flow chart of the model training process of the video feature extraction model provided in an embodiment of the present application.

[0128] Step 201: Obtain a training sample set including multiple sample videos.

[0129] In the embodiment of the present application, each sample video is a training sample, and each training sample includes the following data:

[0130] (1) Sample text information associated with the sample video.

[0131] The sample text information may include any possible text information associated with the sample video.

[0132] Specifically, the sample text information associated with the sample video may be obtained by using one or a combination of the following methods:

[0133] 1) Extracting relevant text information from the description text of the sample video. For example, the sample text information contained in the title and the video introduction can be extracted from the title of the sample video.

[0134] 2) Extracting associated text information from each video frame included in the sample video.

[0135] Specifically, an image text recognition method can be used to extract sample text information contained in each video frame included in the sample video. For example, an OCR method can be used to extract lyrics, subtitles, and product text in the video frames. Product text refers to the text on the product presented in the video screen (such as product name, manufacturer, etc.).

[0136] 3) Extracting the associated text information from the audio data included in the sample video.

[0137] Specifically, a speech-to-text recognition method can be used to extract sample text information contained in the audio data included in the sample video. For example, the audio text can include text converted from speech in the sample video (such as character dialogues in a TV series, lyrics in a MV, etc.). For the video speech text, the automatic speech recognition (Automatic Speech Recognition, ASR) technology can be used to extract the text content.

[0138] (2) Sample image feature information of sample video

[0139] The sample image feature information is obtained by extracting image features from the video frames included in the sample video. The video frames may include all or part of the video frames of the sample video. When part of the video frames of the sample video are included, part of the video frames may be extracted from the sample video. These video frames may be, for example, key frames of the sample video, or may be selected after performing image recognition on each video frame using a certain image recognition method, and then a plurality of video frames that best reflect the video content are selected. For example, in practical applications, FFMPEG (a multimedia processing tool) may be used to extract video frames of a specific length (e.g., 30 frames) from the sample video.

[0140] Specifically, a trained image feature extraction model may be used to extract image features from multiple video frames respectively, thereby obtaining target image feature information corresponding to each of the multiple video frames. For example, the image feature extraction model may use a CNN model, and use the CNN model to extract features from each video frame, and convert each video frame into an n-dimensional embedding.

[0141] In an embodiment of the present application, in addition to text information and image feature information, feature extraction can also be performed based on other possible information. For example, feature extraction can also be performed based on audio information, optical flow information, author name, and video face information, etc. The embodiment of the present application does not limit this.

[0142] (3) Real video labels of sample videos

[0143] In the embodiment of the present application, in order to reduce the large amount of labor cost and time cost consumed by manual labeling, the real video tags used in training are obtained based on the topic tags added when the sample video is released.

[0144] Specifically, for a sample video, after obtaining the original sample video, the hashtag contained in the video title can be extracted by regular expression matching. For example, when a user posts a short video, the title is "This is the most stylish new Chinese antique shelf I have ever seen. It is full of charm and beautiful. #furniture#antique shelf#shelf", then after regular expression matching, three hashtags can be obtained, namely "#furniture", "#antique shelf", and "#shelf", that is, these three hashtags are used as the real video tags of the short video.

[0145] In the embodiment of the present application, considering that only using hashtag classification training lacks discriminative constraints on the features of the model output, it is necessary to increase the loss of self-supervision. The general self-supervised learning method constructs similar samples and non-similar samples through rotation, cropping, local sampling or time series sampling, and then uses triplet loss for training. However, this method requires mining a large number of samples during training, and then selecting difficult samples, and the training efficiency is low. Therefore, based on the idea of ​​face recognition ArcFace, the embodiment of the present application regards each sample video in the training set as a separate video semantics, and gives each sample video a separate serial number (identity, id) as a label. That is to say, in addition to the topic label, the real video label can also include a serial number label. The serial number label is to number all sample videos in order after obtaining all sample videos. Each sample video has a global number, and the number uniquely corresponds to a sample video.

[0146] For example, if the number of sample videos obtained is M, then the video serial numbers of the sample videos are 1, 2, 3, ..., M. During the training process, as the model converges, the distance between similar video features will become closer and closer, and the distance between dissimilar videos will become farther and farther, and the same class can be made more compact, compressing the same class into a tighter and denser space, so that the features learned by the network have more obvious angular distribution characteristics.

[0147] Step 202: Based on the sample text information and sample image feature information of each sample video input in this round, a sample video feature vector corresponding to each sample video is obtained respectively.

[0148] In the embodiment of the present application, the training of the video feature extraction model can be carried out by adopting a method of multiple iterative training. Since each iterative training process is similar, one iterative training process is mainly used as an example for introduction here.

[0149] In each round of iterative training, some sample videos can be extracted from the training sample set as the input of this round. The specific extraction probability can be set according to actual needs. Then, the video feature extraction model of this round, that is, the video feature extraction model adjusted in the previous round of training, can be used to obtain the sample video feature vectors corresponding to each sample video based on the sample text information and sample image feature information of each sample video input in this round.

[0150] See also Figure 3 , which is a structural diagram of a video feature extraction model. The video feature extraction model includes an input layer, a vectorization layer, a feature extraction layer and a loss layer.

[0151] Specifically, after the sample videos required for this round of training are selected, the sample text information and sample image feature information of each sample video are input through the input layer, and then the vectorization layer and the feature extraction layer obtain the sample video feature vectors corresponding to each sample video based on the input sample text information and sample image feature information of each sample video. The data processing process of the vectorization layer and the feature extraction layer will be specifically introduced in the subsequent process, so it will not be described in detail here.

[0152] In an embodiment of the present application, the loss layer is used to calculate the model loss value of this round of iterative training based on the sample video feature vector output by the feature extraction layer.

[0153] Specifically, the loss layer may include the following two subtasks, which calculate the loss values ​​from different angles and methods to improve the accuracy of model training.

[0154] (1) Topic classification

[0155] Topic classification is a multi-label classification task, which is used to predict the topic label to which the current sample video may belong. The various types of label classification (topic labels) are not mutually exclusive, that is, a sample video may correspond to multiple topic labels.

[0156] (2) Serial number classification

[0157] Serial number classification refers to predicting the serial number label corresponding to the current sample video. The various types of serial number classification (serial number labels) are mutually exclusive, that is, a sample video can only correspond to a unique serial number label.

[0158] The two subtasks mentioned above are introduced below.

[0159] Step 203: Perform multi-label classification processing on each sample video feature vector to obtain a topic label feature vector of the sample video corresponding to the sample video feature vector.

[0160] In an embodiment of the present application, the topic tag feature vector of each sample video can be predicted based on the sample video feature vector of the sample video, that is, the predicted video tag of the sample video can be obtained, which is specifically the predicted topic tag here, so that the predicted topic tag can be compared with the actual topic tag, and the topic classification loss value for topic classification can be obtained based on the difference between them.

[0161] The topic tag feature vector includes multiple first indicator bits, one first indicator bit corresponds to one of all topic tags corresponding to multiple sample videos, and the value of each first indicator bit is used to indicate whether the sample video has the corresponding topic tag.

[0162] See also Figure 4 As shown in Figure 4 , it is a structural schematic diagram of a topic tag feature vector. Among them, the topic tag feature vector contains n first indication bits, and the value of n is determined by the total number of topic tags owned by all sample videos in the training sample set. For example, if the total number of topic tags in the training sample set is 10,000, then n is 10,000, that is, the topic tag feature vector of each sample video is a 10,000-dimensional vector. Each first indication bit corresponds to a type of topic tag. For example, Figure 4 in Figure 4 , the 1st bit corresponds to the topic tag "curio cabinet". When the value of the 1st bit is the first value, it can indicate that the sample video has the topic tag "curio cabinet". On the contrary, when the value of the 1st bit is the second value, it can indicate that the sample video does not have the topic tag "curio cabinet". Among them, the first value and the second value can be set by oneself. For example, the first value can be 1 and the second value can be 0, or the first value can be 0 and the second value can be 1.

[0163] Step 204: Based on the obtained topic tag feature vectors of each sample video and the topic tags of the corresponding sample videos, obtain the topic classification loss value.

[0164] In the embodiment of the present application, based on the topic tags of the sample video, a topic tag feature vector with the same Figure 4 structure can also be determined. Taking the above "This is the most stylish new Chinese curio cabinet I've ever seen. The square and round shapes are full of charm. It's so beautiful. #Furniture#Curio cabinet#Storage rack" as an example, based on the topic tags #Furniture#Curio cabinet#Storage rack, a topic tag feature vector with the first 3 bits being 1 and the rest being 0 can be obtained. When the topic tag feature vector of the sample video is predicted, the two can be compared to determine the topic classification loss value.

[0165] Specifically, for topic classification, which belongs to multi-label classification, a conventional binary cross entropy loss (BCE loss) function can be used for calculation.

[0166] However, considering that in the embodiment of the present application, the hashtag of each sample video is uploaded by the user himself, the number of hashtags is extremely rich, but the number of sample videos corresponding to each hashtag is extremely limited. For example, the training sample set includes one million sample videos, but the number of sample videos with the topic tag "curio cabinet" may be only a limited number of dozens or even several. Therefore, there are problems of a large disparity in the positive and negative sample ratios and the coexistence of difficult and easy hashtags in the topic prediction subtask. Based on this, the embodiment of the present application uses an improved Focal loss function as the loss function for topic classification.

[0167] Specifically, for each topic tag feature vector, based on the values ​​of each first indicator bit in the topic tag feature vector, the topic tag of the corresponding sample video, and the set weight parameters and modulation parameters, the sub-loss value of the corresponding sample video is obtained, and then, based on the sub-loss values ​​of each sample video, the topic classification loss value is obtained.

[0168] Among them, the functional expression of the Focal loss loss function is:

[0169]

[0170] Among them, L focal represents the topic classification loss value, p ti It represents the probability that the predicted value on the t-th first indicator bit of sample video i is the same as the true value. α is the weight parameter used to balance the influence of positive samples and negative samples on the topic classification loss value. γ is the modulation parameter used to balance the influence of simple samples and difficult samples on the topic classification loss value.

[0171] In the embodiment of the present application, in order to enhance the loss of misclassification of positive samples, a larger α parameter may be used, for example, the α parameter in the Focal loss function may be increased to 0.7. At the same time, in order to reduce the loss of simple samples, a larger γ parameter may be used, for example, the γ parameter in the Focal loss function may be increased to 4.

[0172] Step 205: Perform feature mapping based on the feature vectors of each sample video to obtain the sequence number label feature vector of each sample video.

[0173] In an embodiment of the present application, the serial label feature vector of each sample video can be predicted based on the sample video feature vector of the sample video, that is, the predicted video label of the sample video can be obtained, which is specifically the predicted serial label here, so that the predicted serial label can be compared with the actual serial label, and the serial classification loss value for serial classification can be obtained based on the difference between them.

[0174] The serial number label feature vector includes multiple second indicator bits, one second indicator bit corresponds to a unique serial number label, and the value of each second indicator bit is used to represent the probability that the predicted serial number of the sample video is the corresponding serial number label.

[0175] See also Figure 5As shown in FIG. 1 , it is a schematic diagram of the structure of the sequence number label feature vector, where the sequence number label feature vector contains m first indicator bits, and the value of m is determined by the number of all sample videos in the training sample set. For example, if the number of sample videos in the training sample set is one million, then m is one million, that is, the sequence number label feature vector of each sample video is a one-million-dimensional vector. Each first indicator bit corresponds to a sequence number label, such as Figure 5 In , the first bit corresponds to the serial number label Bogujia. When the value of the first bit is 81%, it can be said that the probability that the serial number of the sample video is 1 is 81%.

[0176] Step 206: Obtain a sequence classification loss value based on the sequence label feature vector and the sequence label of the corresponding sample video.

[0177] In the embodiment of the present application, a sequence number tag can also be determined based on the sample video. Figure 5 The serial number label feature vectors with the same structure, for example, when the serial number of a sample video is 10, a serial number label feature vector with the 10th bit being 100% and the remaining bits being 0 can be obtained. When the serial number label feature vector of the sample video is predicted, the two can be compared to determine the serial number classification loss value.

[0178] Specifically, for serial number classification, it belongs to self-supervised learning classification. Generally speaking, the method of self-supervised learning can be trained using a triplet loss function, or an Arcface loss function. Relative to other loss functions, in the model training process based on the Arcface loss function, as the model converges, the distance between similar video features will become closer and closer, and the distance between dissimilar videos will become farther and farther, which can make the same class more "compact". Compared with other loss functions, it realizes compressing the same class into a tighter space, which is denser, so that the features learned by the network have more obvious angular distribution characteristics. Of course, in addition to the Arcface loss function, other similar loss functions can also be used, such as the CosinFace loss function or the SphereFace loss function, etc., and the embodiments of the present application do not limit this.

[0179] Taking the Arcface loss function as an example, for a sample video, the first value of the target indicator bit corresponding to the serial label of the sample video in the predicted serial label feature vector is obtained, and based on the set spatial margin, the first value is spatially transformed in the set space to obtain the third value, and the second values ​​of the remaining second indicator bits except the target indicator bit in the serial label feature vector are obtained, so as to obtain the sub-loss value of the sample video based on the obtained second values ​​and third values, and then obtain the topic classification loss value based on the sub-loss values ​​of each sample video.

[0180] For example, when the total number of sample videos is one million, the serial number label of sample video A is 100, then the first value refers to the value of the 100th bit in the serial number label feature vector.

[0181] Among them, the function expression of Arcface loss function is:

[0182]

[0183]

[0184]

[0185]

[0186] Among them, L arcface represents the topic classification loss value, M represents the number of sample videos, is the space margin between labels with different serial numbers, s is the radius of the space, x i represents the sample video feature vector of the i-th sample video, y i Indicates the serial number label of the i-th sample video, j≠y i Represents the remaining sequence labels except the corresponding sequence label for the i-th sample video.

[0187] Step 207: Based on the topic classification loss value and the sequence number classification loss value, determine the model loss value of this round of iterative training.

[0188] In an embodiment of the present application, after obtaining the topic classification loss value and the sequence number classification loss value, the model loss value of this round of iterative training can be determined based on the topic classification loss value and the sequence number classification loss value.

[0189] Specifically, the topic classification loss value and the sequence classification loss value can be superimposed to obtain the model loss value; or, weight values ​​can be set for the two subtasks of topic classification and sequence classification respectively, and then weighted summation is performed based on the topic classification loss value and the sequence classification loss value to obtain the model loss value. The weight value can be pre-set or trained as a model parameter.

[0190] It should be noted that, in actual applications, the specific method used for loss calculation in the loss layer can be set according to actual needs. For example, only multi-label classification can be used to calculate the loss value, then the model loss value is the topic classification loss value, or only serial number classification can be used to calculate the loss value, then the model loss value is the serial number classification loss value, or a combination of the two methods can be used to calculate the loss value, and the embodiments of the present application do not limit this.

[0191] Step 208: Determine whether the video feature extraction model meets the convergence condition.

[0192] In the embodiment of the present application, the convergence condition may include one or more of the following conditions:

[0193] (1) The number of iterations reaches the set threshold.

[0194] (2) The model loss value is less than the set loss threshold.

[0195] Step 209: If the determination process of step 208 is no, the model parameters are adjusted based on the model loss value.

[0196] In the embodiment of the present application, when the number of iterations does not exceed the preset number threshold, and the model loss value is not less than the set loss threshold, the determination process of step 208 is no, that is, it is considered that the current video feature extraction model does not meet the convergence conditions, and then the model parameters need to be adjusted to continue training. The parameters that need to be adjusted may include the parameters of the vectorization layer and the feature extraction layer, and may also include other parameters of the feature extraction model. After the parameters are adjusted, the next iterative training process is entered, that is, jumping to step 202.

[0197] When the number of iterations has exceeded the preset threshold, or the model loss value is less than the set loss threshold, the determination process of step 208 is yes, that is, it is considered that the current video feature extraction model has met the convergence conditions, the model training is completed, and the trained video feature extraction model is obtained, and the video feature extraction model can be used for subsequent feature extraction processes.

[0198] See also Figure 6 , is a flow chart of the video feature extraction method provided in an embodiment of the present application, and the flow of the method is introduced as follows.

[0199] Step 601: for a target video to be processed, target image feature information of a plurality of video frames included in the target video is obtained, and target text information associated with the target video is obtained.

[0200] In the embodiment of the present application, the target image feature information is obtained by extracting image features from the video frames included in the target video, and the target text information may include any text content associated with the target video. Since the process of extracting the target image feature information and the target text information is the same as the method of extracting the sample image feature information and the sample text information during training, the extraction of the target image feature information and the target text information can be referred to. Figure 2 The description of the embodiment shown will not be repeated here.

[0201] Step 602: Input the target text information and the target image feature information into the trained video feature extraction model to obtain the target video feature vector of the target video; wherein the trained video feature extraction model is trained based on the predicted video labels obtained based on the sample video feature vectors corresponding to each sample video, and the corresponding real video labels; each sample video feature vector is obtained based on the sample text information and sample image feature information of the corresponding sample video, and the real video label of each sample video is obtained based on the topic label added when the sample video is released.

[0202] In an embodiment of the present application, the multimodal data information of the obtained target video is input into a video feature extraction model, that is, the above-mentioned target text information and target image feature information are input into a trained video feature extraction model. After being processed by the video feature extraction model, the video feature vector of the target video output by the video feature extraction model can be obtained.

[0203] Among them, the video feature extraction model of the embodiment of the present application uses the topic tags added when the sample video is released as the real video tags, so no additional manual labeling is required, thereby reducing the labor cost of model training and the large amount of time consumed by manual labeling, and thus the model training efficiency is higher. In addition, the topic tags added when the sample video is released are usually closely related to the content of the video, so they can more accurately express the semantics of the video and are more in line with the actual video application scenario, so that not only can the trained model accurately express the semantic content of the video, but also can be more in line with the actual video application scenario.

[0204] Combine the following Figure 3 The structure of the video feature extraction model shown in FIG. 1 introduces the process of obtaining the video feature vector of the target video. Figure 7 As shown in FIG. 1 , it is a schematic diagram of the process of obtaining the video feature vector of the target video, wherein: Figure 7The video feature vector acquisition process shown is applicable to the training stage and the actual application stage. When in the model training stage, the following text information and image feature information are specifically the sample text information and sample image feature information of each sample video. When in the actual application stage, the following text information and image feature information are specifically the target text information and target image feature information of the target video.

[0205] In an embodiment of the present application, the video feature extraction model may adopt any possible model structure, such as a bidirectional encoder representation from transformer (BERT) structure. Of course, a derivative model of BERT may also be adopted.

[0206] Step 701: Perform vectorization processing on each character included in the text information to obtain a character feature vector corresponding to each character.

[0207] Regarding text information, the above-mentioned three types of text information are used as examples for explanation. When more text information is included, the method is also applicable.

[0208] See also Figure 3 As shown, each character included in the text information is processed through the vectorization layer to obtain the character feature vector corresponding to each character.

[0209] Specifically, text information generally consists of multiple text sentences, and a text sentence consists of multiple characters. The processing of each character in each text sentence is basically the same. Here, a text sentence is used as an example for introduction. Figure 8 The figure shows the process flow of obtaining a character vector.

[0210] For each character included in a text sentence, the content feature vector corresponding to each character can be obtained from the preset character encoding dictionary, such as Figure 8 As shown, a text sentence includes characters 1 to n, and the corresponding content feature vectors Ea1 to Ean are obtained respectively. In the character encoding dictionary, each character uniquely corresponds to a content feature vector. For example, a character encoding dictionary based on one-hot encoding can be used. Of course, any other character encoding dictionary can also be used.

[0211] Specifically, the token corresponding to each character can be obtained from a preset character encoding dictionary in advance, and then feature extraction is performed on the token to obtain the embedding of each character, that is, the content feature vector.

[0212] In addition, the position feature vector corresponding to each character can be obtained based on the position of each character in the text sentence. For example, if character 2 is located at the second character position in the sentence, the corresponding position feature vector can indicate the second character position, such as Figure 8 As shown, a text sentence includes characters 1 to n, and corresponding position feature vectors Eb1 to Ebn are obtained respectively.

[0213] And, based on the data type of the text sentence where each character is located, the type feature vector corresponding to each character is obtained, wherein the data type includes title text, image text and voice text. For example, when the text sentence where character 1 is located is extracted from the title, the type feature vector corresponding to character 1 represents the title text type. Figure 8 As shown, a text sentence includes characters 1 to n, and the corresponding type feature vectors Ec1 to Ecn are obtained respectively, which all indicate the data type of the text sentence. In the embodiment of the present application, the text generally comes from different modes such as titles, images and voices, so a data type can actually be understood as a mode, so that subsequent feature extraction is performed based on multimodal information.

[0214] Furthermore, see Figure 8 As shown, the character feature vector corresponding to each character can be obtained according to the content feature vector, position feature vector and type feature vector corresponding to each character.

[0215] Specifically, the content feature vector, position feature vector and type feature vector may be superimposed to obtain a character feature vector corresponding to the corresponding character; or, the content feature vector, position feature vector and type feature vector may be concatenated to obtain a character feature vector corresponding to the corresponding character; or, the content feature vector, position feature vector and type feature vector may be pooled to obtain a character feature vector corresponding to the corresponding character.

[0216] In the embodiment of the present application, in order to distinguish each text sentence, a sentence start marker indicating the starting position of the text sentence is added to the beginning of each text sentence. When vectorization is performed, the sentence start marker is vectorized together. Figure 3 As shown, for a text sentence "Beautiful scenery after snow in winter", a "[cls]" placeholder is added at the beginning of the sentence to separate different sentences. Of course, special marks can also be added at the end of each text sentence.

[0217] In addition, in order to distinguish different modes, special marks are added between text sentences of different modes for separation, for example, "[sep]" can be used for separation.

[0218] Step 702: based on the image feature information corresponding to each of the multiple video frames, obtain the image feature vectors corresponding to each of the multiple video frames.

[0219] In the present application examples, see Fig. 9 As shown, it is a schematic diagram of obtaining the image feature vector of a video frame. For multiple video frames, after using the CNN model to extract features respectively, the target image feature information of each video frame can be obtained, and then based on the target image feature information of each video frame, the content feature vector of each video frame can be obtained; and based on the position of each video frame in the target video frame, that is, the moment when the video frame appears in the video, the position feature vector of each video frame can be obtained accordingly, and based on the data type of the video frame, the type feature vector of each video frame can be obtained, wherein the data type is the video image type, and the type feature vector of each video frame indicates the video image type, that is, the video modality.

[0220] Furthermore, the image feature vector of each video frame can be obtained according to the content feature vector, the position feature vector and the type feature vector of each video frame.

[0221] Similar to text information, the content feature vector, position feature vector and type feature vector can be superimposed to obtain the image feature vector of the corresponding video frame; or, the content feature vector, position feature vector and type feature vector can be concatenated to obtain the image feature vector of the corresponding video frame; or, the content feature vector, position feature vector and type feature vector can be pooled to obtain the image feature vector of the corresponding video frame.

[0222] In an embodiment of the present application, the dimension of the image feature information embedding extracted by the CNN model may be different from that of the input layer of the video feature extraction model. For example, the embedding extracted by the CNN may be 2048, while the dimension of the input layer of the video feature extraction model is 1024. In this case, the dimension can be converted through fully connected layers (FC), that is, 2048 is mapped to 1024 dimensions.

[0223] Step 703: cascade processing is performed on each character feature vector and image feature vector obtained to obtain a target feature vector sequence.

[0224] In an embodiment of the present application, the character feature vectors of multiple characters included in each text sentence in each character can be cascaded to obtain a sentence feature vector of each text sentence, wherein the sentence feature vector includes the aforementioned sentence start mark for indicating the starting position of the text sentence, and then the sentence feature vectors of each text sentence are cascaded to obtain a sentence feature vector sequence, and text sentences of different data types are separated by type separation marks, and then the sentence feature vector sequence and each image feature vector are cascaded to obtain a target feature vector sequence.

[0225] Step 704: Using a serialized feature extraction method, extract features from the target feature vector sequence to obtain a video feature vector.

[0226] In the embodiment of the present application, for the obtained target feature vector sequence, a serialized feature extraction method can be used to extract features, thereby obtaining the corresponding video feature vector. Figure 3 As shown, the target feature vector sequence is input into the visual language model for feature extraction to obtain the video feature vector.

[0227] Specifically, as mentioned above, the visual language model can adopt a model structure composed of a multi-layer bidirectional Transformer encoder such as the BERT model. However, unlike the BERT model that only processes text semantics, the embodiment of the present application uses both image elements and text elements as inputs to the BERT model for feature extraction.

[0228] In the embodiment of the present application, in actual application, the obtained video feature vector is used for subsequent video business applications, such as video retrieval, search, recommendation, and clustering.

[0229] In a possible implementation, the trained video feature extraction model can be used to extract features from multiple videos to be clustered in the video library, that is, the target image feature information and target text information corresponding to each of these videos to be clustered are used to obtain corresponding video feature vectors, and then clustering processing can be performed based on the video feature vectors of these videos to be clustered to obtain multiple video sets, each of which contains at least two videos to be clustered whose similarity between each other is greater than a set similarity threshold, so that each video set is a similar video set. In this way, when a video data request for a first video is received from a target object, the target video set in which the first video is located can be determined from multiple video sets, and then while returning the video data of the first video to the target object, the video data of the remaining videos in the target video set are also returned.

[0230] For example, in a short video application, videos with the same highlights can be aggregated together and recommended to users, making it easier for users to browse a large number of similar videos. Fig.10 As shown, it is a schematic diagram of an interface for displaying similar videos in aggregate, wherein each video displayed in aggregate is a video related to "snow scene". Then, when a user wishes to play this type of video, he or she can easily find multiple videos with similar topics.

[0231] In summary, the embodiment of the present application adopts a method based on a visual language model to fuse the multimodal information of the video, and uses hashtag labels written by users for supervision and model training. The model can be trained without additional manual annotation, so that the model can be efficiently trained and iterated in massive data, and the extracted features fully express the semantics of the video.

[0232] In the embodiment of the present application, a separate sequence number of each sample video is used as the serial number label of the sample video, that is, each sample video is regarded as a category. In the process of training the feature extraction model based on the Arcface loss function, the distance between similar feature vectors will become closer and closer, while the distance between dissimilar feature vectors will become farther and farther. Therefore, after training the feature extraction model based on the Arcface loss function, the distinguishability between the video feature vectors extracted by the trained feature extraction model for dissimilar videos can be increased, and the video feature vectors extracted for similar videos are more compact, thereby facilitating the clustering of videos in the video set based on the video feature vectors of the video. At the same time, the Arcface loss function is introduced to constrain the video feature space, so that the same class is compressed into a more compact space, which is denser, so that the features learned by the network have more obvious angular distribution characteristics.

[0233] See also Fig.11 Based on the same inventive concept, the embodiment of the present application further provides a video feature extraction device 110, which includes:

[0234] The data preprocessing unit 1101 is used to obtain target image feature information of multiple video frames included in the target video to be processed, and obtain text information associated with the target video;

[0235] The feature extraction unit 1102 is used to input the target text information and the target image feature information into the trained video feature extraction model to obtain the target video feature vector of the target video;

[0236] Among them, the trained video feature extraction model is trained based on the predicted video labels obtained based on the sample video feature vectors corresponding to each sample video, and the corresponding real video labels; each sample video feature vector is obtained based on the sample text information and sample image feature information of the corresponding sample video, and the real video label of each sample video is obtained based on the topic tag added when the sample video is released.

[0237] Optionally, the data preprocessing unit 1101 is specifically configured to:

[0238] Extracting image features from the multiple video frames respectively to obtain target image feature information corresponding to the multiple video frames; and

[0239] Use one or more of the following methods to obtain the target text information:

[0240] From the title of the target video, extract the target text information contained in the title;

[0241] Using an image text recognition method, extracting target text information contained in each video frame included in the target video;

[0242] The target text information contained in the audio data included in the target video is extracted by adopting the speech-to-text recognition method.

[0243] Optionally, the feature extraction unit 1102 is specifically configured to:

[0244] Performing vectorization processing on each character included in the target text information to obtain a character feature vector corresponding to each character;

[0245] Based on target image feature information corresponding to each of the multiple video frames, image feature vectors corresponding to each of the multiple video frames are obtained;

[0246] Perform cascade processing on the obtained character feature vectors and image feature vectors to obtain a target feature vector sequence;

[0247] The serialized feature extraction method is used to extract features from the target feature vector sequence to obtain the target video feature vector.

[0248] Optionally, the feature extraction unit 1102 is specifically configured to:

[0249] For each text sentence included in the target text information, perform the following operations respectively:

[0250] For a plurality of characters included in a text sentence, obtaining content feature vectors corresponding to each of the plurality of characters from a preset character encoding dictionary;

[0251] Based on the positions of the multiple characters in the text sentence, respectively, position feature vectors corresponding to the multiple characters are obtained;

[0252] Based on the data type of the text sentence, obtaining type feature vectors corresponding to each of the multiple characters; wherein the data types include title text, image text and voice text;

[0253] According to the content feature vectors, the position feature vectors and the type feature vectors corresponding to the multiple characters, character feature vectors corresponding to the multiple characters are obtained.

[0254] Optionally, the feature extraction unit 1102 is specifically configured to:

[0255] Cascading the character feature vectors of the multiple characters included in each text sentence in each character to obtain a sentence feature vector of each text sentence; wherein the sentence feature vector includes a sentence start marker for indicating the starting position of the text sentence;

[0256] The sentence feature vectors of each text sentence are cascaded to obtain a sentence feature vector sequence; wherein type separation marks are added between text sentences of different text types in the sentence feature vector sequence;

[0257] The sentence feature vector sequence and each image feature vector are cascaded to obtain the target feature vector sequence.

[0258] Optionally, the feature extraction unit 1102 is specifically configured to:

[0259] For multiple video frames, perform the following operations respectively:

[0260] For a video frame, obtaining a content feature vector of the video frame according to target image feature information of the video frame;

[0261] Based on the position of the video frame in the target video, obtaining a position feature vector of the video frame;

[0262] Based on the data type of the video frame, obtaining a type feature vector of the video frame; wherein the data type includes a video image type;

[0263] An image feature vector of the video frame is obtained according to the content feature vector, the position feature vector and the type feature vector of the video frame.

[0264] Optionally, the real video tag of each sample video includes a topic tag and a sequence number tag, and one sequence number tag uniquely corresponds to one sample video. The device also includes a model training unit 1103, which is used to:

[0265] The trained video feature extraction model is obtained by multiple iterative training. Each iterative training includes the following operations:

[0266] Based on the sample text information and sample image feature information corresponding to each sample video input in this round, a sample video feature vector corresponding to each sample video is obtained;

[0267] For each sample video feature vector obtained, perform the following operations respectively:

[0268] For a sample video feature vector, multi-label classification processing is performed based on the sample video feature vector to obtain a topic tag feature vector of the sample video corresponding to the sample video feature vector; wherein the topic tag feature vector includes multiple first indicator bits, one first indicator bit corresponds to one of all topic tags, and the value of each first indicator bit is used to indicate whether the sample video has

[0269] There are corresponding topic tags;

[0270] Performing feature mapping based on the sample video feature vector to obtain a sequence number label feature vector of the corresponding sample video; wherein the sequence number label feature vector includes a plurality of second indicator bits, one second indicator bit corresponds to one sequence number label, and the value of each second indicator bit is used to represent the probability that the predicted sequence number of the sample video is the corresponding sequence number label;

[0271] Based on the obtained feature vectors of each topic tag and the topic tag of the corresponding sample video, a topic classification loss value is obtained, and based on the obtained feature vectors of each sequence number tag and the sequence number tag of the corresponding sample video, a sequence number classification loss value is obtained;

[0272] Based on the topic classification loss value and the sequence classification loss value, the model loss value of this round of iterative training is determined, and the model parameters are adjusted based on the model loss value.

[0273] Optionally, the model training unit 1103 is specifically used for:

[0274] For each obtained topic tag feature vector, perform the following operations respectively:

[0275] For a topic tag feature vector, based on the values ​​of each first indicator bit included in the topic tag feature vector, the topic tag of the corresponding sample video, and the set weight parameter and modulation parameter, the sub-loss value of the corresponding sample video is obtained; wherein the weight parameter is used to balance the influence of positive samples and negative samples on the topic classification loss value, and the modulation parameter is used to balance the influence of simple samples and difficult samples on the topic classification loss value;

[0276] Based on the sub-loss values ​​corresponding to each sample video, the topic classification loss value is obtained.

[0277] Optionally, the model training unit 1103 is specifically used for:

[0278] For each obtained serial number label feature vector, perform the following operations respectively:

[0279] For a sequence number label feature vector, obtain a first value of a target indicator bit corresponding to the sequence number label of the corresponding sample video in the sequence number label feature vector, and obtain a second value of the remaining second indicator bits in the sequence number label feature vector except the target indicator bit;

[0280] Based on the set spatial margin, spatially transform the first value in the set space to obtain a third value;

[0281] Based on the obtained second values ​​and third values, obtain a sub-loss value of the corresponding sample video;

[0282] Based on the sub-loss values ​​corresponding to each sample video, the serial classification loss value is obtained.

[0283] Optionally, the device further includes an execution unit 1104, configured to:

[0284] Obtaining target video feature vectors corresponding to each of the multiple videos to be clustered based on target image feature information and target text information corresponding to each of the multiple videos to be clustered, using the trained video feature extraction model;

[0285] Performing clustering processing based on target video feature vectors of multiple videos to be clustered to obtain multiple video sets, each video set containing at least two videos to be clustered whose similarity between each other is greater than a set similarity threshold;

[0286] In response to a video data request of a target object for a first video, determining a target video set in which the first video is located from a plurality of video sets;

[0287] Based on the target video set, the video data of the first video and the video data of the remaining videos in the target video set are returned to the target object.

[0288] When the above device trains the video feature extraction model, it uses the topic tags added when the sample video is released as the real video tags, so there is no need for manual labeling, which avoids the high error rate caused by manual labeling, improves the accuracy of the trained model, and reduces the labor cost of model training and the large amount of time consumed by manual labeling, so that the model training efficiency is higher. In addition, the topic tags added when the sample video is released are usually closely related to the content of the video, so they can more accurately express the semantics of the video and are more in line with the actual video application scenario, so that not only can the trained model accurately express the semantic content of the video, but also can be more in line with the actual video application scenario.

[0289] In addition, when the above-mentioned device is training, it extracts features based on the multimodal information of the video, that is, text information and image feature information, so that the extracted video feature vector can fully express the video content and improve the accuracy of the video content expression. In addition, considering the problem of large amount of computation in traditional CNN model training, the embodiment of the present application adopts a method based on a visual-language model to fuse the multimodal information of the video, so that the model can be efficiently trained and iterated. At the same time, the idea of ​​the face recognition algorithm ArcFace is introduced in the training process, and the video feature space is constrained based on the loss function used for face recognition, so that the same class is compressed into a more compact space, which is denser, so that the features learned by the network have more obvious angular distribution characteristics.

[0290] The device can be used to execute the methods shown in the various embodiments of the present application. Therefore, for the functions that can be implemented by the various functional modules of the device, reference can be made to the description of the aforementioned embodiments and no further details will be given.

[0291] See also Fig.12 Based on the same technical concept, the present application embodiment also provides a computer device 120, which can be Figure 1 As shown in the terminal device or server, the computer device 120 may include a memory 1201 and a processor 1202 .

[0292] The memory 1201 is used to store computer programs executed by the processor 1202. The memory 1201 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function, etc.; the data storage area may store data created according to the use of the computer device, etc. The processor 1202 may be a central processing unit (CPU), or a digital processing unit, etc. The specific connection medium between the above-mentioned memory 1201 and the processor 1202 is not limited in the embodiments of the present application. The embodiments of the present application are Fig.12 In the embodiment, the memory 1201 and the processor 1202 are connected via a bus 1203. The bus 1203 is connected to the processor 1202 via a bus 1203. Fig.12 The connection between other components is shown by bold lines, and is not intended to be limiting. The bus 1203 can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Fig.12 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0293] The memory 1201 may be a volatile memory, such as a random-access memory (RAM); the memory 1201 may also be a non-volatile memory, such as a read-only memory, a flash memory, a hard disk drive (HDD) or a solid-state drive (SSD), or the memory 1201 is any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto. The memory 1201 may be a combination of the above memories.

[0294] The processor 1202 is used to execute the following when calling the computer program stored in the memory 1201: Figure 2 to Figure 10 The method executed by the device in the illustrated embodiment.

[0295] In some possible implementations, various aspects of the method provided in the present application may also be implemented in the form of a program product, which includes a program code. When the program product is run on a computer device, the program code is used to enable the computer device to execute the steps of the method according to various exemplary embodiments of the present application described above in this specification. For example, the computer device may execute the following steps: Figure 2 to Figure 10 The method executed by the device in the illustrated embodiment.

[0296] The program product may use any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0297] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications falling within the scope of the present application.

[0298] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A video feature extraction method, characterized in that: The method comprises: For a target video to be processed, obtaining target image feature information of a plurality of video frames included in the target video, and obtaining target text information associated with the target video; Perform the following operations using the trained video feature extraction model: For each text sentence included in the target text information, perform the following operations respectively: For a plurality of characters included in a text sentence, obtaining content feature vectors corresponding to each of the plurality of characters from a preset character encoding dictionary; Based on the positions of the multiple characters in the text sentence, respectively, obtaining position feature vectors corresponding to the multiple characters; Based on the data type of the text sentence, obtain the type feature vectors corresponding to each of the multiple characters; wherein the data types include title text, image text and voice text, the image text is the text contained in each video frame included in the target video, and the voice text is the text contained in the audio data included in the target video; Obtaining character feature vectors corresponding to each of the plurality of characters according to the content feature vectors, the position feature vectors and the type feature vectors corresponding to each of the plurality of characters; Obtaining a target video feature vector of the target video according to the character feature vectors corresponding to the respective characters included in the target text information and the target image feature information; Among them, the trained video feature extraction model is obtained by adjusting the model parameters based on the model loss value during training, and the model loss value is determined based on the predicted video label obtained based on the sample video feature vector corresponding to each sample video, and the corresponding real video label; each sample video feature vector is obtained based on the sample text information and sample image feature information of the corresponding sample video, and the real video label of each sample video is obtained based on the topic tag added when the sample video is released.

2. The method according to claim 1, characterized in that For a target video to be processed, obtaining target image feature information of a plurality of video frames included in the target video, and obtaining target text information associated with the target video, including: Extracting image features from the multiple video frames respectively to obtain target image feature information corresponding to each of the multiple video frames; and The target text information is obtained by using one or more of the following methods: Extracting target text information contained in the title from the title of the target video; Using an image text recognition method, extracting target text information contained in each video frame included in the target video; A speech-to-text recognition method is adopted to extract target text information contained in the audio data included in the target video.

3. The method according to claim 1, characterized in that The step of obtaining a target video feature vector of the target video according to the character feature vectors corresponding to the respective characters included in the target text information and the target image feature information comprises: Based on the target image feature information corresponding to each of the multiple video frames, obtaining the image feature vector corresponding to each of the multiple video frames; Perform cascade processing on the obtained character feature vectors and image feature vectors to obtain a target feature vector sequence; A serialized feature extraction method is adopted to extract features from the target feature vector sequence to obtain the target video feature vector.

4. The method according to claim 3, characterized in that The obtained character feature vectors and image feature vectors are cascaded to obtain a target feature vector sequence, including: Cascading the character feature vectors of the multiple characters included in each text sentence in the characters to obtain a sentence feature vector of each text sentence; wherein the sentence feature vector includes a sentence start marker for indicating the starting position of the text sentence; The sentence feature vectors of each text sentence are cascaded to obtain a sentence feature vector sequence; wherein type separation marks are added between text sentences of different text types in the sentence feature vector sequence; The sentence feature vector sequence and each image feature vector are cascaded to obtain the target feature vector sequence.

5. The method according to claim 3, characterized in that Based on the target image feature information corresponding to each of the multiple video frames, obtaining the image feature vectors corresponding to each of the multiple video frames includes: For the multiple video frames, perform the following operations respectively: For a video frame, obtaining a content feature vector of the video frame according to target image feature information of the video frame; Based on the position of the one video frame in the target video, obtaining a position feature vector of the one video frame; Based on the data type of the video frame, obtaining a type feature vector of the video frame; wherein the data type includes a video image type; An image feature vector of the video frame is obtained according to the content feature vector, the position feature vector and the type feature vector of the video frame.

6. The method according to claim 1, characterized in that The real video label of each sample video includes a topic label and a serial number label. One serial number label uniquely corresponds to one sample video. The trained video feature extraction model is obtained by the following training process: The trained video feature extraction model is obtained by training in a manner of multiple iterative training; wherein, in each iterative training, the following operations are performed: Based on the sample text information and sample image feature information corresponding to each sample video input in this round, respectively, a sample video feature vector corresponding to each sample video is obtained; For each sample video feature vector obtained, perform the following operations respectively: For a sample video feature vector, multi-label classification processing is performed based on the sample video feature vector to obtain a topic tag feature vector of the sample video corresponding to the sample video feature vector; wherein the topic tag feature vector includes a plurality of first indicator bits, one first indicator bit corresponds to one of all topic tags, and the value of each first indicator bit is used to indicate whether the sample video has a corresponding topic tag; Performing feature mapping based on the feature vector of the sample video to obtain a sequence number label feature vector of the sample video; wherein the sequence number label feature vector includes a plurality of second indicator bits, one second indicator bit corresponds to one sequence number label, and the value of each second indicator bit is used to characterize the probability that the predicted sequence number of the sample video is the corresponding sequence number label; Based on the obtained feature vectors of each topic tag and the topic tag of the corresponding sample video, a topic classification loss value is obtained, and based on the obtained feature vectors of each sequence number tag and the sequence number tag of the corresponding sample video, a sequence number classification loss value is obtained; Based on the topic classification loss value and the sequence number classification loss value, the model loss value of this round of iterative training is determined, and the model parameters are adjusted based on the model loss value.

7. The method according to claim 6, characterized in that Based on the obtained feature vectors of each topic tag and the topic tags of the corresponding sample videos, the topic classification loss value is obtained, including: For each obtained topic tag feature vector, perform the following operations respectively: For a topic tag feature vector, based on the values ​​of each first indicator bit included in the topic tag feature vector, the topic tag of the corresponding sample video, and the set weight parameter and modulation parameter, a sub-loss value of the corresponding sample video is obtained; wherein the weight parameter is used to balance the influence of positive samples and negative samples on the topic classification loss value, and the modulation parameter is used to balance the influence of simple samples and difficult samples on the topic classification loss value; Based on the sub-loss values ​​corresponding to each sample video, the topic classification loss value is obtained.

8. The method according to claim 6, characterized in that Based on the obtained feature vectors of each serial number label and the serial number label of the corresponding sample video, the serial number classification loss value is obtained, including: For each obtained serial number label feature vector, perform the following operations respectively: For a sequence number label feature vector, obtaining a first value of a target indicator bit corresponding to a sequence number label of a corresponding sample video in the sequence number label feature vector, and obtaining a second value of a second indicator bit other than the target indicator bit in the sequence number label feature vector; Based on the set space margin, spatially transform the first value in the set space to obtain a third value; Based on each of the obtained second values ​​and the third value, obtaining a sub-loss value of a corresponding sample video; Based on the sub-loss values ​​corresponding to each sample video, the serial number classification loss value is obtained.

9. The method according to any one of claims 1 to 8, characterized in that: The method further comprises: Obtaining target video feature vectors corresponding to each of the multiple videos to be clustered based on target image feature information and target text information corresponding to each of the multiple videos to be clustered, using the trained video feature extraction model; Performing clustering processing based on the target video feature vectors of the multiple videos to be clustered to obtain multiple video sets, each video set containing at least two videos to be clustered whose similarity between each other is greater than a set similarity threshold; In response to a video data request made by a target object for a first video, determining a target video set in which the first video is located from the multiple video sets; Based on the target video set, the video data of the first video and the video data of the remaining videos in the target video set are returned to the target object.

10. A video feature extraction device, characterized in that: The device comprises: A data preprocessing unit, configured to obtain target image feature information of a plurality of video frames included in a target video to be processed, and obtain target text information associated with the target video; The feature extraction unit is used to perform the following operations using the trained video feature extraction model: For each text sentence included in the target text information, perform the following operations respectively: For a plurality of characters included in a text sentence, obtaining content feature vectors corresponding to each of the plurality of characters from a preset character encoding dictionary; Based on the positions of the multiple characters in the text sentence, respectively, obtaining position feature vectors corresponding to the multiple characters; Based on the data type of the text sentence, obtain the type feature vectors corresponding to each of the multiple characters; wherein the data types include title text, image text and voice text, the image text is the text contained in each video frame included in the target video, and the voice text is the text contained in the audio data included in the target video; Obtaining character feature vectors corresponding to each of the plurality of characters according to the content feature vectors, the position feature vectors and the type feature vectors corresponding to each of the plurality of characters; Obtaining a target video feature vector of the target video according to the character feature vectors corresponding to the respective characters included in the target text information and the target image feature information; Among them, the trained video feature extraction model is obtained by adjusting the model parameters based on the model loss value during training, and the model loss value is determined based on the predicted video label obtained based on the sample video feature vector corresponding to each sample video, and the corresponding real video label; each sample video feature vector is obtained based on the sample text information and sample image feature information of the corresponding sample video, and the real video label of each sample video is obtained based on the topic tag added when the sample video is released.

11. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 9 are implemented.

12. A computer storage medium having computer program instructions stored thereon, characterized in that: When the computer program instructions are executed by a processor, the steps of the method according to any one of claims 1 to 9 are implemented.

Images