Video feature extraction network training method, device, electronic device and storage medium

Abstract

The present disclosure relates to a video feature extraction network training method, device, electronic device and storage medium, the method comprising: determining a target triple linked to a first sample video in a first sample video set in a multimodal knowledge graph; extracting video features of the first sample video based on a video coding network and determining a first video classification loss; extracting features of a head entity in the target triple based on a label coding network, and determining a first contrastive learning loss based on the features of the head entity and the video features of the corresponding first sample video; extracting features of a tail entity in the target triple based on a label coding network and determining a vector representation of the target triple in combination with video features; determining a knowledge graph embedding loss based on the vector representation of the target triple; and adjusting network parameters based on the above three losses to obtain a video feature extraction network. The video features extracted by the video feature extraction network disclosed herein contain richer information and can better meet the needs of video understanding services.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to a video feature extraction network training method, device, electronic device, and storage medium. Background Art

[0002] Video understanding is a very important task in video scenarios. Video understanding is based on video features. When extracting video features, related technologies only focus on the content that appears in the video, such as video frames, audio, and text. As a result, the video features extracted based on related technologies contain insufficient information and cannot meet the needs of video understanding services. Summary of the Invention

[0003] The present disclosure provides a video feature extraction network training method, device, electronic device, and storage medium to at least address the problem that the video features extracted in related technologies do not contain sufficient information to meet the needs of video understanding services. The technical solutions of the present disclosure are as follows:

[0004] According to a first aspect of an embodiment of the present disclosure, a video feature extraction network training method is provided, comprising:

[0005] Determining a target triplet linked to a first sample video in a first sample video set in a multimodal knowledge graph; each of the first sample videos is linked to a head entity of a triplet in the multimodal knowledge graph, and the linked head entity represents a category label of the first sample video linked to the head entity;

[0006] Extracting video features of each of the first sample videos based on a video coding network, and determining a first video classification loss based on the video features and the annotated category labels of each of the first sample videos;

[0007] extracting features of a head entity in a target triplet corresponding to each of the first sample videos based on a label encoding network, and determining a first contrastive learning loss based on features of the head entity in the target triplet and video features of the first sample video linked to the head entity in the target triplet;

[0008] Extracting features of a tail entity in the target triplet corresponding to each of the first sample videos based on the label encoding network, and determining a vector representation of the target triplet corresponding to each of the first sample videos based on the features of the tail entity in the target triplet corresponding to each of the first sample videos and the video features of each of the first sample videos; wherein the video features of the first sample videos serve as feature representations of the head entity in the target triplet corresponding to the first sample videos;

[0009] Determining a knowledge graph embedding loss based on a vector representation of a target triplet corresponding to each of the first sample videos;

[0010] Joint training is performed based on the first video classification loss, the first contrastive learning loss, and the knowledge graph embedding loss. During the joint training process, network parameters in the video encoding network and the label encoding network are adjusted until a first preset training end condition is reached to obtain a video feature extraction network; the video feature extraction network includes the video encoding network at the end of training.

[0011] In an exemplary embodiment, the method further comprises:

[0012] Obtain a second sample video set and an original knowledge graph; the second sample video in the second sample video set corresponds to a category label;

[0013] Linking each second sample video in the second sample video set with an entity in the original knowledge graph to obtain an entity linking result corresponding to each second sample video;

[0014] For each entity linking result corresponding to the second sample video, when the entity linking result indicates a link to a head entity and the head entity is consistent with the category label of the second sample video to which the head entity is linked, determining the entity linking result as a target entity linking result;

[0015] The multimodal knowledge graph is generated based on the target entity link result and the second sample video corresponding to the target entity link result.

[0016] In an exemplary embodiment, the method further comprises:

[0017] performing feature extraction on a second sample video in the second sample video set based on the video encoding network to obtain a first video feature of the second sample video;

[0018] Extracting features of the category label corresponding to the second sample video based on the initial label encoding network to obtain sample label features corresponding to the second sample video;

[0019] taking the first video feature of the second sample video and the sample label feature of the second sample video as a positive sample pair, and determining a second contrastive learning loss based on the positive sample pair;

[0020] The network parameters of the video encoding network are fixed, and the network parameters of the initial label encoding network are adjusted based on the second contrastive learning loss until a second preset training end condition is met to obtain the label encoding network.

[0021] In an exemplary embodiment, the method further comprises:

[0022] performing feature extraction on the second sample video in the second sample video set based on the initial video coding network to obtain second video features of the second sample video;

[0023] determining a video classification result of the second sample video based on the second video feature of the second sample video;

[0024] determining a second video classification loss based on the video classification result of the second sample video and the category label of the second sample video;

[0025] The network parameters of the initial video coding network are adjusted based on the second video classification loss until a third preset training end condition is reached to obtain the video coding network.

[0026] In an exemplary embodiment, determining the vector representation of the target triple corresponding to each first sample video based on the features of the tail entity in the target triple corresponding to each first sample video and the video features of each first sample video includes:

[0027] Embedding the relationship in the target triplet corresponding to the first sample video based on the embedding network to obtain a relationship embedding representation;

[0028] Using the video features of the first sample video as feature representations of the head entity in the target triplet corresponding to the first sample video;

[0029] Based on the feature representation of the head entity in the target triplet corresponding to the first sample video, the relation embedding representation and the feature of the tail entity in the target triplet, a vector representation of the target triplet corresponding to the first sample video is obtained.

[0030] In an exemplary embodiment, determining the knowledge graph embedding loss based on the vector representation of the target triple corresponding to each of the first sample videos includes:

[0031] Taking the vector representation of the target triplet corresponding to the first sample video as a positive sample;

[0032] Randomly sampling from the multimodal knowledge graph to determine a second preset number of sampling tail entities;

[0033] Based on the features of the second preset number of sampled tail entities, the features of the tail entities in the positive samples are respectively replaced to obtain a second preset number of negative samples corresponding to the positive samples;

[0034] Determine a knowledge graph embedding loss based on the positive sample and the second preset number of negative samples.

[0035] In an exemplary embodiment, extracting video features of the first sample video based on a video coding network includes:

[0036] Extracting a first preset number of video frames from the first sample video, and performing serialization processing on the first preset number of video frames to obtain a video frame sequence;

[0037] Determining a video text and an audio text of the first sample video, and performing serialization processing on the video text and the audio text of the first sample video to obtain a text sequence;

[0038] splicing the video frame sequence and the text sequence according to a preset format to obtain a sample input sequence;

[0039] The sample input sequence is input into a video encoding network for encoding, and video features of the first sample video are obtained based on features used for classification in the encoding result.

[0040] According to a second aspect of an embodiment of the present disclosure, a video feature extraction network training device is provided, comprising:

[0041] a target triplet determination unit, configured to determine a target triplet to which a first sample video in a first sample video set is linked in a multimodal knowledge graph; each first sample video is linked to a head entity of a triplet in the multimodal knowledge graph, and the linked head entity represents a category label of the first sample video linked to the head entity;

[0042] a first classification loss determining unit, configured to extract video features of each of the first sample videos based on a video coding network, and determine a first video classification loss based on the video features and the annotated category labels of each of the first sample videos;

[0043] a first contrastive learning loss determining unit, configured to extract features of a head entity in a target triplet corresponding to each of the first sample videos based on a label encoding network, and determine a first contrastive learning loss based on features of the head entity in the target triplet and video features of the first sample video linked to the head entity in the target triplet;

[0044] a triplet vector representation unit, configured to extract features of a tail entity in a target triplet corresponding to each of the first sample videos based on the label encoding network, and determine a vector representation of the target triplet corresponding to each of the first sample videos based on the features of the tail entity in the target triplet corresponding to each of the first sample videos and video features of the first sample videos; wherein the video features of the first sample videos serve as feature representations of the head entity in the target triplet corresponding to the first sample videos;

[0045] a knowledge graph embedding loss determining unit, configured to determine a knowledge graph embedding loss based on a vector representation of a target triple corresponding to each of the first sample videos;

[0046] A training unit is used to perform joint training based on the first video classification loss, the first contrastive learning loss and the knowledge graph embedding loss, and adjust network parameters in the video encoding network and the label encoding network during the joint training process until a first preset training end condition is reached to obtain a video feature extraction network; the video feature extraction network includes the video encoding network at the end of training.

[0047] In an exemplary embodiment, the apparatus further comprises:

[0048] The first acquisition unit is configured to acquire a second sample video set and an original knowledge graph; the second sample video in the second sample video set corresponds to a category label;

[0049] a linking unit, configured to link each second sample video in the second sample video set with an entity in the original knowledge graph to obtain an entity linking result corresponding to each second sample video;

[0050] a screening unit, configured to, for each entity linking result corresponding to the second sample video, determine the entity linking result as a target entity linking result when the entity linking result indicates a link to a head entity and the head entity is consistent with a category label of the second sample video to which the head entity is linked;

[0051] A multimodal knowledge graph generation unit is used to generate the multimodal knowledge graph based on the target entity link result and the second sample video corresponding to the target entity link result.

[0052] In an exemplary embodiment, the apparatus further comprises:

[0053] a first feature extraction unit, configured to extract features of a second sample video in the second sample video set based on the video coding network to obtain a first video feature of the second sample video;

[0054] A second feature extraction unit is configured to extract features of the category label corresponding to the second sample video based on the initial label encoding network to obtain a sample label feature corresponding to the second sample video;

[0055] a second contrastive learning loss determining unit, configured to take the first video feature of the second sample video and the sample label feature of the second sample video as a positive sample pair, and determine a second contrastive learning loss based on the positive sample pair;

[0056] A first parameter adjustment unit is used to fix the network parameters of the video encoding network and adjust the network parameters of the initial label encoding network based on the second contrastive learning loss until a second preset training end condition is met to obtain the label encoding network.

[0057] In an exemplary embodiment, the apparatus further comprises:

[0058] a third feature extraction unit, configured to extract features of the second sample video in the second sample video set based on the initial video coding network to obtain second video features of the second sample video;

[0059] a classification result determining unit, configured to determine a video classification result of the second sample video based on a second video feature of the second sample video;

[0060] A second classification loss determining unit, configured to determine a second video classification loss based on the video classification result of the second sample video and the category label of the second sample video;

[0061] A second parameter adjustment unit is used to adjust the network parameters of the initial video coding network based on the second video classification loss until a third preset training end condition is reached to obtain the video coding network.

[0062] In an exemplary embodiment, the triple vector representation unit includes:

[0063] a relation embedding unit, configured to embed the relation in the target triple corresponding to the first sample video based on an embedding network to obtain a relation embedding representation;

[0064] a head entity feature determining unit, configured to use the video feature of the first sample video as a feature representation of the head entity in the target triplet corresponding to the first sample video;

[0065] The triplet vector representation subunit is used to obtain the vector representation of the target triplet corresponding to the first sample video based on the feature representation of the head entity in the target triplet corresponding to the first sample video, the relationship embedding representation and the feature of the tail entity in the target triplet.

[0066] In an exemplary embodiment, the knowledge graph embedding loss determination unit includes:

[0067] a positive sample determining unit, configured to take the vector representation of the target triplet corresponding to the first sample video as a positive sample;

[0068] A random sampling unit, configured to randomly sample and determine a second preset number of sampling tail entities from the multimodal knowledge graph;

[0069] a negative sample determining unit, configured to replace the features of the tail entities in the positive sample based on the features of the second preset number of sampled tail entities, to obtain a second preset number of negative samples corresponding to the positive sample;

[0070] The knowledge graph embedding loss determination subunit is used to determine the knowledge graph embedding loss based on the positive sample and the second preset number of negative samples.

[0071] In an exemplary embodiment, the first classification loss determining unit includes:

[0072] a video frame serialization unit, configured to extract a first preset number of video frames from the first sample video, and perform serialization processing on the first preset number of video frames to obtain a video frame sequence;

[0073] a text serialization unit, configured to determine a video text and an audio text of the first sample video, and perform serialization processing on the video text and the audio text of the first sample video to obtain a text sequence;

[0074] an input sequence determining unit, configured to splice the video frame sequence and the text sequence according to a preset format to obtain a sample input sequence;

[0075] The encoding unit is used to input the sample input sequence into a video encoding network for encoding, and classify the features used for classification in the encoding result to obtain the video features of the first sample video.

[0076] According to a third aspect of an embodiment of the present disclosure, there is provided an electronic device, including:

[0077] processor;

[0078] a memory for storing instructions executable by the processor;

[0079] The processor is configured to execute the instructions to implement the video feature extraction network training method of the first aspect mentioned above.

[0080] According to a fourth aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided. When the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the video feature extraction network training method of the first aspect mentioned above.

[0081] According to a fifth aspect of an embodiment of the present disclosure, a computer program product is provided, comprising a computer program, which, when executed by a processor, implements the video feature extraction network training method of the first aspect.

[0082] The technical solutions provided by the embodiments of the present disclosure bring at least the following beneficial effects:

[0083] By introducing a multimodal knowledge graph, and extracting the video features of the first sample video linked in the multimodal knowledge graph based on a video coding network, determining a first video classification loss based on the video features of each first sample video, and extracting the features of the head entity in the target triple corresponding to each first sample video based on a label coding network, determining a first contrastive learning loss based on the features of the head entity and the video features of the corresponding first sample video, extracting the features of the tail entity in the target triple corresponding to the first sample video based on the label coding network, and determining a vector representation of the target triple based on the features of the tail entity and the video features of the corresponding first sample video, and then determining a vector representation of the target triple corresponding to each first sample video. Determine the knowledge graph embedding loss, and combine the above-mentioned first video classification loss, first contrastive learning loss and knowledge graph embedding loss to jointly train and adjust the parameters of each network to obtain a video feature extraction network. The video feature extraction network includes a video encoding network at the end of training, so that the information in the video and the common sense knowledge in the knowledge graph can be fully mined during the training process, so that the video features extracted by the trained video feature extraction network contain richer information, which can not only fully express the content characteristics of the video, but also be embedded in the feature space of the knowledge graph, and have the ability of common sense reasoning, which greatly improves the accuracy of video feature extraction and better meets the needs of video understanding business.

[0084] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0085] The accompanying drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description are used to explain the principles of the present disclosure, and do not constitute an improper limitation of the present disclosure.

[0086] Figure 1 is a flow chart of a video feature extraction network training method according to an exemplary embodiment;

[0087] Figure 2 An example of a multimodal knowledge graph is shown according to an exemplary embodiment.

[0088] Figure 3 is a flow chart illustrating another video feature extraction network training method according to an exemplary embodiment;

[0089] Figure 4 is a flow chart illustrating another video feature extraction network training method according to an exemplary embodiment;

[0090] Figure 5 is a schematic diagram of a joint training architecture according to an exemplary embodiment;

[0091] Figure 6 is a structural block diagram of a video feature extraction network training device according to an exemplary embodiment;

[0092] Figure 7 It is a block diagram of an electronic device according to an exemplary embodiment. DETAILED DESCRIPTION

[0093] In order to enable ordinary persons in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below with reference to the accompanying drawings.

[0094] It should be noted that the terms "first," "second," and the like in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or precedence. It should be understood that the numbers used in this manner are interchangeable where appropriate so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of apparatus and methods consistent with certain aspects of the present disclosure as detailed in the appended claims.

[0095] It should also be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for display, data for analysis, etc.) involved in this disclosure are all information and data authorized by the user or fully authorized by all parties.

[0096] An embodiment of the present disclosure provides a video feature extraction network training method, which can be applied to a video feature extraction network training device of an embodiment of the present disclosure. The video feature extraction network training device can be configured in an electronic device, which can be a terminal or a server.

[0097] The terminal may be, but is not limited to, a smartphone, tablet computer, laptop computer, or desktop computer. Client software, such as an application (App), providing video processing functionality may be installed in the terminal. The application may be a standalone application or a subroutine within the application. For example, the application may include a live streaming application, a short video application, or the like. Terminal users may log in to the application using pre-registered user information, which may include an account number and password.

[0098] The server can be a server that provides background services for applications in the terminal. The background services may include video understanding services. The video understanding services can be implemented based on video features extracted from the video to be processed by a video feature extraction network, wherein the video feature extraction network is a neural network trained using the video feature extraction network training method of the embodiment of the present disclosure. Specifically, the server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers. It can also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms.

[0099] See also Figure 1 , which is a flow chart of a video feature extraction network training method according to an exemplary embodiment. Figure 1 As shown, the following steps are included.

[0100] In step S101, a target triple linked to a first sample video in a first sample video set in a multimodal knowledge graph is determined.

[0101] Each first sample video in the first sample video set is linked to a head entity of a triple in the multimodal knowledge graph, and the linked head entity represents the category label of the first sample video linked to the head entity.

[0102] The multimodal knowledge graph in the disclosed embodiment includes a multimodal entity portion and a commonsense knowledge portion. The multimodal entity portion includes multiple first sample videos, each of which can be annotated with a category label. The commonsense knowledge portion contains a very rich set of commonsense knowledge triples, which can be derived from a knowledge graph dataset. Each first sample video in the multimodal entity portion is linked to the head entity of a triple in the commonsense knowledge portion, and the head entity of the link is the category label of the first sample video.

[0103] like Figure 2 The figure shows an example of a multimodal knowledge graph provided by an embodiment of the present disclosure. The multimodal knowledge graph includes a common sense knowledge part and a multimodal entity part from the knowledge graph, wherein the multimodal entity part includes a first sample video set (such as video 1, video 2, video 3, and video 4) and a category label set (such as category label 1, category label 2, category label 3, and category label 4). The common sense knowledge part includes multiple triples, each of which can be represented as (head entity, relationship, tail entity). Figure 2The entities in are the representations of head entities and tail entities. It should be noted that the head entity of a triple in a multimodal knowledge graph will be linked to at least one first sample video, but some of its tail entities may not be linked to any first sample video. For the head entity linked to the first sample video, a head entity in a text modality may be linked to more than one first sample video, but each first sample video can only be linked to one unique head entity.

[0104] The first sample video set in the embodiment of the present disclosure may be all the first sample videos in the multimodal knowledge graph, or may be part of the first sample videos in the multimodal knowledge graph.

[0105] Based on this, before the above step S101, the method of the embodiment of the present disclosure may further include generating a multimodal knowledge graph. Specifically, generating a multimodal knowledge graph may include Figure 3 The following steps:

[0106] In step S301, a second sample video set and an original knowledge graph are obtained.

[0107] The second sample videos in the second sample video set are each associated with a category label, which may represent a summary of the video content of the corresponding second sample video. The category label for each second sample video may be generated based on the category information annotated by the publisher of the corresponding video, or may be generated after classification processing based on a classification model. The present embodiment does not specifically limit the method for generating the category label for the second sample video.

[0108] The original knowledge graph is a commonsense knowledge graph in text mode, that is, the triples therein are all represented by text. For example, the original knowledge graph can be a knowledge graph extracted from the Chinese encyclopedia knowledge graph that matches the video processing application.

[0109] In step S303, each second sample video in the second sample video set is linked to an entity in the original knowledge graph to obtain an entity link result corresponding to each second sample video.

[0110] In a specific implementation, each second sample video in the second sample video set can be linked to an entity in the original knowledge graph through a multimodal entity linking algorithm. It is understood that the linked entities at this time may include a head entity and a tail entity. The multimodal entity linking algorithm can be found in related art and will not be described in detail here.

[0111] In step S305, for each entity linking result corresponding to the second sample video, when the entity linking result indicates a link to a head entity and the head entity is consistent with the category label of the second sample video to which the head entity is linked, the entity linking result is determined as the target entity linking result.

[0112] Specifically, after obtaining the entity linking result corresponding to each second sample video, filtering processing can be performed based on the entity linking result to find the target entity linking result, which indicates that the corresponding second sample video is linked to the head entity, and the head entity is consistent with the category label of the second sample video to which it is linked.

[0113] In step S307, a multimodal knowledge graph is generated based on the target entity linking result and the second sample video corresponding to the target entity linking result.

[0114] That is to say, the multimodal entity part of the multimodal knowledge graph includes the second sample video corresponding to the target entity link result (in the embodiment of the present disclosure, the second sample video corresponding to the target entity link result is referred to as the aforementioned first sample video), and the common sense knowledge part of the multimodal knowledge graph includes the triple corresponding to the target entity link result, and the head entity of the triple is linked to the second sample video corresponding to the target entity link result.

[0115] The above implementation generates a video-based multimodal knowledge graph by linking and filtering the second sample video in the second sample video set with the original knowledge graph. The multimodal knowledge graph contains rich multimodal video entities and common-sense triple relationships.

[0116] In step S103, video features of each first sample video are extracted based on the video coding network, and a first video classification loss is determined based on the video features of each first sample video and the annotated category label.

[0117] In the disclosed embodiment, video understanding is modeled as a classification task, the classification target is the category label of the video, and the input of the classification head is used as the feature representation of the corresponding video. Exemplarily, the pre-trained video encoding network can be a Transformer-based video feature extraction network.

[0118] Specifically, for each first sample video in the first sample video set, video features of the first sample video are extracted based on a pre-trained video coding network, and then video classification is performed on the first sample video based on the video features of the first sample video to obtain a video classification result corresponding to the first sample video, thereby determining a first video classification loss based on the difference between the video classification results of each first sample video in the first sample video set and the category label annotated with the first sample video. In a specific implementation, the first video classification loss can be calculated using a cross-entropy loss.

[0119] In order to enable the video feature extraction network to have multimodal understanding capabilities, in an exemplary embodiment, the above step S103 may include, when extracting the video features of each first sample video based on the video encoding network:

[0120] (1) Extracting a first preset number of video frames from a first sample video, and performing serialization processing on the first preset number of video frames to obtain a video frame sequence.

[0121] The first preset number may be set based on actual needs, for example, it may be 4 or 5.

[0122] The extraction method can also be set based on actual needs, for example, it can be random extraction or uniform extraction.

[0123] In a specific implementation, when the first preset number of video frames are serialized, the serialization can be performed based on a pre-trained ViT (Vision Transformer) model. The ViT model divides the input image into multiple image blocks (such as 16x16), which are then composed into a patch sequence, and each patch in the patch sequence is projected into a vector of a fixed length and fed into the Transformer. In the embodiment of the present disclosure, the ViT model can be used to divide the first preset number of video frames into multiple patches and compose a patch sequence to obtain a video frame sequence.

[0124] (2) Determine the video text and audio text of the first sample video, and perform serialization processing on the video text and audio text of the first sample video to obtain a text sequence.

[0125] The video text of the first sample video may include subtitle text, title text, and description information edited by the video publisher extracted from the video frame by an optical character recognition algorithm (OCR).

[0126] The audio text of the first sample video may include text in the audio obtained by performing audio recognition processing on the first sample video through an automatic speech recognition algorithm (ASR, Automatic Speech Recognition).

[0127] In a specific implementation, when the video text and audio text of the first sample video are serialized, they can be serialized through a BERT word segmenter to obtain a text sequence.

[0128] (3) The video frame sequence and the text sequence of the first sample video are spliced ​​according to a preset format to obtain a sample input sequence.

[0129] In a specific implementation, the video frame sequence and the text sequence can be separated by a preset identifier (such as <sep>) separated. In addition, in order to obtain the video features for classification later, after splicing the video frame sequence and the text sequence, a preset identifier can be inserted into the head of the sample input sequence. <cls>Of course, you can also insert other preset identifiers into the sample input sequence according to actual needs. For example, you can insert a preset identifier at the end of the sample input sequence. <eot>.

[0130] (4) Inputting the sample input sequence of the first sample video into the video encoding network for encoding, and obtaining the video features of the first sample video based on the features used for classification in the encoding results.

[0131] Specifically, the sample input sequence of the first sample video is input into the video coding network for coding processing to obtain a coding result, which can be the hidden state output by the last coding layer of the video coding network. The coding result includes feature information corresponding to each position in the sample input sequence. By obtaining a preset identifier <cls>The feature information of the corresponding position is processed, and after linear mapping processing is performed on the feature information, the video features of the first sample video can be obtained, wherein the parameters in the linear mapping processing can be determined by pre-training the video coding network.

[0132] In the above embodiment, by utilizing the video modality, audio modality and text modality of the video to obtain a sample input sequence input into the video coding network, the video features of the first sample video encoded by the video coding network contain multimodal information, which is beneficial to improving the ability of the trained video feature extraction network to understand multimodal information, and further helps to improve the accuracy of the extracted video features.

[0133] In an exemplary embodiment, before step S103, the embodiment of the present disclosure may further include a step of pre-training a video coding network. Specifically, the pre-training of the video coding network may include:

[0134] (1) Feature extraction is performed on a second sample video in a second sample video set based on an initial video coding network to obtain a second video feature of the second sample video.

[0135] The initial video encoding network may be a Transformer-based video feature extraction network.

[0136] In the disclosed embodiment, the training data used to pre-train the initial video coding network adopts the aforementioned second sample video set used to generate the multimodal knowledge graph.

[0137] In a specific implementation, when extracting features from the second sample video based on the initial video coding network, video modality information, audio modality information, and text modality information of the video may also be used. Specifically, extracting features from the second sample video in the second sample video set based on the initial video coding network may include:

[0138] Extracting a first preset number of video frames from the second sample video, and performing serialization processing on the first preset number of video frames to obtain a video frame sequence corresponding to the second sample video;

[0139] Determining a video text and an audio text of a second sample video, and performing serialization processing on the video text and the audio text of the second sample video to obtain a text sequence corresponding to the second sample video;

[0140] splicing the video frame sequence corresponding to the second sample video and the text sequence according to a preset format to obtain a sample input sequence corresponding to the second sample video;

[0141] The sample input sequence corresponding to the second sample video is input into the initial video encoding network for encoding, and the second video feature of the second sample video is obtained based on the features used for classification in the encoding result.

[0142] The specific implementation of each of the above steps can refer to the description of the video features of the first sample video in the above step S103, which will not be repeated here.

[0143] (2) Determine a video classification result of the second sample video based on the second video feature of the second sample video.

[0144] Specifically, the second video feature of the second sample video may be used as an input of the initial classification layer, and the final video classification result may be output through the initial classification layer.

[0145] (3) Determine a second video classification loss based on the video classification result of the second sample video and the category label of the second sample video.

[0146] Assume that the category label of the second sample video is t, and the second video feature of the second sample video is Z V , then the video classification result corresponding to the second sample video can be expressed as:

[0147]

[0148] Where s∈R T is a T-dimensional vector, s i ∈[0, 1] and T represents the number of categories of all category labels in the second sample video set; W2∈R T×k is the weight of the initial classification layer; softmax() is the normalization function.

[0149] in, C represents the preset identifier in the initial video encoder <cls>The corresponding hidden layer output; σ() represents the linear mapping function, W1 and b are the learnable parameters in the linear mapping function, and represent the bias coefficients of the weights respectively.

[0150] Then, the second video classification loss determined based on the video classification result of the second sample video and the category label of the second sample video can be expressed as follows using cross entropy loss:

[0151]

[0152] Among them, y i Represents the category label, when i=t is y i =1, when i≠1, y i =0.

[0153] (4) Adjusting the network parameters of the initial video coding network based on the second video classification loss until a third preset training end condition is reached to obtain the video coding network used in step S103.

[0154] Among them, the third preset training end condition can be that a preset iteration number threshold is reached, or the second video classification loss reaches a preset video classification loss threshold, or the difference between two adjacent second video classification losses reaches a preset difference threshold.

[0155] In a specific implementation, the network parameters of the initial video coding network are reversely adjusted based on the second video classification loss, and iterative training is continued based on the adjusted initial video coding network until the third preset training end condition is met to obtain a pre-trained video coding network. It should be noted that when the network parameters of the initial video coding network are reversely adjusted, the network parameters of the initial classification network that performs the classification task must also be adjusted.

[0156] In the above embodiment, by using the category label as the supervision signal and the category label as the classification target, a video coding network is trained based on the second sample video set, and when the pre-training uses the multimodal information of the second sample video as input, a video coding network with the ability to understand multimodal information can be obtained.

[0157] In step S105, features of the head entity in the target triplet corresponding to each of the first sample videos are extracted based on the label encoding network, and a first contrastive learning loss is determined based on the features of the head entity and the video features of the first sample video linked to the head entity in the target triplet.

[0158] Among them, the label encoding network can be an encoding network based on the BERT backbone network, and its output features come from the preset identifier <cls>The corresponding hidden layer output, that is, step S105 extracts the features of the head entity in the target triplet corresponding to the first sample video based on the label encoding network from the preset identifier <cls>The corresponding hidden layer output.

[0159] Specifically, since the head entity of the target triplet is consistent with the category label of the first sample video to which it is linked, the features obtained by extracting the head entity in the target triplet corresponding to the first sample video based on the label encoding network are the category label features corresponding to the first sample video, that is, the video features of the first sample video and the features of the corresponding head entity can be used as positive sample pairs in comparative learning.

[0160] The label encoding network in step S105 needs to be pre-trained. Based on this, in an exemplary embodiment, the embodiment of the present disclosure may further include the step of pre-training the label encoder. Specifically, the pre-training label encoding network may include:

[0161] (1) extracting features of a second sample video in a second sample video set based on a video coding network to obtain first video features of the second sample video;

[0162] The video coding network is pre-trained based on the aforementioned method of the embodiment of the present disclosure. When pre-training the label coding network, the pre-trained video coding network is utilized to extract video features of the second sample video. Furthermore, the training data used for the pre-trained label coding network is also the aforementioned second sample video set used to generate the multimodal knowledge graph.

[0163] Regarding the video features of the sample video extracted by the video coding network, please refer to the aforementioned relevant description of the embodiments of the present disclosure, which will not be repeated here.

[0164] (2) Based on the initial label encoding network, feature extraction is performed on the category label corresponding to the second sample video to obtain the sample label feature corresponding to the second sample video.

[0165] Among them, the sample label feature is based on the preset identifier in the input sequence <cls>The corresponding hidden layer output is obtained.

[0166] (3) The first video feature of the second sample video and the sample label feature of the second sample video are used as a positive sample pair, and a second contrastive learning loss is determined based on the positive sample pair.

[0167] The second contrastive learning loss may be a loss in a contrastive learning method based on CLIP (Contrastive Language–Image Pre-training, a large-scale image-text pre-training model based on contrastive learning), for example, the InfoNEC loss shown below:

[0168]

[0169] in, A first video feature representing a second sample video i; represents the sample label feature corresponding to the second sample video i; τ represents the learnable temperature parameter; B represents the number of small batch second sample videos during training.

[0170] (4) Fixing the network parameters of the video encoding network, and adjusting the network parameters of the initial label encoding network based on the second contrastive learning loss until the second preset training end condition is reached to obtain the label encoding network.

[0171] Among them, the second preset training end condition can be that a preset iteration number threshold is reached, or the second contrastive learning loss reaches a preset video classification loss threshold, or the difference between two adjacent second contrastive learning losses reaches a preset difference threshold.

[0172] In a specific implementation, the network parameters of the initial label encoding network are reversely adjusted based on the second contrastive learning loss, and iterative training is continued based on the adjusted initial label encoding network until the second preset training end condition is met to obtain a pre-trained label encoding network. It should be noted that when reversely adjusting the network parameters of the initial label encoding network, the network parameters of the video encoding network need to be fixed.

[0173] It can be understood that when determining the first contrastive learning loss based on the video features of the first sample video and the features of the corresponding head entity in the above step S105, reference can be made to the relevant description of step (3) in the above-mentioned initial label encoding network pre-training process, that is, the first contrastive learning loss can also be a loss based on the CLIP contrastive learning method, for example, it can be the InfoNEC loss.

[0174] The above embodiment determines the contrastive learning loss by using the video features of the second sample video extracted based on the video coding network and the label features of the category label corresponding to the second sample video extracted based on the initial label coding network, and then adjusts only the network parameters of the initial label coding network based on the contrastive learning loss, so as to map the video representation obtained by the video coding network to the same semantic space as the output features of the label coding network, thereby facilitating the subsequent realization of the purpose of mapping the video representation and the entity representation in the knowledge graph to the same semantic space.

[0175] In step S107, the features of the tail entity in the target triplet corresponding to each first sample video are extracted based on the label encoding network, and the vector representation of the target triplet corresponding to each first sample video is determined based on the features of the tail entity in the target triplet corresponding to each first sample video and the video features of each first sample video.

[0176] The video features of the first sample video are used as feature representations of the head entity in the target triplet corresponding to the first sample video.

[0177] Specifically, determining the vector representation of the target triplet corresponding to the first sample video based on the features of the tail entity in the target triplet corresponding to the first sample video and the video features of the first sample video may include:

[0178] Embedding the relationship in the target triplet corresponding to the first sample video based on the embedding network to obtain a relationship embedding representation;

[0179] Using the video features of the first sample video as the feature representation of the head entity in the target triplet corresponding to the first sample video;

[0180] Based on the feature representation of the head entity in the target triplet corresponding to the first sample video, the relation embedding representation and the feature of the tail entity in the target triplet, a vector representation of the target triplet corresponding to the first sample video is obtained.

[0181] Specifically, assuming the target triple is (h, r, t), the vector representation corresponding to the target triple can be as follows:

[0182] h=E video (video h )

[0183] t=E tag (text t )

[0184] r=T r

[0185] Among them, video h Denotes the first sample video linked to the head entity h, E video (video h ) represents the video features of the first sample video linked to the head entity h extracted based on the video coding network; text t Indicates the text corresponding to the tail entity t, E tag (text t ) represents the features of the tail entity in the target triplet extracted based on the label encoding network; T r It represents the relation embedding representation obtained by embedding the relation in the target triplet based on the embedding network.

[0186] The above implementation method maps the video representation and the entity representation in the knowledge graph into the same semantic space based on the video encoding network and the label encoding network, thereby integrating video information into the vector representation of the triplet.

[0187] In step S109 , the knowledge graph embedding loss is determined based on the vector representation of the target triplet corresponding to each first sample video.

[0188] Among them, the knowledge graph embedding loss can adopt the loss function corresponding to any knowledge graph embedding model.

[0189] In an exemplary embodiment, to ensure the stability of training, when determining the knowledge graph embedding loss based on the vector representation of the target triplet corresponding to each first sample video, negative sample information can be introduced using a negative sample sampling method. Based on this, the above step S109 may include the following when implemented:

[0190] The vector representation of the target triplet corresponding to the first sample video is taken as a positive sample;

[0191] Randomly sampling from the multimodal knowledge graph to determine a second preset number of sampling tail entities;

[0192] Based on the features of the second preset number of sampled tail entities, the features of the tail entities in the positive samples are respectively replaced to obtain a second preset number of negative samples corresponding to the positive samples;

[0193] Determine a knowledge graph embedding loss based on the positive sample and the second preset number of negative samples.

[0194] The second preset number can be set based on actual needs.

[0195] In the specific implementation, the knowledge graph embedding loss L can be determined based on the following formula: KG :

[0196]

[0197] Among them, (h,r,t′ i ) represents negative samples; n represents the number of negative samples, i.e., the second preset number mentioned above; γ represents a fixed margin, which can be set according to actual needs; σ() represents the activation function; d(·) represents the distance function defined by TransE: d(h+r,t)=‖h+rt‖ p , where the norm p can be set to 2. TransE is a classic model for knowledge graph embedding, and its basic idea is to make the sum of the head vector and the relation vector as close to the tail vector as possible.

[0198] The above implementation method determines the knowledge graph embedding loss by utilizing negative sample sampling, which is beneficial to improving the stability of the overall training.

[0199] In step S111, joint training is performed based on the first video classification loss, the first contrastive learning loss and the knowledge graph embedding loss. During the joint training process, the network parameters in the video encoding network and the label encoding network are adjusted until the first preset training end condition is reached to obtain a video feature extraction network.

[0200] The video feature extraction network includes a video encoding network based on the completion of training from step S101 to step S111.

[0201] Among them, the first preset training end condition can be that a preset iteration number threshold is reached, or the total loss reaches a preset video classification loss threshold, or the difference between two adjacent total losses reaches a preset difference threshold.

[0202] Specifically, the first video classification loss, the first contrastive learning loss and the knowledge graph embedding loss can be added to obtain a multi-task loss, and each network parameter can be adjusted based on the multi-task loss. Then, iterative training is continued based on the adjusted network parameters until the first preset training end condition is reached.

[0203] For example, when calculating the multi-task loss, a balance coefficient can be set for each of the first video classification loss, the first contrastive learning loss, and the knowledge graph embedding loss. Then, based on the balance coefficient, a weighted sum of the first video classification loss, the first contrastive learning loss, and the knowledge graph embedding loss is performed to obtain the multi-task loss. This can improve the final training effect. Specifically, the multi-task loss can be expressed as follows:

[0204]

[0205] in, is the knowledge graph embedding loss, λ1 represents the balance coefficient of the knowledge graph embedding loss; is the first contrastive learning loss, λ2 represents the balance coefficient of the first contrastive learning loss; is the first video classification loss, λ3 represents the balance coefficient of the first video classification loss; L is the multi-task loss.

[0206] In the embodiment of the present disclosure, by utilizing a multimodal knowledge graph to perform joint training of video classification tasks, contrastive learning tasks and knowledge graph embedding tasks, during the joint training, the first contrastive learning loss will further optimize the heterogeneous space formed by the video and the knowledge graph, the knowledge graph embedding loss implicitly integrates the common sense knowledge from the knowledge graph, and the first video classification loss retains the multimodal understanding ability of the video encoder. Therefore, the training method of the embodiment of the present disclosure can fully mine the multimodal information in the video and the common sense knowledge in the knowledge graph, so that the trained video feature extraction network can extract richer video features with common sense reasoning capabilities, thereby improving the accuracy of the extracted video features.

[0207] In an exemplary embodiment, after the video feature extraction network is trained, Figure 4 As shown, the method may further include:

[0208] In step S401, a video to be processed is obtained.

[0209] In step S403, feature extraction is performed on the video to be processed based on the video feature extraction network to obtain video features corresponding to the video to be processed.

[0210] In a specific implementation, the multimodal information of the video to be processed, that is, the video text, audio text and a preset number of video frames of the video to be processed can be utilized, and then based on the aforementioned relevant description of the embodiment of the present disclosure, the information of multiple modalities can be spliced ​​to obtain an input sequence, and then the input sequence is input into the video feature extraction network for feature extraction, thereby obtaining the video features of the video to be processed.

[0211] In the above implementation, since the multimodal knowledge graph is used to perform joint training of video classification tasks, comparative learning tasks and knowledge graph embedding tasks during the training process of the video feature extraction network, the multimodal information in the video and the common sense knowledge in the knowledge graph can be fully mined, and the video features extracted based on the video feature extraction network have the ability of common sense reasoning, and the accuracy of the video features is greatly improved.

[0212] In order to facilitate understanding of the technical solutions of the embodiments of the present disclosure, Figure 5 The joint training architecture diagram shown in FIG is used for specific explanation.

[0213] Before training, the dataset needs to be prepared. Table 1 shows examples of the dataset involved.

[0214] Table 1

[0215]

[0216] Among them, V represents video mode, A represents audio mode, and T represents text mode.

[0217] Each video in the Company-400M dataset is labeled with a tag, either annotated by the video author or generated by the model. The Company-400M dataset contains over 400,000 tag categories. The balance of the overall video data distribution has been fully considered during the video acquisition and tagging process.

[0218] CN-DBpedia sub is a knowledge graph dataset selected from the Chinese encyclopedia knowledge graph dataset to match short video platforms. The CN-DBpedia sub dataset contains 465,714 triples, 101,002 entities, and 4,987 relations.

[0219] The Company-5M dataset is a video-based multimodal knowledge graph dataset obtained by entity linking the Company-400M dataset and the CN-DBpedia sub-dataset. The number of head entities and tail entities contained in the Company-5M dataset are 84,838 and 191,474 respectively. The total number of entities and triples contained in the dataset is 248,324 and 832,577 respectively. The number of relationships is 5,150, and the number of linked videos is 5,714,531. The head entity of the Company-5M dataset triple will match at least one video, but some of the tail entities may not be linked to any video. For entities linked to videos, a text-modal entity may be linked to many videos, but a video can only be linked to one unique text entity.

[0220] The overall training process can be divided into the following three stages:

[0221] Phase 1: Pre-training the video encoding network E using the Company-400M dataset video During the pre-training process, for each video in the small batch, the video encoding network optimizes the classification target of the tag and is supervised by Tag Loss (the second video classification loss mentioned above).

[0222] The second stage: video encoding network E video After pre-training, add label encoder E tag Pre-training, pre-training label encoding network E tag The tag and video features are mapped into the same feature space through CLIP-based contrastive learning. tag It is designed based on the BERT backbone network, and the tag representation it generates comes from <cls>The hidden layer output corresponding to the token. Specifically, the video-text pair (V (i) ,T (i) )∈B for training, where B represents the small batch in the training process, V (i) T stands for video (i) Represents the tag label corresponding to the video. (i) With T (i) Input video encoding network E respectively video and label encoder E tag Generate the corresponding video representation and tag representation Combined with the aforementioned InfoNEC loss, optimization is performed. tag During pre-training, the parameters of the video encoder will be frozen.

[0223] The third stage: After the above pre-training process, the pre-trained video coding network E is obtained. video and label encoder E tag , in this stage, representation information from the knowledge graph is added, such as Figure 5 As shown, the joint training is performed on the Company-5M dataset. Each video in the Company-5M dataset is matched with a unique head entity h in the knowledge graph, and the knowledge graph triple (h, r, t) corresponding to this head entity is encoded using the method described in step S107 of the present embodiment.

[0224] The video and tail entities are respectively passed through the video encoding network E video With label encoding network E tag The encoding obtains the corresponding feature representation h and t, and the relationship is embedded based on the embedding network to obtain the corresponding embedding representation r. Among them, the input video is converted into a video frame sequence and a text sequence for splicing. There is a special <cls>token, such as Figure 5 As shown in the video coding network E video The output features are <cls>The hidden layer output corresponding to the token.

[0225] At the same time, based on the aforementioned steps S103 to S111 of the embodiment of the present disclosure, a multi-task loss can be determined, and the multi-task loss is used for optimization during joint training. Figure 5 All network parameters in the shown architecture can be used to obtain an optimized video coding network at the end of the final training. The optimized video coding network is the video feature extraction network ultimately required by the embodiment of the present disclosure.

[0226] Figure 6 1 is a structural block diagram of a video feature extraction network training device according to an exemplary embodiment. Figure 6 , the device comprises:

[0227] A target triplet determination unit 910 is configured to determine a target triplet to which a first sample video in a first sample video set is linked in a multimodal knowledge graph; each first sample video is linked to a head entity of a triplet in the multimodal knowledge graph, and the linked head entity represents a category label of the first sample video linked to the head entity;

[0228] A first classification loss determining unit 620 is configured to extract video features of each of the first sample videos based on a video coding network, and determine a first video classification loss based on the video features and the annotated category labels of each of the first sample videos;

[0229] A first contrastive learning loss determining unit 630 is configured to extract features of a head entity in a target triplet corresponding to each of the first sample videos based on a label encoding network, and determine a first contrastive learning loss based on features of the head entity in the target triplet and video features of the first sample video linked to the head entity in the target triplet;

[0230] The triplet vector representation unit 640 is configured to extract features of a tail entity in the target triplet corresponding to each of the first sample videos based on the label encoding network, and determine a vector representation of the target triplet corresponding to each of the first sample videos based on the features of the tail entity in the target triplet corresponding to each of the first sample videos and the video features of the first sample videos; wherein the video features of the first sample videos serve as feature representations of the head entity in the target triplet corresponding to the first sample videos;

[0231] a knowledge graph embedding loss determining unit 650, configured to determine a knowledge graph embedding loss based on a vector representation of a target triple corresponding to each of the first sample videos;

[0232] A training unit 660 is configured to perform joint training based on the first video classification loss, the first contrastive learning loss, and the knowledge graph embedding loss, and to adjust network parameters in the video encoding network and the label encoding network during the joint training process until a first preset training end condition is reached to obtain a video feature extraction network; the video feature extraction network includes the video encoding network at the end of training.

[0233] In an exemplary embodiment, the apparatus further comprises:

[0234] The first acquisition unit is configured to acquire a second sample video set and an original knowledge graph; the second sample video in the second sample video set corresponds to a category label;

[0235] a linking unit, configured to link each second sample video in the second sample video set with an entity in the original knowledge graph to obtain an entity linking result corresponding to each second sample video;

[0236] a screening unit, configured to, for each entity linking result corresponding to the second sample video, determine the entity linking result as a target entity linking result when the entity linking result indicates a link to a head entity and the head entity is consistent with a category label of the second sample video to which the head entity is linked;

[0237] A multimodal knowledge graph generation unit is used to generate the multimodal knowledge graph based on the target entity link result and the second sample video corresponding to the target entity link result.

[0238] In an exemplary embodiment, the apparatus further comprises:

[0239] a first feature extraction unit, configured to extract features of a second sample video in the second sample video set based on the video coding network to obtain a first video feature of the second sample video;

[0240] A second feature extraction unit is configured to extract features of the category label corresponding to the second sample video based on the initial label encoding network to obtain a sample label feature corresponding to the second sample video;

[0241] a second contrastive learning loss determining unit, configured to take the first video feature of the second sample video and the sample label feature of the second sample video as a positive sample pair, and determine a second contrastive learning loss based on the positive sample pair;

[0242] A first parameter adjustment unit is used to fix the network parameters of the video encoding network and adjust the network parameters of the initial label encoding network based on the second contrastive learning loss until a second preset training end condition is met to obtain the label encoding network.

[0243] In an exemplary embodiment, the apparatus further comprises:

[0244] a third feature extraction unit, configured to extract features of the second sample video in the second sample video set based on the initial video coding network to obtain second video features of the second sample video;

[0245] a classification result determining unit, configured to determine a video classification result of the second sample video based on a second video feature of the second sample video;

[0246] A second classification loss determining unit, configured to determine a second video classification loss based on the video classification result of the second sample video and the category label of the second sample video;

[0247] A second parameter adjustment unit is used to adjust the network parameters of the initial video coding network based on the second video classification loss until a third preset training end condition is reached to obtain the video coding network.

[0248] In an exemplary embodiment, the triple vector representation unit 640 includes:

[0249] a relation embedding unit, configured to embed the relation in the target triple corresponding to the first sample video based on an embedding network to obtain a relation embedding representation;

[0250] a head entity feature determining unit, configured to use the video feature of the first sample video as a feature representation of the head entity in the target triplet corresponding to the first sample video;

[0251] The triplet vector representation subunit is used to obtain the vector representation of the target triplet corresponding to the first sample video based on the feature representation of the head entity in the target triplet corresponding to the first sample video, the relationship embedding representation and the feature of the tail entity in the target triplet.

[0252] In an exemplary embodiment, the knowledge graph embedding loss determination unit 650 includes:

[0253] a positive sample determining unit, configured to take the vector representation of the target triplet corresponding to the first sample video as a positive sample;

[0254] A random sampling unit, configured to randomly sample and determine a second preset number of sampling tail entities from the multimodal knowledge graph;

[0255] a negative sample determining unit, configured to replace the features of the tail entities in the positive sample based on the features of the second preset number of sampled tail entities, to obtain a second preset number of negative samples corresponding to the positive sample;

[0256] The knowledge graph embedding loss determination subunit is used to determine the knowledge graph embedding loss based on the positive sample and the second preset number of negative samples.

[0257] In an exemplary embodiment, the first classification loss determining unit 620 includes:

[0258] a video frame serialization unit, configured to extract a first preset number of video frames from the first sample video, and perform serialization processing on the first preset number of video frames to obtain a video frame sequence;

[0259] a text serialization unit, configured to determine a video text and an audio text of the first sample video, and perform serialization processing on the video text and the audio text of the first sample video to obtain a text sequence;

[0260] an input sequence determining unit, configured to splice the video frame sequence and the text sequence according to a preset format to obtain a sample input sequence;

[0261] The encoding unit is used to input the sample input sequence into a video encoding network for encoding, and classify the features used for classification in the encoding result to obtain the video features of the first sample video.

[0262] Regarding the apparatus in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0263] In an exemplary embodiment, an electronic device is also provided, including a processor; a memory for storing processor-executable instructions; wherein, when the processor is configured to execute the instructions stored in the memory, it implements the video feature extraction network training method provided in any of the above embodiments.

[0264] The electronic device may be a terminal, a server or a similar computing device. For example, the electronic device is a server. Figure 7 is a block diagram of an electronic device for video feature extraction network training according to an exemplary embodiment. Figure 7 As shown, the server 700 may have relatively large differences due to different configurations or performances, and may include one or more central processing units (CPUs) 710 (the processor 710 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA), a memory 730 for storing data, and one or more storage media 720 (such as one or more mass storage devices) for storing application programs 723 or data 722. Among them, the memory 730 and the storage medium 720 can be temporary storage or permanent storage. The program stored in the storage medium 720 may include one or more modules, each module may include a series of instruction operations on the server. Furthermore, the central processing unit 710 can be configured to communicate with the storage medium 720 to execute a series of instruction operations in the storage medium 720 on the server 700. The server 700 may also include one or more power supplies 760, one or more wired or wireless network interfaces 750, one or more input and output interfaces 740, and / or one or more operating systems 721, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, etc.

[0265] The input / output interface 740 can be used to receive or send data via a network. Specific examples of the aforementioned network may include a wireless network provided by the communication provider of the server 700. In one embodiment, the input / output interface 740 includes a network interface controller (NIC), which can be connected to other network devices via a base station to communicate with the Internet. In one embodiment, the input / output interface 740 can be a radio frequency (RF) module for wirelessly communicating with the Internet.

[0266] It can be understood by those skilled in the art that Figure 7 The structure shown is only for illustration and does not limit the structure of the above electronic device. Figure 7 More or fewer components than shown, or with Figure 7 Different configurations shown.

[0267] In an exemplary embodiment, a computer-readable storage medium including instructions is further provided, such as a memory 730 including instructions. The instructions can be executed by the processor 710 of the apparatus 700 to perform the above-mentioned method of the embodiment of the present disclosure. Alternatively, the computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0268] In an exemplary embodiment, a computer program product is also provided, including a computer program, which, when executed by a processor, implements the video feature extraction network training method provided in any of the above embodiments.

[0269] Other embodiments of the present disclosure will readily occur to those skilled in the art after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art not disclosed herein. The description and examples are to be considered as exemplary only, with the true scope and spirit of the present disclosure being indicated by the following claims.

[0270] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes can be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.< / cls> < / cls> < / cls> < / cls> < / cls> < / cls> < / cls> < / cls> < / eot> < / cls> < / sep>

Claims

1. A video feature extraction network training method, characterized in that: include: Determine a target triple linked to a first sample video in a first sample video set in a multimodal knowledge graph; Each of the first sample videos is linked to a head entity of a triple in the multimodal knowledge graph, and the linked head entity represents a category label of the first sample video linked to the head entity; Extracting video features of each of the first sample videos based on a video coding network, and determining a first video classification loss based on the video features and the annotated category labels of each of the first sample videos; extracting features of a head entity in a target triplet corresponding to each of the first sample videos based on a label encoding network, and determining a first contrastive learning loss based on features of the head entity in the target triplet and video features of the first sample video linked to the head entity in the target triplet; Extracting features of the tail entity in the target triplet corresponding to each of the first sample videos based on the label encoding network; Embedding the relationship in the target triple corresponding to the first sample video based on the embedding network to obtain a relationship embedding representation; using the video features of the first sample video as the feature representation of the head entity in the target triple corresponding to the first sample video; and obtaining a vector representation of the target triple corresponding to the first sample video based on the feature representation of the head entity in the target triple corresponding to the first sample video, the relationship embedding representation, and the features of the tail entity in the target triple; Taking the vector representation of the target triplet corresponding to the first sample video as a positive sample; Randomly sampling from the multimodal knowledge graph to determine a second preset number of sampled tail entities; replacing the features of the tail entities in the positive samples based on the features of the second preset number of sampled tail entities, to obtain a second preset number of negative samples corresponding to the positive samples; determining the knowledge graph embedding loss based on the positive samples and the second preset number of negative samples; Joint training is performed based on the first video classification loss, the first contrastive learning loss, and the knowledge graph embedding loss. During the joint training process, network parameters in the video encoding network and the label encoding network are adjusted until a first preset training end condition is reached to obtain a video feature extraction network; the video feature extraction network includes the video encoding network at the end of training.

2. The method according to claim 1, characterized in that The method further comprises: Obtain a second sample video set and an original knowledge graph; the second sample video in the second sample video set corresponds to a category label; Linking each second sample video in the second sample video set with an entity in the original knowledge graph to obtain an entity linking result corresponding to each second sample video; For each entity linking result corresponding to the second sample video, when the entity linking result indicates a link to a head entity and the head entity is consistent with the category label of the second sample video to which the head entity is linked, determining the entity linking result as a target entity linking result; The multimodal knowledge graph is generated based on the target entity link result and the second sample video corresponding to the target entity link result.

3. The method according to claim 2, characterized in that The method further comprises: performing feature extraction on a second sample video in the second sample video set based on the video encoding network to obtain a first video feature of the second sample video; Extracting features of the category label corresponding to the second sample video based on the initial label encoding network to obtain sample label features corresponding to the second sample video; taking the first video feature of the second sample video and the sample label feature of the second sample video as a positive sample pair, and determining a second contrastive learning loss based on the positive sample pair; The network parameters of the video encoding network are fixed, and the network parameters of the initial label encoding network are adjusted based on the second contrastive learning loss until a second preset training end condition is met to obtain the label encoding network.

4. The method according to claim 2, characterized in that The method further comprises: performing feature extraction on a second sample video in the second sample video set based on the initial video coding network, obtaining a second video feature of the second sample video; determining a video classification result of the second sample video based on the second video feature of the second sample video; determining a second video classification loss based on the video classification result of the second sample video and the category label of the second sample video; The network parameters of the initial video coding network are adjusted based on the second video classification loss until a third preset training end condition is reached to obtain the video coding network.

5. The method according to any one of claims 1 to 4, characterized in that The extracting the video features of the first sample video based on the video coding network includes: Extracting a first preset number of video frames from the first sample video, and performing serialization processing on the first preset number of video frames to obtain a video frame sequence; Determining a video text and an audio text of the first sample video, and performing serialization processing on the video text and the audio text of the first sample video to obtain a text sequence; splicing the video frame sequence and the text sequence according to a preset format to obtain a sample input sequence; The sample input sequence is input into a video encoding network for encoding, and video features of the first sample video are obtained based on features used for classification in the encoding result.

6. A video feature extraction network training device, characterized in that: include: a target triplet determination unit, configured to determine a target triplet linked to the first sample video in the first sample video set in the multimodal knowledge graph; Each of the first sample videos is linked to a head entity of a triple in the multimodal knowledge graph, and the head entity represents a category label of the first sample video linked to the head entity; a classification loss determining unit, configured to extract video features of each of the first sample videos based on a video coding network, and determine a first video classification loss based on the video features and the annotated category labels of each of the first sample videos; a contrastive learning loss determining unit, configured to extract features of a head entity in a target triplet corresponding to each of the first sample videos based on a label encoding network, and determine a first contrastive learning loss based on features of the head entity in the target triplet and video features of the first sample video linked to the head entity in the target triplet; A triplet vector representation unit, configured to extract features of a tail entity in a target triplet corresponding to each of the first sample videos based on the label encoding network; Embedding the relationship in the target triple corresponding to the first sample video based on the embedding network to obtain a relationship embedding representation; using the video features of the first sample video as the feature representation of the head entity in the target triple corresponding to the first sample video; and obtaining a vector representation of the target triple corresponding to the first sample video based on the feature representation of the head entity in the target triple corresponding to the first sample video, the relationship embedding representation, and the features of the tail entity in the target triple; a knowledge graph embedding loss determination unit, configured to take the vector representation of the target triplet corresponding to the first sample video as a positive sample; Randomly sampling from the multimodal knowledge graph to determine a second preset number of sampled tail entities; replacing the features of the tail entities in the positive samples based on the features of the second preset number of sampled tail entities, to obtain a second preset number of negative samples corresponding to the positive samples; determining the knowledge graph embedding loss based on the positive samples and the second preset number of negative samples; A training unit is used to perform joint training based on the first video classification loss, the first contrastive learning loss and the knowledge graph embedding loss, and adjust network parameters in the video encoding network and the label encoding network during the joint training process until a first preset training end condition is reached to obtain a video feature extraction network; the video feature extraction network includes the video encoding network at the end of training.

7. An electronic device, characterized in that: include: processor; a memory for storing instructions executable by the processor; The processor is configured to execute the instructions to implement the video feature extraction network training method according to any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that When the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to perform the video feature extraction network training method according to any one of claims 1 to 5.

Images