Video annotation method, device, computer equipment and computer program product

Abstract

The present application relates to a video annotation method, apparatus, computer equipment, storage medium and computer program product. The method comprises: for a training video in a training video set, obtaining a training triplet corresponding to the training video; the training triplet comprises a video identifier, a training label and a training relationship; respectively obtaining semantic expression features corresponding to the video identifier, the first training relationship and the training label; inputting the semantic expression features corresponding to the video identifier and the content extraction features corresponding to the video identifier into a fusion model to obtain training fusion features; based on the training fusion features and the semantic expression features corresponding to the first training relationship, determining the predicted label features; based on the difference between the semantic expression features corresponding to the training label and the predicted label features, adjusting each semantic expression feature and the model parameters of the fusion model to obtain each target semantic expression feature and the target fusion model. The use of this method can improve the label diversity during video annotation.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a video annotation method, apparatus, computer equipment, storage medium and computer program product. Background Art

[0002] With the development of computer technology, video annotation technology has emerged. Video annotation refers to the process of adding labels to videos.

[0003] In traditional technology, video annotation is usually based on the classification of video content, such as classifying the key frames of a video, or classifying the background music of a video. However, this method cannot label tags that are not directly related to the content of the video, resulting in a single labeled tag. Summary of the invention

[0004] Based on this, it is necessary to provide a video annotation method, device, computer equipment, computer-readable storage medium and computer program product that can improve label diversity in response to the above technical problems.

[0005] In one aspect, the present application provides a video annotation method. The method comprises:

[0006] Determine a training video set, and for a training video in the training video set, obtain a first training triplet corresponding to the training video; the first training triplet includes a video identifier and a training label corresponding to the training video, and a first training relationship between the training video and the training label;

[0007] Respectively obtaining semantic expression features corresponding to the video identifier, the first training relationship, and the training label;

[0008] Inputting the semantic expression feature corresponding to the video identifier and the content extraction feature corresponding to the video identifier into a fusion model to obtain a training fusion feature; the content extraction feature is obtained by extracting the feature of the associated content corresponding to the video identifier;

[0009] Determining a first predicted label feature corresponding to the first training relationship based on the training fusion feature and the semantic expression feature corresponding to the first training relationship;

[0010] Based on a first difference between the semantic expression feature corresponding to the training label and the first predicted label feature, adjusting each semantic expression feature and a model parameter of the fusion model to obtain each adjusted target semantic expression feature and an adjusted target fusion model;

[0011] Each of the target semantic expression features and the target fusion model are used to cooperate in labeling the training video.

[0012] On the other hand, the present application also provides a video annotation device. The device includes:

[0013] A training sample acquisition module, used to determine a training video set, and for a training video in the training video set, obtain a first training triplet corresponding to the training video; the first training triplet includes a video identifier and a training label corresponding to the training video, and a first training relationship between the training video and the training label;

[0014] A semantic expression feature acquisition module, used to respectively acquire the semantic expression features corresponding to the video identifier, the first training relationship, and the training label;

[0015] A fusion module is used to input the semantic expression feature corresponding to the video identifier and the content extraction feature corresponding to the video identifier into a fusion model to obtain a training fusion feature; the content extraction feature is obtained by extracting the feature of the associated content corresponding to the video identifier;

[0016] A label prediction module, used to determine a first predicted label feature corresponding to the first training relationship based on the training fusion feature and the semantic expression feature corresponding to the first training relationship;

[0017] A parameter adjustment module is used to adjust each semantic expression feature and the model parameters of the fusion model based on the first difference between the semantic expression feature corresponding to the training label and the first predicted label feature, so as to obtain the adjusted target semantic expression features and the adjusted target fusion model; each target semantic expression feature and the target fusion model are used to cooperate in realizing the annotation of the training video.

[0018] On the other hand, the present application further provides a computer device, wherein the computer device comprises a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of the above-mentioned video annotation method are implemented.

[0019] On the other hand, the present application further provides a computer-readable storage medium having a computer program stored thereon, and the computer program implements the steps of the above-mentioned video tagging method when executed by a processor.

[0020] On the other hand, the present application also provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, the steps of the above-mentioned video annotation method are implemented.

[0021] The above-mentioned video annotation method, device, computer equipment, storage medium and computer program product, after determining the training video set, obtains the first training triplet corresponding to the training video for each training video in the training video set, obtains the semantic expression features corresponding to the video identifier, the first training relationship and the training label in the first target triplet respectively, inputs the semantic expression features corresponding to the video identifier and the content extraction features corresponding to the video identifier into the fusion model to obtain the training fusion features, and determines the first predicted label features corresponding to the first training relationship based on the training fusion features and the semantic expression features corresponding to the first training relationship. Each of the first target triples and the target fusion model are used to cooperate to implement the annotation of the training video. Since the associated content corresponding to the video identifier is fused during the adjustment process, each target semantic expression feature can express the semantic relationship between the training video and the label, and then based on each target semantic expression feature, when annotating, labels related to semantics can be annotated, thereby improving the diversity of video labels.

[0022] On the other hand, the present application also provides a video annotation device. The device includes:

[0023] A fusion feature acquisition module is used to acquire a target fusion feature corresponding to the target video to be annotated; the target fusion feature is obtained by inputting a target semantic expression feature corresponding to the video identifier of the target video and a content extraction feature corresponding to the video identifier into a target fusion model;

[0024] A label prediction module, used to obtain a target prediction label feature corresponding to the first target relationship based on the target fusion feature and the target semantic expression feature corresponding to the first target relationship;

[0025] A feature matching module is used to match the target semantic expression features corresponding to each training label with the target prediction label features, and determine the training label corresponding to the successfully matched target semantic expression feature as the first annotation label corresponding to the target video;

[0026] Among them, the target fusion model and each target semantic expression feature are obtained by adjusting the fusion model to be adjusted and the each semantic expression feature to be adjusted based on the first difference corresponding to the first training triplet; the first training triplet corresponds to a training video in the training video set to which the target video belongs, including a video identifier and a training label corresponding to the training video, and a first training relationship between the training video and the training label; the first difference is the difference between the semantic expression feature corresponding to the training label and the first predicted label feature; the first predicted label feature corresponds to the first training relationship, and is determined based on the training fusion feature and the semantic expression feature corresponding to the first training relationship; the training fusion feature is obtained by inputting the semantic expression feature corresponding to the video identifier and the content extraction feature corresponding to the video identifier into the fusion model; the content extraction feature is obtained by extracting the feature of the associated content corresponding to the video identifier.

[0027] On the other hand, the present application also provides a video annotation method. The method comprises:

[0028] Obtaining a target fusion feature corresponding to the target video to be annotated; the target fusion feature is obtained by inputting a target semantic expression feature corresponding to the video identifier of the target video and a content extraction feature corresponding to the video identifier into a target fusion model;

[0029] Based on the target fusion feature and the target semantic expression feature corresponding to the first target relationship, obtaining a target prediction label feature corresponding to the first target relationship;

[0030] Matching the target semantic expression features corresponding to each training label with the target prediction label features respectively, and determining the training label corresponding to the successfully matched target semantic expression feature as the first annotation label corresponding to the target video;

[0031] The target fusion model and each target semantic expression feature are obtained by adjusting the fusion model to be adjusted and each semantic expression feature to be adjusted based on the first difference corresponding to the first training triplet; the first training triplet corresponds to a training video in the training video set to which the target video belongs, including a video identifier and a training label corresponding to the training video, and a first training relationship between the training video and the training label; the first difference is the difference between the semantic expression feature corresponding to the training label and the first predicted label feature; the first predicted label feature corresponds to the first training relationship, and is determined based on the training fusion feature and the semantic expression feature corresponding to the first training relationship; the training fusion feature is obtained by inputting the semantic expression feature corresponding to the video identifier and the content extraction feature corresponding to the video identifier into the fusion model; the content extraction feature is obtained by extracting the feature of the associated content corresponding to the video identifier.

[0032] On the other hand, the present application further provides a computer device, wherein the computer device comprises a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of the above-mentioned video annotation method are implemented.

[0033] On the other hand, the present application further provides a computer-readable storage medium having a computer program stored thereon, and the computer program implements the steps of the above-mentioned video tagging method when executed by a processor.

[0034] On the other hand, the present application also provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, the steps of the above-mentioned video annotation method are implemented.

[0035] The above-mentioned video annotation method, device, computer equipment, storage medium and computer program product, because the target fusion model and each target semantic expression feature are obtained by adjusting the fusion model to be adjusted and each semantic expression feature to be adjusted based on the first difference corresponding to the first training triplet; the first training triplet corresponds to the training video in the training video set to which the target video belongs, including the video identifier and training label corresponding to the training video, and the first training relationship between the training video and the training label; the first difference is the difference between the semantic expression feature corresponding to the training label and the first predicted label feature; the first predicted label feature corresponds to the first training relationship and is determined based on the training fusion feature and the semantic expression feature corresponding to the first training relationship; the training fusion feature is obtained by inputting the semantic expression feature corresponding to the video identifier and the content extraction feature corresponding to the video identifier into the fusion model; the content extraction feature is obtained by extracting the associated content corresponding to the video identifier, so that the associated content corresponding to the video identifier is integrated into each target semantic expression feature obtained, and the semantic relationship between the training video and the label can be expressed accurately, and then after obtaining the target predicted label feature, by matching with the target semantic expression feature corresponding to each training label, accurate semantic labels can be obtained, thereby improving the diversity of video labels. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 A diagram of an application environment of a video annotation method in an embodiment;

[0037] Figure 2 A schematic diagram of a process flow of a video annotation method in an embodiment;

[0038] Figure 3 A schematic diagram of labels annotated on a video in one embodiment;

[0039] Figure 4 A schematic diagram of a specific fusion process in an embodiment;

[0040] Figure 5 is a flow chart of a video annotation method in another embodiment;

[0041] Figure 6 A schematic diagram of a partial flow chart of a video annotation method in one embodiment;

[0042] Figure 7 is a flow chart of a video annotation method in another embodiment;

[0043] Figure 8 A schematic diagram of a process of a video annotation method in one embodiment;

[0044] Fig. 9 is a schematic diagram of test results performed on a third data set in one embodiment;

[0045] Fig.10 is a schematic diagram of experimental data under different knowledge embedding methods in one embodiment;

[0046] Fig.11 A schematic diagram of an expandable visualization experiment result in one embodiment;

[0047] Fig.12 is a structural block diagram of a video annotation device in one embodiment;

[0048] Fig.13 is a structural block diagram of a video annotation device in one embodiment;

[0049] Fig.14 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0050] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0051] The video annotation method provided in the embodiment of the present application involves artificial intelligence computer perspective, machine learning and other technologies, wherein:

[0052] Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that machines have the functions of perception, reasoning and decision-making.

[0053] Artificial intelligence technology is a comprehensive discipline that covers a wide range of fields, including both hardware-level and software-level technologies. The basic technologies of artificial intelligence generally include sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics and other technologies. Artificial intelligence software technology mainly includes computer vision technology, speech processing technology, natural language processing technology, as well as machine learning / deep learning, autonomous driving, smart transportation and other major directions.

[0054] Computer Vision (CV) is a science that studies how to make machines "see". To put it more specifically, it refers to the use of cameras and computers to replace human eyes to identify and measure targets, and further process graphics so that the computer processes the images into images that are more suitable for human observation or transmission to instruments for detection. As a scientific discipline, computer vision studies related theories and technologies, and attempts to establish an artificial intelligence system that can obtain information from images or multi-dimensional data. Computer vision technology usually includes image processing, image recognition, image semantic understanding, image retrieval, OCR, video processing, video semantic understanding, video content recognition, three-dimensional object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous positioning and mapping, automatic driving, smart transportation and other technologies, as well as common biometric recognition technologies such as face recognition and fingerprint recognition.

[0055] Machine Learning (ML) is a multi-disciplinary subject that involves probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory and other disciplines. It specializes in studying how computers simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications are spread across all areas of artificial intelligence. Machine learning and deep learning usually include artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and self-learning.

[0056] With the research and advancement of artificial intelligence technology, artificial intelligence technology has been studied and applied in many fields, such as common smart homes, smart wearable devices, virtual assistants, smart speakers, smart marketing, unmanned driving, automatic driving, drones, robots, smart medical care, smart customer service, Internet of Vehicles, automatic driving, smart transportation, etc. I believe that with the development of technology, artificial intelligence technology will be applied in more fields and play an increasingly important role.

[0057] The computer vision, machine learning and other technologies involved in this application are specifically described through the following embodiments:

[0058] The video annotation method provided in the embodiment of the present application can be applied to Figure 1In the application environment shown. Among them, the terminal 102 communicates with the server 104 through the network. The data storage system can store the data that the server 104 needs to process. The data storage system can be integrated on the server 104, or it can be placed on the cloud or other network servers. Among them, the terminal 102 can be but is not limited to various desktop computers, laptops, smart phones, tablets, Internet of Things devices and portable wearable devices. The Internet of Things devices can be smart speakers, smart TVs, smart air conditioners, smart car-mounted devices, etc. Portable wearable devices can be smart watches, smart bracelets, head-mounted devices, etc. The terminal can be provided with a client for video playback. The server 104 can be implemented with an independent server or a server cluster or cloud server consisting of multiple servers.

[0059] Specifically, the server can determine a training video set, and for a training video in the video set, obtain a first training triplet corresponding to the training video, the first training triplet includes a video identifier and a training label corresponding to the training video, and a first training relationship between the training video and the training label. The server further obtains semantic expression features corresponding to the video identifier, the first training relationship, and the training label, respectively, and inputs the semantic expression features corresponding to the video identifier and the content extraction features corresponding to the video identifier into a fusion model to obtain training fusion features. The content extraction features are obtained by extracting features of associated content corresponding to the video identifier. The server can then determine a first predicted label feature corresponding to the first training relationship based on the training fusion features and the semantic expression features corresponding to the first training relationship. Finally, based on the first difference between the semantic expression features corresponding to the training label and the first predicted label features, the server adjusts each semantic expression feature and the model parameters of the fusion model to obtain each adjusted target semantic expression feature and the adjusted target fusion model.

[0060] The server can annotate the training videos in the training video set using each target semantic expression feature and the target fusion model. Specifically, taking one of the training videos in the training video set as the target video to be annotated as an example, the server can obtain the target fusion feature corresponding to the target video to be annotated, and obtain the target prediction label feature corresponding to the first target relationship based on the target fusion feature and the target semantic expression feature corresponding to the first target relationship, and respectively match the target semantic expression features corresponding to each training label with the target prediction label feature, and determine the training label corresponding to the successfully matched target semantic expression feature as the annotation label corresponding to the target video, wherein the target fusion feature is obtained by inputting the target semantic expression feature corresponding to the video identifier of the target video and the content extraction feature corresponding to the video identifier into the target fusion model.

[0061] The server can further use the annotation tags obtained by annotation to implement video annotation or user promotion. For details, please refer to the description in the embodiments below.

[0062] In one embodiment, Figure 2 As shown, a video annotation method is provided, which is applied to Figure 1 The server in the example is used to illustrate the following steps:

[0063] Step 202, determine a training video set, and for a training video in the training video set, obtain a first training triplet corresponding to the training video; the first training triplet includes a video identifier corresponding to the training video, a training label, and a first training relationship between the training video and the training label.

[0064] Among them, the training video set refers to a set composed of multiple training videos, and the training videos in the training video set can be collected from the video platform. Each training video in the training video set has multiple different attributes. These attributes can be generated by artificial intelligence or manually annotated. These attributes can be, for example, "content includes", "uploaded from", "contains characters", "original attributes", etc. There are corresponding values ​​for each attribute. After collecting the training videos, the server can convert the attribute value pairs of these training videos into relationships and video tags, and use the video identifiers, relationships and video tags as knowledge nodes of the knowledge graph to construct the knowledge graph. In one embodiment, the attribute value pairs of each training video can be converted through a pre-defined mapping relationship to generate relationships and video tags, that is, the attributes are used as relationships, and the values ​​corresponding to the attributes are used as video tags. For example, assuming that a training video has an attribute value pair (there are characters: male), then "there are characters" can be converted into a relationship, and the gender "male" can be converted into a corresponding video tag. It can be understood that for the relationship "there are characters", the corresponding candidate video tags can also be "female" and "neutral". For another example, suppose a training video has an attribute value pair (original mark: original), the attribute value can represent the originality of the video, the "original mark" can be converted into a relationship, and the "original" can be converted into a corresponding video label. In the knowledge graph established above, for each training video, a first triple such as (video identifier, relationship, video label) can be obtained, and these first triples can be used as training samples to participate in the training. The first triples participating in the training can be called the first training triples, the relationship corresponding to the first training triples is the first training relationship, and the video label corresponding to the first training triples is the training label. It can be understood that the first triple here is a positive training sample. In an embodiment of the present application, a predefined knowledge embedding method (knowledge embeddings, KGE) is used in the training process, and the goal of the training is to make the labels obtained based on the video identifier and relationship prediction close to the correct labels in the first triples.

[0065] Specifically, the server first determines the videos involved in the training, and forms these videos into a training video set. For each video in the training video set, the server can obtain the corresponding first triple from the pre-established knowledge graph, and use the obtained first triple as the first training triple. It can be understood that for each training video, one or more first triples can be obtained, thereby obtaining one or more first training triples. The multiple mentioned in this application refers to at least two.

[0066] Step 204: Obtain semantic expression features corresponding to the video identifier, the first training relationship, and the training label, respectively.

[0067] The semantic expression feature refers to a feature vector in the semantic space. It is understandable that the semantic expression features of the video identifier, the first training relationship, and the training label can be located in the same semantic space or in different semantic spaces.

[0068] Specifically, at the beginning of training, the server can generate initialized semantic expression features, that is, generate initialization vectors of preset dimensions for each semantic node in the knowledge graph, so that for each first training triple, the server can obtain the initialization vectors of the video identifier, the first training relationship, and the training label in the first training triple, as the semantic expression features corresponding to the video identifier, the first training relationship, and the training label. In one embodiment, the generated initialization vectors can also be normalized to reduce the amount of calculation and improve training efficiency.

[0069] During the training process, the initialized semantic expression features are iteratively adjusted, that is, in each adjustment process, the semantic expression features obtained in the previous adjustment are obtained as the initial value of the adjustment. When the convergence conditions are met, the adjusted target semantic expression features corresponding to the video identifier, the adjusted target semantic expression features corresponding to the training label, and the adjusted target semantic expression features corresponding to the first training relationship are obtained.

[0070] Step 206, input the semantic expression features corresponding to the video identifier and the content extraction features corresponding to the video identifier into the fusion model to obtain training fusion features; the content extraction features are obtained by extracting features of the associated content corresponding to the video identifier.

[0071] Among them, the content extraction feature is obtained by extracting the features of the associated content corresponding to the video identifier. The associated content corresponding to the video identifier can be at least one of the video frame corresponding to the video identifier, the background music corresponding to the video identifier, or the associated text corresponding to the video identifier. Different associated content belongs to different modalities. The modality here refers to the source or form of information. For example, a set of video frames belongs to the visual modality (Visual, hereinafter referred to as V), the background music belongs to the auditory modality (Audio, hereinafter referred to as A), and the associated text belongs to the text modality (Texts, hereinafter referred to as T). Different associated content extractions obtain different content extraction features. The fusion model refers to a machine learning model used to fuse features.

[0072] Specifically, the server can obtain at least one content extraction feature corresponding to the video identifier, input the semantic expression feature corresponding to the video identifier and the content extraction feature corresponding to the video identifier into a fusion model, and fuse the semantic expression feature corresponding to the video identifier and at least one content extraction feature through the fusion model to obtain a training fusion feature.

[0073] It can be understood that at the beginning of training, the model parameters of the fusion model are initialized, and the model parameters of the fusion model are continuously iteratively adjusted during the training process, that is, each time the model parameters obtained by the previous adjustment are used for the current fusion processing, when the convergence conditions are met, the adjusted target fusion model is obtained.

[0074] Step 208: Determine a first predicted label feature corresponding to the first training relationship based on the training fusion feature and the semantic expression feature corresponding to the first training relationship.

[0075] The first predicted label feature refers to a semantic expression feature predicted based on the video identifier and the first training relationship, and is used to express the semantics of the predicted label. For the same video identifier, different first predicted label features can be predicted under different training relationships. Therefore, the first predicted label feature corresponds to the first training relationship.

[0076] Specifically, in the embodiment of the present application, each target semantic expression feature is finally obtained, which is equivalent to embedding the knowledge of the knowledge graph into the vector space for expression. In the present application, when implementing knowledge embedding, a knowledge embedding method based on vector translation distance is adopted, for example, any one of TransE, TransH, TransR and TransD. In different embodiments, the server can specifically predefine different knowledge embedding methods to embed knowledge.

[0077] In one embodiment, when the predefined knowledge embedding method is TransE, the server may add the training fusion feature and the semantic expression feature corresponding to the first training relationship to obtain a first predicted label feature corresponding to the first training relationship.

[0078] In another embodiment, when the predefined knowledge embedding method is TransH, the server first projects the training fusion features onto the hyperplane corresponding to the first training relationship, adds the mapped feature vector and the semantic expression features corresponding to the first training relationship, and obtains the first predicted label feature corresponding to the first training relationship.

[0079] Step 210, based on the first difference between the semantic expression feature corresponding to the training label and the first predicted label feature, adjust each semantic expression feature and the model parameters of the fusion model to obtain adjusted target semantic expression features and an adjusted target fusion model; each target semantic expression feature and the target fusion model are used to cooperate in annotating the training video.

[0080] The first difference is used to characterize the difference between the predicted label and the training label. The larger the first difference, the less accurate the predicted label. On the contrary, the smaller the first difference, the more accurate the predicted label.

[0081] Specifically, the way in which the server calculates the first difference is determined by a predefined knowledge embedding method, and different difference calculation methods are used in different knowledge embedding methods. In one embodiment, when the predefined knowledge embedding method is TransE, the server can calculate the L2 distance between the semantic expression features corresponding to the training labels and the first predicted label features to obtain the first difference. In another embodiment, when the predefined knowledge embedding method is TransH, the first predicted label feature is that the server can project the semantic expression features corresponding to the training labels onto the hyperplane corresponding to the first training relationship, calculate the L2 distance between the mapped feature vector and the first predicted label feature, and obtain the first difference.

[0082] After calculating the first difference, the server can calculate the target loss value based on the first difference, and iteratively adjust the semantic expression features corresponding to the first training triple and the model parameters of the fusion model based on the target loss value, that is, return to step 204 to repeat the steps of adjusting the semantic expression features corresponding to the first training triple and the model parameters of the fusion model based on the first difference between the semantic expression features corresponding to the training label and the first predicted label features in step 204 to step 210 until the convergence condition is met, and the adjusted target semantic expression features and the adjusted target fusion model are obtained. The convergence condition can be any one of the following: the number of adjustments reaches a preset number of times, the total adjustment time reaches a preset time, or the target loss value reaches a minimum value. In the process of adjustment, any one of the algorithms in the stochastic gradient descent algorithm, Adagrad (Adaptive Gradient, adaptive gradient) algorithm, Adadelta (improvement of AdaGrad algorithm), RMSprop (improvement of AdaGrad algorithm), and Adam (Adaptive Moment Estimation, adaptive moment estimation) algorithm can be used for adjustment.

[0083] In one embodiment, after obtaining each target semantic expression feature and the target fusion model, the server may annotate the training video in the training video based on each target semantic expression feature and the target fusion model.

[0084] In other embodiments, the server may send each target semantic expression feature and the adjusted target fusion model to other computer devices, and the other computer devices may annotate the training videos in the training video set.

[0085] In the above-mentioned video annotation method, after determining the training video set, for each training video in the training video set, the first training triplet corresponding to the training video is obtained, and the semantic expression features corresponding to the video identifier, the first training relationship and the training label in the first target triplet are respectively obtained, and the semantic expression features corresponding to the video identifier and the content extraction features corresponding to the video identifier are input into the fusion model to obtain the training fusion features, and the first predicted label features corresponding to the first training relationship are determined based on the training fusion features and the semantic expression features corresponding to the first training relationship. Each first target triplet and the target fusion model are used to cooperate to implement the annotation of the training video. Since the associated content corresponding to the video identifier is fused in the adjustment process, each target semantic expression feature can express the semantic relationship between the training video and the label, and then based on each target semantic expression feature, when annotating, labels related to semantics can be annotated, thereby improving the diversity of video labels.

[0086] In one embodiment, the fusion model includes a mapping layer; the semantic expression features corresponding to the video identifier and the content extraction features corresponding to the video identifier are input into the fusion model to obtain training fusion features, including: mapping the content extraction features to the semantic space where the semantic expression features of the video identifier are located through the mapping layer to obtain mapping features corresponding to the content extraction features; fusing the mapping features with the semantic expression features corresponding to the video identifier to obtain training fusion features corresponding to the training video; based on the first difference between the semantic expression features corresponding to the training label and the first predicted label features, adjusting each semantic expression feature and the model parameters of the fusion model to obtain each adjusted target semantic expression feature and the adjusted target fusion model, including: based on the first difference, adjusting each semantic expression feature corresponding to the first training triplet and the model parameters of the mapping layer to obtain each adjusted target semantic expression feature and the adjusted target mapping layer.

[0087] The semantic space where the semantic expression features of the video identifier are located refers to the vector space containing the semantic expression features of the video identifier. Fusion refers to the process of expressing multiple features with one feature. Fusion can be specifically combined, spliced, or weighted summed.

[0088] Specifically, the server may input the content extraction feature map corresponding to the video identifier into the mapping layer, and the mapping layer may map the input content extraction feature map to the semantic space where the semantic expression feature of the video identifier is located through a nonlinear affine transformation to obtain the mapping feature corresponding to the content extraction feature. The server may further fuse the mapping feature and the semantic expression feature corresponding to the video identifier to obtain the training fusion feature corresponding to the training video.

[0089] In one embodiment, the server may perform affine transformation according to the following formula (1):

[0090]

[0091] in, is the obtained mapping feature, is the feature map extracted from the content, σ represents the nonlinear transformation, is the transformation matrix, b is the bias vector, and b are the model parameters of the mapping layer. Since the associated content may belong to multiple different modes, M i represents the i-th mode.

[0092] In one embodiment, the server may fuse the fusion mapping feature and the semantic expression feature corresponding to the video identifier according to the following formula (2):

[0093]

[0094] Among them, h vid To train the fusion features, z (k) is the semantic expression feature corresponding to the video tag, is the mapping feature of the first mode to which the associated content belongs, is the mapping feature of the i-th mode to which the associated content belongs.

[0095] Further, when adjusting the model parameters, the service can determine the target loss value based on the first difference, and iteratively adjust the model parameters of each semantic expression feature and the mapping layer based on the target loss value. When the convergence conditions are met, the adjusted target semantic expression features and the adjusted target mapping layer are obtained.

[0096] In the above embodiment, by mapping the content extraction features to the semantic space where the semantic expression features of the video identifier are located, better fusion can be performed in the same semantic space, and the obtained training fusion features are more accurate.

[0097] In one embodiment, the fusion model also includes an attention allocation layer; the mapping features and the semantic expression features corresponding to the video identifier are fused to obtain the training fusion features corresponding to the training video, including: calculating the attention of the mapping features and the semantic expression features of the video identifier respectively through the attention allocation layer, and obtaining the first attention corresponding to the mapping features and the second attention corresponding to the semantic expression features of the video identifier; performing attention processing on the mapping features based on the first attention to obtain the content attention features, and performing attention processing on the semantic expression features of the video identifier based on the second attention to obtain the identifier attention features; splicing the content attention features and the identifier attention features to obtain the training fusion features; based on the first difference, adjusting the semantic expression features corresponding to the first training triplet and the model parameters of the mapping layer to obtain the adjusted target semantic expression features and the adjusted target mapping layer, including: based on the first difference, adjusting the model parameters of the semantic expression features, the mapping layer and the model parameters of the attention allocation layer to obtain the adjusted target semantic expression features, the adjusted target mapping layer and the adjusted target attention allocation layer.

[0098] Among them, the attention degree is used to characterize the importance of the features of each modality in the video annotation process. The attention degree is positively correlated with the importance, that is, the greater the attention degree, the higher the importance. The first attention degree is used to characterize the importance of the modality to which the associated content belongs in the video annotation process. The second attention degree is used to characterize the importance of the semantic expression feature in the video annotation process, where the semantic expression feature can be regarded as a feature under the knowledge modality.

[0099] Specifically, if the knowledge modality and the modalities to which the associated content belongs constitute a modality set, then when a certain relationship is given to annotate a video with a corresponding label, under different relationships, the contribution of each modality to the annotation is different. The greater the contribution, the more important the features under the modality are for video annotation. For example, refer to Figure 3 For Video 1, Video 2, and Video 3, when given a “contains” relationship and labeling “cat” and “dog”, the content extraction features under the visual modality (i.e., video frame features) are obviously more important; and when given a “type” relationship and labeling “nature” is required, the semantic expression features under the knowledge modality are obviously more important.

[0100] In one embodiment, the mapping features corresponding to the associated content and the semantic expression features of the video identifier may be input into the attention distribution layer, and the attention distribution layer may calculate the attention by the following formula (3):

[0101]

[0102] Among them, M' represents the mode set, p i is the attention of the ith mode in the mode set, V and a are the model parameters of the attention allocation layer, Represents the input features of the i-th mode in the mode set.

[0103] Furthermore, the server can use the first attention as a weight to perform dot multiplication with the mapping feature to obtain the content attention feature, use the second attention as a weight to perform dot multiplication with the semantic expression feature of the video identifier to obtain the identifier attention feature, and then can concatenate the content attention feature and the identifier attention feature to obtain the training fusion feature. Finally, the server can obtain the target loss value based on the first difference, and iteratively adjust the semantic expression features corresponding to the first training triplet, the model parameters of the mapping layer, and the model parameters of the attention allocation layer based on the target loss value. When the convergence condition is met, the adjusted target semantic expression features, the adjusted target mapping layer, and the adjusted target attention allocation layer are obtained.

[0104] In one embodiment, the server may refer to the following formula (4) to obtain the training fusion feature:

[0105]

[0106] Among them, h vid To train the fusion feature, p0·z (k) represents the process of paying attention to the semantic expression features of the video identifier based on the second attention level, Represents the process of performing attention processing on the mapping features under the i-th modality to which the associated content belongs based on the first attention level of the modality.

[0107] In a specific embodiment, Figure 4 As shown in Figure 1, it is a schematic diagram of the specific process of fusion. Figure 3 The server inputs the content extraction features of each modality to which the associated content belongs into the mapping layer, maps the content extraction features to the semantic space where the semantic expression features of the video identifier are located in the mapping layer, and obtains the mapping features. The mapping features under each modality are input into the attention allocation layer together with the semantic expression features corresponding to the video identifier, and the second attention p0 corresponding to the video identifier, the first attention p1 under the visual modality to which the associated content belongs, and the first attention p2 under the auditory modality to which the associated content belongs are calculated. Based on p0, the semantic expression features (i.e., knowledge features) of the video identifier are subjected to attention processing, based on p1, the mapping features (i.e., video features) under the visual modality are subjected to attention processing, and based on p2, the mapping features (i.e., audio features) under the auditory modality are subjected to attention processing. Finally, the features obtained by the attention processing are spliced ​​to obtain the training fusion features.

[0108] In the above embodiment, different attention levels are allocated to each modality through the attention level allocation layer to reflect the importance of features in each modality in the annotation process, which can make the annotated labels more accurate.

[0109] In one embodiment, before adjusting each semantic expression feature and the model parameters of the fusion model based on the first difference between the semantic expression feature corresponding to the training label and the first predicted label feature to obtain the adjusted target semantic expression features and the adjusted target fusion model, the method also includes: obtaining a replacement fusion feature; replacing the fusion feature with a training fusion feature corresponding to other training videos; determining a second predicted label feature corresponding to the first training relationship based on the replacement fusion feature and the semantic expression feature corresponding to the first training relationship; adjusting each semantic expression feature and the model parameters of the fusion model based on the first difference between the semantic expression feature corresponding to the training label and the first predicted label feature to obtain the adjusted target semantic expression features and the adjusted target fusion model, including: determining a second difference based on the semantic expression feature corresponding to the training label and the second predicted label feature; adjusting each semantic expression feature and the model parameters of the fusion model based on the second difference and the first target difference between the first difference to obtain the adjusted target semantic expression features and the adjusted target fusion model.

[0110] It should be noted that the first training triplet in the above text is a correct triplet, that is, the semantic relationship between the video and the label represented by the triplet is real, that is, the first training triplet can be used as a positive sample. In this embodiment, interference copies of the first training triplet can be generated, and the interference copies can be used as negative samples. In the process of adjusting the model parameters, the loss value is determined by the first difference obtained from the positive sample and the second difference obtained from the negative sample.

[0111] Specifically, for the training fusion features corresponding to the video identifier in the first training triplet, the server can replace them with the training fusion features corresponding to other training videos to obtain the replacement fusion features corresponding to the negative sample, and determine the second prediction label features corresponding to the negative sample based on the replacement fusion features and the semantic expression features corresponding to the first training relationship, so that the second difference can be determined based on the semantic expression features corresponding to the training label and the second prediction label features, and finally, the target loss value is determined based on the first target difference between the first difference and the second difference, and the model parameters of each semantic expression feature and the fusion model are iteratively adjusted through the target loss value to obtain each adjusted target semantic expression feature triple and the adjusted target fusion model.

[0112] For example, assuming that the training video set in the above text includes three training videos, namely video 1, video 2 and video 3, for a certain video 1, the corresponding first training triplet is obtained as (h1, r1, t1), for video 2, the corresponding first training triplet is obtained as (h2, r2, t2), and for video 3, the corresponding first training triplet is obtained as (h3, r2, t3), then the server can replace h1 with any one of h2 and h3 to obtain the negative sample corresponding to the first training triplet (h1, r1, t1). Assuming that the server uses h2 as a replacement to generate a negative sample, when calculating the second difference corresponding to the negative sample, the server can obtain the training fusion feature corresponding to h2, determine the second predicted label feature X based on the training fusion feature corresponding to h2 and the semantic expression feature corresponding to r1, determine the first target difference based on the second difference between X and the semantic expression feature corresponding to t1, and the first difference, and calculate the target loss value based on the first target difference.

[0113] In one embodiment, the knowledge embedding method used by the server is TransE, and the server can refer to the following formula (5) to calculate the target loss value:

[0114]

[0115] Among them, (h, r, t) represents the first training triplet, (h′, r, t) represents the negative sample corresponding to the first training triplet, h+r represents the first predicted label feature, h′+r represents the second predicted label feature, d(h+r, t) represents the vector distance between the first predicted label feature and the semantic expression feature corresponding to the training label, which can be used as the first difference, d(h′+r, t) represents the vector distance between the second predicted label feature and the semantic expression feature corresponding to the training label, which can be used as the second difference, and γ is a preset value that can be set as needed. The vector distance can be L2 distance or L1 distance.

[0116] It can be seen from the above loss function that the training process will adjust the model parameters in the direction of reducing the first difference and increasing the second difference. Therefore, the vector sum of the feature representation of the video and semantic relationship can be made to approximate the correct label in the Euclidean space and punish the wrong label.

[0117] In the above embodiment, the second difference corresponding to the negative sample is obtained by obtaining the replacement fusion feature, and then the various semantic expression features corresponding to the first training triplet and the model parameters of the fusion model can be adjusted based on the first target difference between the first difference and the second difference. Since both positive samples and negative samples are considered at the same time, the obtained first target triplet and the model parameters of the target fusion model are more accurate.

[0118] In one embodiment, based on the first difference between the semantic expression feature corresponding to the training label and the first predicted label feature, each semantic expression feature and the model parameters of the fusion model are adjusted to obtain each adjusted target semantic expression feature and the adjusted target fusion model. The method also includes: obtaining a replacement label feature; replacing the label feature with the semantic expression feature corresponding to the training label corresponding to other training videos; based on the first difference between the semantic expression feature corresponding to the training label and the first predicted label feature, adjusting each semantic expression feature and the model parameters of the fusion model to obtain each adjusted target semantic expression feature and the adjusted target fusion model, including: determining a third difference based on the first predicted label feature and the replacement label feature; adjusting each semantic expression feature and the model parameters of the fusion model based on a second target difference between the first difference and the third difference to obtain each adjusted target semantic expression feature and the adjusted target fusion model.

[0119] In this embodiment, for the semantic expression features corresponding to the training labels in the first training triplet, the server can replace them with the semantic expression features corresponding to the training labels of other training videos, thereby obtaining the replacement label features corresponding to the negative samples, and determine the difference between the replacement label features and the first predicted label features as the third difference corresponding to the negative sample. Finally, based on the difference between the first difference and the third difference, the second target difference is obtained, and then the target loss value is determined according to the second target difference. The target loss value is iteratively adjusted to adjust each semantic expression feature and the model parameters of the fusion model to obtain the adjusted target semantic expression features and the adjusted target fusion model.

[0120] For example, as in the above example, the server can replace t1 with any one of t2 and t3 to obtain a negative sample corresponding to the first training triple (h1, r1, t1). Assuming that the server uses t2 as a replacement to generate a negative sample, when calculating the second difference corresponding to the negative sample, the server can obtain the semantic expression feature corresponding to t2, determine the second target difference based on the third difference between the semantic expression feature corresponding to t2 and the first predicted label feature, and the first difference, and calculate the target loss value based on the second target difference.

[0121] In the above embodiment, the third difference corresponding to the negative sample is obtained by obtaining the replacement label feature, and then the various semantic expression features corresponding to the first training triplet and the model parameters of the fusion model can be adjusted based on the second target difference between the first difference and the third difference. Since both positive samples and negative samples are considered at the same time, the obtained first target triplet and the model parameters of the target fusion model are more accurate.

[0122] In one embodiment, before inputting the semantic expression features corresponding to the video identifier and the content extraction features corresponding to the video identifier into the fusion model to obtain the trained fusion features, the method also: obtains the associated content corresponding to the video identifier; inputs the associated content into a content classification model that has been trained and corresponds to the associated content, and extracts content features for classification of the associated content through the content classification model; and determines the extracted content features for classification as the content extraction features corresponding to the associated content.

[0123] The associated content corresponding to the video identifier refers to the content associated with the video identifier. The associated content includes at least one of the video frame corresponding to the training identifier, the background music, and the associated text. The video frame may be one or more video frames randomly extracted from the video corresponding to the training identifier. The background music may be multiple audio frames extracted from the background audio corresponding to the training identifier. The associated text may be the title text of the video. Different associated contents correspond to different content classification models. The content classification model refers to a machine learning model used to classify content.

[0124] Specifically, the server can obtain a content classification model corresponding to the associated content, input the associated content into the trained content classification model corresponding to the associated content, extract content features for classification of the associated content through the content classification model, and determine the extracted content features for classification as content extraction features corresponding to the associated content.

[0125] For example, when the associated content is a video frame, the server can obtain a trained StNet classification model. The StNet classification model refers to a classification model with ResNet (residual network) as the backbone network; when the associated content is background music, the server can obtain a trained Vggish classification model. The Vggish model is pre-trained on YouTube's AudioSet dataset; when the associated content is associated text, the server can obtain a trained LSTM (Long Short-Term Memory)-attention (long short-term classification network based on attention mechanism) classification model.

[0126] In one embodiment, the last layer of the content classification model is usually a fully connected layer, which is used to classify the features extracted by the aforementioned network layer. Therefore, the server can obtain the input features of the last fully connected layer and determine the extracted input features as content extraction features corresponding to the associated content.

[0127] In the above embodiment, content features for classification are extracted from associated content through a trained content classification model, and the obtained content extraction features can better express the semantics of the associated content, so that the fused features can better express the semantics of the video, thereby obtaining semantically related labels.

[0128] In one embodiment, Figure 5 As shown, a video annotation method is provided, which is applied to Figure 1 Taking the server in as an example, the method specifically includes:

[0129] Step 502, determine a training video set, and for a training video in the training video set, obtain a first training triplet corresponding to the training video; the first training triplet includes a video identifier corresponding to the training video, a training label, and a first training relationship between the training video and the training label.

[0130] Step 504: Obtain semantic expression features corresponding to the video identifier, the first training relationship, and the training label, respectively.

[0131] Step 506, inputting the semantic expression feature corresponding to the video identifier and the content extraction feature corresponding to the video identifier into the fusion model to obtain the training fusion feature; the content extraction feature is obtained by extracting the feature of the associated content corresponding to the video identifier.

[0132] Step 508: input the semantic expression feature corresponding to the first training relationship into the relationship attention distribution model, and calculate the relationship attention corresponding to the first training relationship through the relationship attention distribution model.

[0133] Specifically, considering that some relations are of high importance and can almost uniquely determine the labels to be predicted. For example, the importance of semantic relations such as "main actor" and "background music type" is significantly higher than that of relations such as "video quality" and "video source", and can more accurately describe the unique properties of the video. Therefore, these relations can be differentiated by assigning relational attention through the relational attention allocation model. The higher the relational attention, the more important the relation is for labeling semantically related labels.

[0134] The server may input the semantic expression feature corresponding to the first training relationship into the relationship attention allocation model, and calculate the relationship attention corresponding to the first training relationship in the relationship attention allocation model by referring to the following formula (6):

[0135]

[0136] Among them, q i is the relationship attention, a r and v ris the model parameter of the relationship attention allocation model, r is the semantic expression feature corresponding to the training relationship, and |R| is the total number of training relationships.

[0137] It can be understood that, at the beginning of training, the model parameters of the relational attention allocation model can be obtained by initialization, and then the model parameters can be iteratively adjusted.

[0138] Step 510 , performing attention processing on the semantic expression feature corresponding to the first training relationship based on the relationship attention degree corresponding to the first training relationship, to obtain the relationship attention feature corresponding to the first training relationship.

[0139] Specifically, the server may perform a dot product of the relationship attention corresponding to the first training relationship as a weight and the semantic expression feature corresponding to the first training relationship to obtain the relationship attention feature corresponding to the first training relationship.

[0140] Step 512: Determine a first predicted label feature corresponding to the first training relationship based on the training fusion feature and the relationship focus feature corresponding to the first training relationship.

[0141] Step 514, based on the first difference between the semantic expression feature corresponding to the training label and the first predicted label feature, adjust each semantic expression feature, the model parameters of the fusion model, and the model parameters of the relationship attention allocation model to obtain each target semantic expression feature, the adjusted target fusion model, and the adjusted target relationship attention allocation model.

[0142] Specifically, after obtaining the first difference, the server can determine the target loss value based on the first difference, and iteratively adjust each semantic expression feature, the model parameters of the fusion model, and the model parameters of the relationship attention allocation model based on the target loss value. When the convergence conditions are met, the adjusted target semantic expression features, target fusion model, and target relationship attention allocation model are obtained.

[0143] In the above embodiment, by allocating relationship attention to the training relationship, the importance of each training relationship can be distinguished, so that when labeling, the importance of different relationships can be fully considered, making the labeled labels more accurate.

[0144] In one embodiment, the training labels in each first training triplet constitute a training label set; Figure 6 As shown, the above method also includes:

[0145] Step 602, obtaining a second training triplet; the second training triplet includes a training head label, a training tail label, and a second training relationship between the training head label and the training tail label; the training head label and the training tail label both belong to the training label set.

[0146] In this embodiment, for the knowledge graph established above, the relationship between labels can be established. For example, assuming that video 1 contains actor 1, video 2 contains actor 2, and actor 1 and actor 2 are friends, then a friend relationship between actor 1 and actor 2 can be established. In this way, a large number of triplets of (label 1, relationship, label 2) can be generated for the established knowledge graph. When these triplets are used as samples for training, they can be called second training triplets. Among them, label 1 is the training head label, label 2 is the training tail label, and the relationship therein is the second training relationship between the training head label and the training tail label. The server can obtain multiple second training triplets from the established knowledge graph for training, so that the semantic relationship between labels can be learned. Through the semantic relationship between labels, the video can be annotated with richer labels.

[0147] Step 604, respectively obtain the semantic expression features corresponding to the training head label, the training tail label and the second training relationship.

[0148] Specifically, at the beginning of training, the server can randomly initialize the semantic expression features of each node in the knowledge graph, and the server can obtain these initialized semantic expression features. The subsequent training process is iterative training, and each training obtains the adjustment result obtained in the previous time as the initial value in the current training process.

[0149] Step 606: Determine the predicted tail tag feature based on the semantic expression feature corresponding to the training head tag and the semantic expression feature corresponding to the second training relationship.

[0150] Specifically, during the training process, the server can preset different knowledge embedding methods. In different knowledge embedding methods, the server can determine the predicted tail tag features based on different methods. Taking TransE as an example, the server can add the semantic expression features corresponding to the training head tag and the semantic expression features corresponding to the second training relationship to obtain the predicted tail tag features.

[0151] Step 608: Obtain the tag feature difference between the predicted tail tag feature and the semantic expression feature corresponding to the training tail tag.

[0152] Specifically, for different preset knowledge embedding methods, the server can use corresponding methods to obtain the label feature difference between the predicted tail label feature and the semantic expression feature corresponding to the training tail label. Taking TransE as an example, the server can calculate the vector distance between the predicted tail label feature and the semantic expression feature corresponding to the training tail label to obtain the label feature difference. The vector distance can be, for example, L1 distance or L2 distance.

[0153] Step 610, based on the label feature difference, adjust the semantic expression features corresponding to the training head label, the training tail label and the second training relationship, and obtain the adjusted target semantic expression features corresponding to the training head label, the training tail label and the second training relationship.

[0154] Specifically, the server can determine the target loss value based on the label feature difference, and iteratively adjust the semantic expression features corresponding to the training head label, the training tail label, and the second training relationship based on the target loss value. When the convergence conditions are met, the adjusted target semantic expression features corresponding to the training head label, the training tail label, and the second training relationship are obtained.

[0155] It can be understood that the second training triplet in the above text is a correct triplet, that is, the semantic relationship between the label represented by the triplet and the label is real, that is, the second training triplet can be used as a positive sample. In one embodiment, the interference copy item of the second training triplet can be generated, and the interference copy item can be used as a negative sample. In the process of adjusting the model parameters, the target loss value is determined by the first difference obtained by the positive sample and the second difference obtained by the negative sample. In the specific implementation process, a negative sample corresponding to a second training triplet can be generated. Specifically, the training head label in the triplet can be replaced with the training head label of other second training triplet, or the training tail label in the triplet can be replaced with the training tail label of other second training triplet. After obtaining the negative sample, the server can calculate the label feature difference between the predicted tail label feature and the training tail label in the negative sample, based on the label feature difference corresponding to the positive sample and the label feature difference corresponding to the negative sample, and refer to formula (5) in the above embodiment to calculate the target loss value.

[0156] In the above embodiment, by obtaining the second training triplet, the semantic relationship between tags can be learned, so that richer features can be annotated for the video based on the semantic relationship between tags.

[0157] In one embodiment, the method also includes: inputting the semantic expression features corresponding to the second training relationship into a relationship attention allocation model, and calculating the relationship attention corresponding to the second training relationship through the relationship attention allocation model; determining the predicted tail label features based on the semantic expression features corresponding to the training head label and the semantic expression features corresponding to the second training relationship, including: performing attention processing on the semantic expression features corresponding to the second training relationship based on the relationship attention corresponding to the second training relationship, and obtaining the relationship attention features corresponding to the second training relationship; determining the predicted tail label features based on the semantic expression features corresponding to the training head label and the relationship attention features corresponding to the second training relationship; adjusting the semantic expression features corresponding to the training head label, the training tail label, and the second training relationship based on the label feature difference, and obtaining the adjusted target semantic expression features corresponding to the training head label, the training tail label, and the second training relationship, including: adjusting the semantic expression features corresponding to the training head label, the training tail label, and the second training relationship, and the model parameters of the relationship attention allocation model based on the label feature difference, and obtaining the adjusted target semantic expression features corresponding to the training head label, the training tail label, and the second training relationship, and the adjusted target relationship attention allocation model.

[0158] Specifically, the server can refer to the above formula (6) to calculate the relationship attention corresponding to the second training relationship, use the calculated relationship attention as a weight to perform point multiplication with the semantic expression feature corresponding to the second training relationship to obtain the relationship attention feature corresponding to the second training relationship, and then determine the predicted tail label feature based on the semantic expression feature corresponding to the training head label and the relationship attention feature corresponding to the second training relationship, and obtain the label feature difference based on the difference between the predicted tail label feature and the semantic expression feature corresponding to the training tail label. The target loss value can be determined based on the label feature difference, and the semantic expression features corresponding to the training head label, the training tail label and the second training relationship, as well as the model parameters of the relationship attention allocation model, are iterated based on the target loss value until the convergence condition is met. The specific implementation details of this embodiment can be referred to the description in the above embodiment, and this application will not repeat them here.

[0159] In one embodiment, Figure 7 As shown, a video annotation method is provided, and the method is applied to the above Figure 1 The server in the example is used as an example to illustrate the specific steps:

[0160] Step 702, obtaining a target fusion feature corresponding to the target video to be annotated; the target fusion feature is obtained by inputting a target semantic expression feature corresponding to the video identifier of the target video and a content extraction feature corresponding to the video identifier into a target fusion model.

[0161] The target video to be labeled refers to the video that needs to be labeled.

[0162] Step 704: Based on the target fusion feature and the target semantic expression feature corresponding to the first target relationship, obtain the target prediction label feature corresponding to the first target relationship.

[0163] The first target relationship refers to a pre-specified relationship of interest, which may be any relationship involved in the training video set.

[0164] Specifically, the server can determine the target prediction label feature corresponding to the first target relationship in a corresponding manner based on the knowledge embedding method used in the process of determining the first target triple. Taking TransE as an example, the server can add the target fusion feature and the adjusted semantic expression feature corresponding to the first target relationship to obtain the target prediction label feature corresponding to the first target relationship.

[0165] Step 706, the target semantic expression features corresponding to each training label are matched with the target predicted label features respectively, and the training label corresponding to the successfully matched target semantic expression feature is determined as the first annotation label corresponding to the target video.

[0166] Among them, the target fusion model and each target semantic expression feature are obtained by adjusting the fusion model to be adjusted and each semantic expression feature to be adjusted based on the first difference corresponding to the first training triplet; the first training triplet corresponds to a training video in the training video set to which the target video belongs, including a video identifier, a training label corresponding to the training video, and a first training relationship between the training video and the training label; the first difference is the difference between the semantic expression feature corresponding to the training label and the first predicted label feature; the first predicted label feature corresponds to the first training relationship, and is determined based on the training fusion feature and the semantic expression feature corresponding to the first training relationship; the training fusion feature is obtained by inputting the semantic expression feature corresponding to the video identifier and the content extraction feature corresponding to the video identifier into the fusion model; the content extraction feature is obtained by extracting the feature of the associated content corresponding to the video identifier.

[0167] Specifically, during the matching process, the server can calculate the vector distance between the adjusted semantic expression features corresponding to the training labels and the target predicted label features, determine the semantic expression features with the closest vector distance as the successfully matched semantic expression features, and determine the training labels corresponding to the successfully matched semantic expression features as the first annotation labels corresponding to the target video, thereby annotating the target video with semantic-level labels.

[0168] In the above embodiment, the target fusion model and each target semantic expression feature are obtained by adjusting the fusion model to be adjusted and each semantic expression feature to be adjusted based on the first difference corresponding to the first training triplet; the first training triplet corresponds to the training video in the training video set to which the target video belongs, including the video identifier, training label corresponding to the training video, and the first training relationship between the training video and the training label; the first difference is the difference between the semantic expression feature corresponding to the training label and the first predicted label feature; the first predicted label feature corresponds to the first training relationship, and is determined based on the training fusion feature and the semantic expression feature corresponding to the first training relationship; the training fusion feature is obtained by inputting the semantic expression feature corresponding to the video identifier and the content extraction feature corresponding to the video identifier into the fusion model; the content extraction feature is obtained by extracting the associated content corresponding to the video identifier, so that the associated content corresponding to the video identifier is integrated into the obtained target semantic expression features, and the semantic relationship between the training video and the label can be expressed accurately, and then after obtaining the target predicted label feature, by matching with the target semantic expression feature corresponding to each training label, accurate semantic labels can be obtained, thereby improving the diversity of video labels.

[0169] In one embodiment, the method further includes: obtaining a target semantic expression feature corresponding to a target tag of a target video; determining a target tail tag feature based on the target semantic expression feature corresponding to the target tag and the target semantic expression feature corresponding to the second target relationship; matching the target semantic expression features corresponding to each training tail tag with the target tail tag feature, and determining the training tail tag corresponding to the successfully matched target semantic expression feature as the second annotation tag corresponding to the target video; the target semantic expression feature corresponding to the target tag, the target semantic expression feature corresponding to the second target relationship, and the target semantic expression feature corresponding to each training tail tag are obtained by adjusting the corresponding semantic expression features to be adjusted based on the label feature difference corresponding to the second training triple; the second training triple includes a training head tag, a training tail tag, and a second training relationship between the training head tag and the training tail tag; the training head tag and the training tail tag both belong to a training tag set; the training tag set is composed of the training tags in each first training triple corresponding to the training video set; the label feature difference is the difference between the predicted tail tag feature and the semantic expression feature corresponding to the training tail tag; the predicted tail tag feature is determined based on the semantic expression feature corresponding to the training head tag and the semantic expression feature corresponding to the second training relationship.

[0170] The target label refers to the annotated label corresponding to the target video. The second target relationship refers to a pre-specified relationship of interest, which can be any relationship involved in the training video set.

[0171] In one embodiment, after obtaining the target semantic expression feature corresponding to the target tag, the server can add the target semantic expression feature corresponding to the target tag and the target semantic expression feature corresponding to the second target relationship to obtain the target tail tag feature, respectively calculate the vector distance between the target semantic expression feature corresponding to each training tail tag and the target tail tag feature, and determine the training tail tag corresponding to the target semantic expression feature with the closest vector distance as the second annotation tag corresponding to the target video.

[0172] In the above embodiment, by obtaining the second training triplet, the semantic relationship between tags can be learned, so that richer features can be annotated for the video based on the semantic relationship between tags.

[0173] In one embodiment, the first annotation tag and the second annotation tag are used to form a target tag set corresponding to the target video; the method also includes: obtaining a set of videos to be pushed; the set of videos to be pushed includes multiple videos to be pushed, and each video to be pushed has a corresponding first associated tag; matching the tags in the target tag set with each first associated tag, and determining the video to be pushed corresponding to the successfully matched first associated tag as the target pushed video corresponding to the target video; pushing the target pushed video to the terminal corresponding to the target user corresponding to the target video.

[0174] The first associated tag corresponding to the video to be pushed refers to the tagged tag of the video to be pushed. The target user corresponding to the target video may be the user who views the target video.

[0175] Specifically, the server can match the tags in the target tag set with each first associated tag. When a first associated tag that is the same as a tag in the target tag set is matched, it means that the video to be pushed corresponding to the first associated tag has the same characteristics as the target video. Then the server can determine the video to be pushed corresponding to the first associated tag as the target pushed video, and push the target pushed video to the target user corresponding to the target video, thereby recommending suitable videos to the target user.

[0176] In one embodiment, the first annotation tag and the second annotation tag are used to form a target tag set corresponding to the target video; the above method also includes: obtaining a candidate user set; the candidate user set includes multiple candidate users, and each candidate user has a corresponding second associated tag; matching the tags in the target tag set with each second associated tag, and determining the associated user corresponding to the successfully matched second associated tag as the target push user corresponding to the target video; pushing the target video to the terminal corresponding to the target push user.

[0177] The first associated tag corresponding to the candidate user refers to the marked tag of the candidate user.

[0178] Specifically, the server can match the tags in the target tag set with each second associated tag. When a second associated tag that is the same as the tag in the target tag set is matched, it means that the candidate user corresponding to the second associated tag has the same characteristics as the target video. Then the server can determine the associated user corresponding to the second associated tag as the target push user corresponding to the target video, and push the target video to the terminal corresponding to the target push user, thereby expanding more users for the target video.

[0179] The present application also provides an application scenario, in which the above-mentioned video annotation method is applied. In this application scenario, the server can implement video recommendation. Figure 8 ,The application of the video annotation method in this application scenario is as follows:

[0180] 1. First, a knowledge graph expressing multiple relationships between videos and labels and labels and labels is constructed from the attribute data of the training video set, and a specific knowledge embedding method based on vector translation distance is defined (taking TransE as an example) as well as the semantic expression features of each node in the initialization knowledge graph. For each training video in the training video set, the server can obtain the first training triplet as the first positive sample from the established knowledge graph. The first training triplet includes the video identifier and training label corresponding to the training video, and the first training relationship between the training video and the training label. The server can also obtain the second training triplet as the second positive sample from the established knowledge graph. The second training triplet includes a training head label, a training tail label, and a second training relationship between the training head label and the training tail label.

[0181] 2. Obtain content extraction features corresponding to the associated contents of the training videos in the training video set under multiple modalities. Figure 8 , multiple modalities may include background music in the auditory modality, for example, multiple audio frames can be extracted from the background music and input into the VGGish classification model, and the input features of the last fully connected layer of the classification model are used as content extraction features in the auditory modality. Figure 8 For example, multiple modalities may also include video frames in a visual modality. Multiple video frames may be extracted from a training video and input into a ResNet classification model, and the input features of the last fully connected layer of the classification model may be used as content extraction features in the visual modality.

[0182] 3. Input the content extraction features corresponding to the related content under multiple modes into the mapping layer of the fusion model (the mapping layer is the network layer in the fusion model, Figure 8(not shown in the figure), the content extraction features are mapped to the semantic space where the semantic expression features of the video identifier are located through the mapping layer, and the mapping features corresponding to the content extraction features under each mode are obtained.

[0183] 4. Input the mapping features corresponding to the content extraction features under each modality and the semantic expression features corresponding to the video identifier into the attention allocation layer of the fusion model (the mapping layer is the network layer in the fusion model, Figure 8 (not shown in the figure), the attention degree of each mapping feature and the semantic expression feature of the video identifier are calculated through the attention distribution layer, and the first attention degree corresponding to each mapping feature and the second attention degree corresponding to the semantic expression feature of the video identifier are obtained. Based on the first attention degrees, the corresponding mapping features are subjected to attention processing to obtain each content attention feature, and based on the second attention degrees, the semantic expression features of the video identifier are subjected to attention processing to obtain the identifier attention feature, and the content attention features and the identifier attention features are concatenated to obtain the training fusion feature.

[0184] 5. Input the semantic expression features corresponding to the first training relation into the relation attention allocation model ( Figure 8 (not shown in the figure), the relationship attention corresponding to the first training relationship is calculated through the relationship attention distribution model, the semantic expression feature corresponding to the first training relationship is paid attention to based on the relationship attention corresponding to the first training relationship, and the relationship attention feature corresponding to the first training relationship is obtained, and the first predicted label feature corresponding to the first training relationship is determined based on the scoring function corresponding to the predefined knowledge embedding method, and the first difference between the semantic expression feature corresponding to the training label and the first predicted label feature is obtained.

[0185] Taking TransE as an example, its scoring function is as follows (7):

[0186] f r (h,t)=||h+rt|| (7)

[0187] Therefore, the server can add the training fusion feature and the relationship attention feature corresponding to the first training relationship to obtain the first predicted label feature corresponding to the first training relationship.

[0188] 6. Obtain a first negative training sample corresponding to the first training triplet, obtain a second difference corresponding to the first negative training sample, and calculate a first target loss value based on the first difference and the second difference with reference to the above formula (5).

[0189] Among them, the first negative training sample can replace the training fusion feature corresponding to the video label in the first training triplet to obtain the replaced fusion feature, and the corresponding second difference can be obtained as follows: based on the relationship attention feature corresponding to the replaced fusion feature and the first training relationship, determine the second predicted label feature corresponding to the first training relationship, and determine the difference between the semantic expression feature corresponding to the training label and the second predicted label feature as the second difference.

[0190] 7. Input the semantic expression features corresponding to the second training relationship into the relationship attention allocation model, calculate the relationship attention corresponding to the second training relationship through the relationship attention allocation model, perform attention processing on the semantic expression features corresponding to the second training relationship based on the relationship attention corresponding to the second training relationship, and obtain the relationship attention features corresponding to the second training relationship, determine the predicted tail label features based on the semantic expression features corresponding to the training head label and the relationship attention features corresponding to the second training relationship, and obtain the first label feature difference between the predicted tail label features and the semantic expression features corresponding to the training tail label.

[0191] 8. Obtain the second negative training sample corresponding to the second training triplet, obtain the second label feature difference corresponding to the second negative training sample, and calculate the second target loss value based on the first label feature difference and the second label feature difference with reference to the above formula (5). Among them, the second negative training sample can replace the semantic expression feature corresponding to the training head label in the second training triplet to obtain the replacement fusion feature, and then the corresponding second label feature difference can be obtained by: based on the semantic expression feature corresponding to the training head label and the relationship attention feature corresponding to the second training relationship, determine the predicted tail label feature corresponding to the second training relationship, and determine the difference between the obtained predicted tail label feature and the semantic expression feature corresponding to the training tail label as the second label feature difference.

[0192] 9. For the first training triplet and the second training triplet in a batch, the server obtains all the first target loss values ​​and the second target loss values ​​through the above steps 2 to 8, and then adds and averages them to obtain the final loss value, and adjusts the semantic expression features of all nodes involved in the batch in the knowledge graph through the loss value for iterative adjustment until the convergence condition is met, and the target semantic expression features of these nodes are obtained. At the same time, the server can also iteratively adjust the model parameters of the fusion model and the relationship attention allocation model based on the final loss value, and obtain the target fusion model and the target relationship attention allocation model when the convergence condition is met.

[0193] After obtaining the target fusion model and each target semantic expression feature, for each video identifier, the server can input its corresponding target semantic expression feature into the target fusion model to obtain the corresponding target fusion feature, and can establish an association between the target fusion feature and the video identifier. In the subsequent labeling process, the target fusion feature can be directly used to label the video.

[0194] 10. When a training video in the training video set needs to be labeled, the server determines the training video as the target video to be labeled. Figure 8 The server can annotate the video based on the fusion features of the target video (the fusion features in this case are the target fusion features) and each target semantic expression feature to obtain the annotation label. Specifically, there are two ways:

[0195] 1) Based on the target fusion feature and the target semantic expression feature corresponding to the first target relationship, the target prediction label feature corresponding to the first target relationship is obtained; the target semantic expression features corresponding to each training label are matched with the target prediction label feature respectively, and the training label corresponding to the successfully matched target semantic expression feature is determined as the first annotation label corresponding to the target video.

[0196] 2) Determine the target tail label feature based on the target semantic expression feature corresponding to the target label and the target semantic expression feature corresponding to the second target relationship; match the target semantic expression features corresponding to each training tail label with the target tail label feature respectively, and determine the training tail label corresponding to the successfully matched target semantic expression feature as the second annotation label corresponding to the target video.

[0197] 12. The aforementioned first annotation tag and the second annotation tag are used to form a target tag set corresponding to the target video. The server can also obtain a set of videos to be pushed; the set of videos to be pushed includes multiple videos to be pushed, and each video to be pushed has a corresponding associated tag; the tags in the target tag set are matched with each associated tag, and the video to be pushed corresponding to the successfully matched associated tag is determined as the target push video corresponding to the target video; the target push video is pushed to the terminal corresponding to the target user corresponding to the target video.

[0198] The present application also provides another application scenario, which applies the above-mentioned video annotation method. In this application scenario, the server can realize user promotion. Specifically, after the server obtains the first annotation label and the second annotation label through the steps in the previous text, it can form a target label set corresponding to the target video. The server can further obtain a set of videos to be pushed including multiple videos to be pushed, wherein each video to be pushed has a corresponding associated label. The server can match the label in the target label set with each associated label, and determine the video to be pushed corresponding to the successfully matched associated label as the target pushed video corresponding to the target video; and push the target pushed video to the terminal corresponding to the target user corresponding to the target video.

[0199] It should be understood that, although the steps in the flowcharts involved in the above embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps is not strictly limited in order, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above embodiments may include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0200] The video annotation method provided by the present invention can be used to conduct experiments on the first dataset (Company-200K), the second dataset (Company-300K), and the third dataset (FB15K-237). The experimental data obtained from these experiments fully verify the annotation effect of the present application. Among them, MR, HITS@1 and HITS@10 are three different accuracy evaluation indicators in the field of video annotation. The details are as follows:

[0201] like Fig. 9 As shown in FIG. 1 , it is a schematic diagram of the test results on the third data set. Among them, methods 1 to 4 are all methods in the prior art, and the last one is the video annotation method of this application. Fig. 9 It can be seen that the marking accuracy of this application is the highest.

[0202] like Fig.10 As shown in Figure , it is the experimental data under the condition of pre-defined different knowledge embedding methods (TransE, TransH, TransR and TransD). Fig.10 It can be seen that this application can achieve good performance when predefining different knowledge embedding methods.

[0203] like Fig.11 As shown in the figure, it is a schematic diagram of the scalable visualization experimental results. Fig.11 , we can see that the running time of the variant that combines different modes basically increases linearly with the data size. Fig.11 Figure (a) shows the experiment conducted on the first dataset, while Fig.11 Figure (b) in the figure is an experiment conducted on the first dataset. Method 1-VAT refers to a method that integrates visual modalities, auditory modalities, and textual modalities; Method 2-VA refers to a method that integrates visual modalities and auditory modalities; Method 3-V refers to a method that integrates visual modalities; Method 3-None refers to a method that does not integrate any modality related to the associated content; The original method refers to the traditional technology method, that is, directly classifying and labeling video frames.

[0204] Based on the same inventive concept, the embodiment of the present application also provides a video annotation device for implementing the above-mentioned video annotation method. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above-mentioned method, so the specific limitations in one or more video annotation device embodiments provided below can refer to the limitations of the video annotation method above, and will not be repeated here.

[0205] In one embodiment, Fig.12 As shown, a video annotation device 1200 is provided, including:

[0206] The training sample acquisition module 1202 is used to determine a training video set, and for a training video in the training video set, obtain a first training triplet corresponding to the training video; the first training triplet includes a video identifier corresponding to the training video, a training label, and a first training relationship between the training video and the training label;

[0207] The semantic expression feature acquisition module 1204 is used to respectively acquire the semantic expression features corresponding to the video identifier, the first training relationship and the training label;

[0208] Fusion module 1206, used to input the semantic expression feature corresponding to the video identifier and the content extraction feature corresponding to the video identifier into the fusion model to obtain a training fusion feature; the content extraction feature is obtained by extracting the feature of the associated content corresponding to the video identifier;

[0209] A label prediction module 1208, configured to determine a first predicted label feature corresponding to the first training relationship based on the training fusion feature and the semantic expression feature corresponding to the first training relationship;

[0210] The parameter adjustment module 1210 is used to adjust each semantic expression feature and the model parameters of the fusion model based on the first difference between the semantic expression feature corresponding to the training label and the first predicted label feature, so as to obtain the adjusted target semantic expression features and the adjusted target fusion model; each target semantic expression feature and the target fusion model are used to cooperate in realizing the annotation of the training video.

[0211] After determining the training video set, the above-mentioned video annotation device obtains the first training triplet corresponding to the training video for each training video in the training video set, obtains the semantic expression features corresponding to the video identifier, the first training relationship and the training label in the first target triplet respectively, inputs the semantic expression features corresponding to the video identifier and the content extraction features corresponding to the video identifier into the fusion model to obtain the training fusion features, and determines the first predicted label features corresponding to the first training relationship based on the training fusion features and the semantic expression features corresponding to the first training relationship. Each first target triplet and the target fusion model are used to cooperate to implement the annotation of the training video. Since the associated content corresponding to the video identifier is fused in the adjustment process, each target semantic expression feature can express the semantic relationship between the training video and the label, and then based on each target semantic expression feature, when annotating, labels related to semantics can be annotated, thereby improving the diversity of video labels.

[0212] In one embodiment, the fusion module is also used to map the content extraction features to the semantic space where the semantic expression features of the video identifier are located through the mapping layer, obtain the mapping features corresponding to the content extraction features, fuse the mapping features with the semantic expression features corresponding to the video identifier, and obtain the training fusion features corresponding to the training video; the parameter adjustment module is also used to adjust the semantic expression features corresponding to the first training triplet and the model parameters of the mapping layer based on the first difference, and obtain the adjusted target semantic expression features and the adjusted target mapping layer.

[0213] In one embodiment, the fusion module is also used to calculate the respective attention levels of the mapping feature and the semantic expression feature of the video identifier through the attention distribution layer, obtain the first attention level corresponding to the mapping feature and the second attention level corresponding to the semantic expression feature of the video identifier, perform attention processing on the mapping feature based on the first attention level to obtain the content attention feature, perform attention processing on the semantic expression feature of the video identifier based on the second attention level to obtain the identifier attention feature, splice the content attention feature and the identifier attention feature to obtain the training fusion feature; the parameter adjustment module is also used to adjust each semantic expression feature, the model parameters of the mapping layer and the model parameters of the attention distribution layer based on the first difference, and obtain each adjusted target semantic expression feature, the adjusted target mapping layer and the adjusted target attention distribution layer.

[0214] In one embodiment, the parameter adjustment module is also used to obtain replacement fusion features; replace the fusion features with training fusion features corresponding to other training videos; determine the second prediction label feature corresponding to the first training relationship based on the replacement fusion features and the semantic expression features corresponding to the first training relationship; determine the second difference based on the semantic expression features corresponding to the training label and the second prediction label features; adjust each semantic expression feature and the model parameters of the fusion model based on the second difference and the first target difference between the first difference to obtain each adjusted target semantic expression feature and the adjusted target fusion model.

[0215] In one embodiment, the parameter adjustment module is also used to obtain replacement label features; the replacement label features are semantic expression features corresponding to training labels corresponding to other training videos; the third difference is determined based on the first predicted label features and the replacement label features; based on the second target difference between the first difference and the third difference, each semantic expression feature and the model parameters of the fusion model are adjusted to obtain each adjusted target semantic expression feature and the adjusted target fusion model.

[0216] In one embodiment, the above-mentioned device also includes a content extraction module, which is used to obtain associated content corresponding to the video identifier; input the associated content into a content classification model that has been trained and corresponds to the associated content, and extract content features for classification of the associated content through the content classification model; and determine the extracted content features for classification as content extraction features corresponding to the associated content.

[0217] In one embodiment, the associated content includes at least one of a video frame corresponding to the training identifier, background music, and associated text; and different associated contents correspond to different content classification models.

[0218] In one embodiment, the above-mentioned device also includes a relationship allocation module, which is used to input the semantic expression features corresponding to the first training relationship into the relationship attention allocation model, and calculate the relationship attention corresponding to the first training relationship through the relationship attention allocation model; the label prediction module is also used to perform attention processing on the semantic expression features corresponding to the first training relationship based on the relationship attention corresponding to the first training relationship, and obtain the relationship attention features corresponding to the first training relationship; based on the training fusion features and the relationship attention features corresponding to the first training relationship, determine the first predicted label features corresponding to the first training relationship; the parameter adjustment module is also used to adjust each semantic expression feature, the model parameters of the fusion model and the model parameters of the relationship attention allocation model based on the first difference, and obtain the adjusted target semantic expression features, the adjusted target fusion model and the adjusted target relationship attention allocation model.

[0219] In one embodiment, the training labels in each first training triple constitute a training label set; the training sample acquisition module is also used to obtain the second training triple; the second training triple includes a training head label, a training tail label, and a second training relationship between the training head label and the training tail label; the training head label and the training tail label both belong to the training label set; the semantic expression feature acquisition module is also used to respectively obtain the semantic expression features corresponding to the training head label, the training tail label and the second training relationship; the label prediction module is also used to determine the predicted tail label feature based on the semantic expression feature corresponding to the training head label and the semantic expression feature corresponding to the second training relationship; the parameter adjustment module is also used to obtain the label feature difference between the predicted tail label feature and the semantic expression feature corresponding to the training tail label; based on the label feature difference, the semantic expression features corresponding to the training head label, the training tail label and the second training relationship are adjusted to obtain the adjusted target semantic expression features corresponding to the training head label, the training tail label and the second training relationship.

[0220] In one embodiment, the above-mentioned device also includes a relationship attention allocation module, which is used to input the semantic expression features corresponding to the second training relationship into a relationship attention allocation model, and calculate the relationship attention corresponding to the second training relationship through the relationship attention allocation model; the label prediction module is also used to perform attention processing on the semantic expression features corresponding to the second training relationship based on the relationship attention corresponding to the second training relationship, and obtain the relationship attention features corresponding to the second training relationship; based on the semantic expression features corresponding to the training head label and the relationship attention features corresponding to the second training relationship, determine the predicted tail label features; the parameter adjustment module is also used to adjust the semantic expression features corresponding to the training head label, the training tail label and the second training relationship, as well as the model parameters of the relationship attention allocation model based on the label feature differences, to obtain the adjusted target semantic expression features corresponding to the training head label, the training tail label and the second training relationship, and the adjusted target relationship attention allocation model.

[0221] In one embodiment, Fig.13 As shown, another video annotation device 1300 is provided, including:

[0222] The fusion feature acquisition module 1302 is used to acquire the target fusion feature corresponding to the target video to be annotated; the target fusion feature is obtained by inputting the target semantic expression feature corresponding to the video identifier of the target video and the content extraction feature corresponding to the video identifier into the target fusion model;

[0223] The label prediction module 1304 is used to obtain a target prediction label feature corresponding to the first target relationship based on the target fusion feature and the target semantic expression feature corresponding to the first target relationship;

[0224] The feature matching module 1306 is used to match the target semantic expression features corresponding to each training label with the target prediction label features, and determine the training label corresponding to the successfully matched target semantic expression feature as the first annotation label corresponding to the target video;

[0225] Among them, the target fusion model and each target semantic expression feature are obtained by adjusting the fusion model to be adjusted and each semantic expression feature to be adjusted based on the first difference corresponding to the first training triplet; the first training triplet corresponds to a training video in the training video set to which the target video belongs, including a video identifier, a training label corresponding to the training video, and a first training relationship between the training video and the training label; the first difference is the difference between the semantic expression feature corresponding to the training label and the first predicted label feature; the first predicted label feature corresponds to the first training relationship, and is determined based on the training fusion feature and the semantic expression feature corresponding to the first training relationship; the training fusion feature is obtained by inputting the semantic expression feature corresponding to the video identifier and the content extraction feature corresponding to the video identifier into the fusion model; the content extraction feature is obtained by extracting the feature of the associated content corresponding to the video identifier.

[0226] The above-mentioned video annotation device, since the target fusion model and each target semantic expression feature are obtained by adjusting the fusion model to be adjusted and each semantic expression feature to be adjusted based on the first difference corresponding to the first training triplet; the first training triplet corresponds to the training video in the training video set to which the target video belongs, including the video identifier and training label corresponding to the training video, and the first training relationship between the training video and the training label; the first difference is the difference between the semantic expression feature corresponding to the training label and the first predicted label feature; the first predicted label feature corresponds to the first training relationship, and is determined based on the training fusion feature and the semantic expression feature corresponding to the first training relationship; the training fusion feature is obtained by inputting the semantic expression feature corresponding to the video identifier and the content extraction feature corresponding to the video identifier into the fusion model; the content extraction feature is obtained by extracting the associated content corresponding to the video identifier, so that the associated content corresponding to the video identifier is integrated into the obtained each target semantic expression feature, and the semantic relationship between the training video and the label can be expressed accurately, and then after obtaining the target predicted label feature, by matching with the target semantic expression feature corresponding to each training label, an accurate semantic label can be obtained, thereby improving the diversity of video labels.

[0227] In one embodiment, the fusion feature acquisition module is further used to obtain the target semantic expression feature corresponding to the target label of the target video; the label prediction module is further used to determine the target tail label feature based on the target semantic expression feature corresponding to the target label and the target semantic expression feature corresponding to the second target relationship; the feature matching module is further used to match the target semantic expression features corresponding to each training tail label with the target tail label feature, respectively, and determine the training tail label corresponding to the successfully matched target semantic expression feature as the second annotation label corresponding to the target video;

[0228] Among them, the target semantic expression feature corresponding to the target label, the target semantic expression feature corresponding to the second target relationship and the target semantic expression feature corresponding to each training tail label are obtained by adjusting the corresponding semantic expression features to be adjusted based on the label feature difference corresponding to the second training triplet; the second training triplet includes a training head label, a training tail label, and a second training relationship between the training head label and the training tail label; the training head label and the training tail label both belong to a training label set; the training label set is composed of the training labels in the first training triplets corresponding to the training video set; the label feature difference is the difference between the predicted tail label feature and the semantic expression feature corresponding to the training tail label; the predicted tail label feature is determined based on the semantic expression feature corresponding to the training head label and the semantic expression feature corresponding to the second training relationship.

[0229] In one embodiment, the first annotation tag and the second annotation tag are used to form a target tag set corresponding to the target video; the above-mentioned device also includes a first push module, which is used to obtain a set of videos to be pushed; the set of videos to be pushed includes multiple videos to be pushed, and each video to be pushed has a corresponding first associated tag; the tags in the target tag set are matched with each first associated tag, and the video to be pushed corresponding to the successfully matched first associated tag is determined as the target pushed video corresponding to the target video; the target pushed video is pushed to the terminal corresponding to the target user corresponding to the target video.

[0230] In one embodiment, the first annotation tag and the second annotation tag are used to form a target tag set corresponding to the target video; the above-mentioned device also includes a second push module, which is used to obtain a candidate user set; the candidate user set includes multiple candidate users, and each candidate user has a corresponding second associated tag; the tags in the target tag set are matched with each second associated tag, and the associated user corresponding to the successfully matched second associated tag is determined as the target push user corresponding to the target video; the target video is pushed to the terminal corresponding to the target push user.

[0231] Each module in the above video annotation device can be implemented in whole or in part by software, hardware, or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each module.

[0232] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Fig.14 As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store video data. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a video annotation method is implemented.

[0233] Those skilled in the art will understand that Fig.14 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0234] In one embodiment, a computer device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above method embodiments when executing the computer program.

[0235] In one embodiment, a computer-readable storage medium is provided, storing a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0236] In one embodiment, a computer program product or computer program is provided, the computer program product or computer program includes computer instructions, the computer instructions are stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device performs the steps in the above-mentioned method embodiments.

[0237] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.

[0238] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.

[0239] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0240] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be construed as limiting the scope of the present application. It should be noted that, for a person of ordinary skill in the art, several modifications and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A video annotation method, characterized in that: The method comprises: A knowledge graph expressing multiple relationships between videos and labels and labels and labels is constructed from the attribute data of the training video set, the semantic expression features of each node in the knowledge graph are initialized, and a first training triplet is obtained from the established knowledge graph, wherein the first training triplet includes a video identifier and a training label corresponding to a training video in the training video set, and a first training relationship between the training video and the training label; a second training triplet is obtained from the established knowledge graph, wherein the second training triplet includes a training head label, a training tail label, and a second training relationship between the training head label and the training tail label; Respectively obtaining semantic expression features corresponding to the video identifier, the first training relationship, and the training label; Inputting the semantic expression feature corresponding to the video identifier and the content extraction feature corresponding to the video identifier into a fusion model to obtain a training fusion feature; the content extraction feature is obtained by extracting the feature of the associated content corresponding to the video identifier; Determining a first predicted label feature corresponding to the first training relationship based on the training fusion feature and the semantic expression feature corresponding to the first training relationship; Obtaining a first difference between a semantic expression feature corresponding to the training label and the first predicted label feature, obtaining a first negative training sample corresponding to the first training triplet, obtaining a second difference corresponding to the first negative training sample, and calculating a first target loss value based on the first difference and the second difference; Respectively obtain the semantic expression features corresponding to the training head label, the training tail label, and the second training relationship; determine the predicted tail label feature based on the semantic expression feature corresponding to the training head label and the semantic expression feature corresponding to the second training relationship; obtain the first label feature difference between the predicted tail label feature and the semantic expression feature corresponding to the training tail label, obtain the second negative training sample corresponding to the second training triplet, obtain the second label feature difference corresponding to the second negative training sample, and calculate the second target loss value based on the first label feature difference and the second label feature difference; For the first training triplet and the second training triplet in a batch, all the first target loss values ​​and the second target loss values ​​obtained through the above steps are added and averaged to obtain a final loss value, and the semantic expression features of all nodes involved in the batch in the knowledge graph are adjusted by the loss value for iterative adjustment until the convergence condition is met to obtain the target semantic expression features of these nodes; the fusion model is iteratively adjusted based on the final loss value, and the target fusion model is obtained when the convergence condition is met; Obtaining a target fusion feature corresponding to the target video to be annotated; the target fusion feature is obtained by inputting a target semantic expression feature corresponding to the video identifier of the target video and a content extraction feature corresponding to the video identifier into a target fusion model; Based on the target fusion feature and the target semantic expression feature corresponding to the first target relationship, obtaining a target prediction label feature corresponding to the first target relationship; Matching the target semantic expression features corresponding to each training label with the target prediction label features respectively, and determining the training label corresponding to the successfully matched target semantic expression feature as the first annotation label corresponding to the target video; Obtaining a target semantic expression feature corresponding to a target tag of the target video; Determine a target tail tag feature based on a target semantic expression feature corresponding to the target tag and a target semantic expression feature corresponding to the second target relationship; Matching the target semantic expression features corresponding to each training tail label with the target tail label feature, respectively, and determining the training tail label corresponding to the successfully matched target semantic expression feature as the second annotation label corresponding to the target video; The first annotation tag and the second annotation tag are used to form a target tag set corresponding to the target video.

2. The method according to claim 1, characterized in that The fusion model includes a mapping layer; the semantic expression feature corresponding to the video identifier and the content extraction feature corresponding to the video identifier are input into the fusion model to obtain the training fusion feature, including: Mapping the content extraction feature to the semantic space where the semantic expression feature of the video identifier is located through the mapping layer to obtain a mapping feature corresponding to the content extraction feature; The mapping feature and the semantic expression feature corresponding to the video identifier are fused to obtain a training fusion feature corresponding to the training video.

3. The method according to claim 2, characterized in that The fusion model also includes an attention distribution layer; the fusion of the mapping features and the semantic expression features corresponding to the video identifier to obtain the training fusion features corresponding to the training video includes: The attention degree distribution layer calculates the attention degrees of the mapping feature and the semantic expression feature of the video identifier respectively, and obtains a first attention degree corresponding to the mapping feature and a second attention degree corresponding to the semantic expression feature of the video identifier; Based on the first attention level, the mapping feature is subjected to attention processing to obtain a content attention feature, and based on the second attention level, the semantic expression feature of the video identifier is subjected to attention processing to obtain an identifier attention feature; The content focus feature and the logo focus feature are concatenated to obtain the training fusion feature.

4. The method according to claim 1, characterized in that: The method further comprises: Obtaining a replacement fusion feature; the replacement fusion feature is a training fusion feature corresponding to other training videos; Determining a second predicted label feature corresponding to the first training relationship based on the replaced fusion feature and the semantic expression feature corresponding to the first training relationship; Determining a second difference based on the semantic expression feature corresponding to the training label and the second predicted label feature; A first target difference between the second difference and the first difference is obtained, and is used to adjust each semantic expression feature and a model parameter of the fusion model.

5. The method according to claim 1, characterized in that The method further comprises: Obtaining replacement label features; the replacement label features are semantic expression features corresponding to training labels corresponding to other training videos; determining a third difference based on the first predicted label feature and the replacement label feature; A second target difference between the first difference and the third difference is obtained, which is used to adjust each semantic expression feature and a model parameter of the fusion model.

6. The method according to claim 1, characterized in that Before inputting the semantic expression feature corresponding to the video identifier and the content extraction feature corresponding to the video identifier into a fusion model to obtain a training fusion feature, the method further includes: Obtaining the associated content corresponding to the video identifier; Inputting the associated content into a content classification model that has been trained and corresponds to the associated content, and extracting content features for classification from the associated content through the content classification model; The extracted content features for classification are determined as content extraction features corresponding to the associated content.

7. The method according to claim 6, characterized in that The associated content includes at least one of a video frame, background music, and associated text corresponding to the training identifier; and different associated contents correspond to different content classification models.

8. The method according to claim 1, characterized in that The method further comprises: Inputting the semantic expression feature corresponding to the first training relationship into a relationship attention distribution model, and calculating the relationship attention corresponding to the first training relationship through the relationship attention distribution model; The determining, based on the training fusion feature and the semantic expression feature corresponding to the first training relationship, a first prediction label feature corresponding to the first training relationship includes: Performing attention processing on the semantic expression feature corresponding to the first training relationship based on the relationship attention degree corresponding to the first training relationship to obtain the relationship attention feature corresponding to the first training relationship; Determining a first predicted label feature corresponding to the first training relationship based on the training fusion feature and the relationship focus feature corresponding to the first training relationship; The first difference is used to adjust each semantic expression feature, the model parameters of the fusion model and the model parameters of the relationship attention allocation model.

9. The method according to claim 1, characterized in that: The method further comprises: Inputting the semantic expression feature corresponding to the second training relationship into a relationship attention allocation model, and calculating the relationship attention corresponding to the second training relationship through the relationship attention allocation model; The step of determining the predicted tail label feature based on the semantic expression feature corresponding to the training head label and the semantic expression feature corresponding to the second training relationship includes: Performing attention processing on the semantic expression feature corresponding to the second training relationship based on the relationship attention degree corresponding to the second training relationship to obtain a relationship attention feature corresponding to the second training relationship; Determining a predicted tail label feature based on a semantic expression feature corresponding to the training head label and a relation attention feature corresponding to the second training relation; The label feature difference is used to adjust the semantic expression features corresponding to the training head label, the training tail label and the second training relationship, as well as the model parameters of the relationship attention allocation model.

10. The method according to claim 1, characterized in that The method further comprises: Obtain a set of videos to be pushed; the set of videos to be pushed includes multiple videos to be pushed, and each of the videos to be pushed has a corresponding first associated tag; Matching the tags in the target tag set with each first associated tag, and determining the to-be-pushed video corresponding to the successfully matched first associated tag as the target pushed video corresponding to the target video; The target pushed video is pushed to a terminal corresponding to a target user corresponding to the target video.

11. The method according to claim 1, characterized in that: The method further comprises: Acquire a candidate user set; the candidate user set includes multiple candidate users, each of which has a corresponding second association tag; Matching the tags in the target tag set with each second associated tag, and determining the associated user corresponding to the successfully matched second associated tag as the target push user corresponding to the target video; The target video is pushed to the terminal corresponding to the target push user.

12. A video annotation device, characterized in that: The device comprises: A training sample acquisition module is used to construct a knowledge graph expressing multiple relationships between videos and labels and labels and labels from the attribute data of the training video set, initialize the semantic expression features of each node in the knowledge graph, and obtain a first training triple from the established knowledge graph, wherein the first training triple includes a video identifier and a training label corresponding to a training video in the training video set, and a first training relationship between the training video and the training label; obtain a second training triple from the established knowledge graph, wherein the second training triple includes a training head label, a training tail label, and a second training relationship between the training head label and the training tail label; A semantic expression feature acquisition module, used to respectively acquire the semantic expression features corresponding to the video identifier, the first training relationship, and the training label; A fusion module is used to input the semantic expression feature corresponding to the video identifier and the content extraction feature corresponding to the video identifier into a fusion model to obtain a training fusion feature; the content extraction feature is obtained by extracting the associated content corresponding to the video identifier; A label prediction module, used to determine a first predicted label feature corresponding to the first training relationship based on the training fusion feature and the semantic expression feature corresponding to the first training relationship; A parameter adjustment module, used to obtain a first difference between the semantic expression feature corresponding to the training label and the first predicted label feature, obtain a first negative training sample corresponding to the first training triple, obtain a second difference corresponding to the first negative training sample, and calculate a first target loss value based on the first difference and the second difference; respectively obtain the semantic expression features corresponding to the training head label, the training tail label, and the second training relationship; determine the predicted tail label feature based on the semantic expression feature corresponding to the training head label and the semantic expression feature corresponding to the second training relationship; obtain a first label feature difference between the predicted tail label feature and the semantic expression feature corresponding to the training tail label, obtain a second negative training sample corresponding to the second training triple, obtain a second label feature difference corresponding to the second negative training sample, and calculate a second target loss value based on the first label feature difference and the second label feature difference; For the first training triplet and the second training triplet in a batch, all the first target loss values ​​and the second target loss values ​​obtained through the above steps are added and averaged to obtain a final loss value, and the semantic expression features of all nodes involved in the batch in the knowledge graph are adjusted by the loss value for iterative adjustment until the convergence condition is met to obtain the target semantic expression features of these nodes; the fusion model is iteratively adjusted based on the final loss value, and the target fusion model is obtained when the convergence condition is met; The labeling module is used to obtain the target fusion feature corresponding to the target video to be labeled; the target fusion feature is obtained by inputting the target semantic expression feature corresponding to the video identifier of the target video and the content extraction feature corresponding to the video identifier into the target fusion model; based on the target fusion feature and the target semantic expression feature corresponding to the first target relationship, the target prediction label feature corresponding to the first target relationship is obtained; the target semantic expression features corresponding to each training label are matched with the target prediction label feature respectively, and the training label corresponding to the successfully matched target semantic expression feature is determined as the first labeling label corresponding to the target video; the target semantic expression feature corresponding to the target label of the target video is obtained; based on the target semantic expression feature corresponding to the target label and the target semantic expression feature corresponding to the second target relationship, the target tail label feature is determined; the target semantic expression features corresponding to each training tail label are matched with the target tail label feature respectively, and the training tail label corresponding to the successfully matched target semantic expression feature is determined as the second labeling label corresponding to the target video; the first labeling label and the second labeling label are used to form a target label set corresponding to the target video.

13. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 11 are implemented.

14. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 11 are implemented.

15. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 11 are implemented.

Images