A video tag generation method, video determination method and device

Abstract

The present disclosure is about a method for generating a video tag, a method for determining a video, and a device, and relates to the field of computer technology. The method includes: obtaining a video to be identified; inputting the video to be identified into an audio-video joint representation model to obtain the audio-video joint features corresponding to the video to be identified, and / or inputting the video to be identified into a text representation model to obtain the text features corresponding to the video to be identified; reading each topic feature included in the feature library to determine the first similarity and / or the second similarity; determining the topic tag corresponding to the topic feature whose first similarity or the second similarity meets the preset conditions as the target topic tag of the video to be identified. The scheme of the present disclosure can fully integrate the data features of each modality included in the video, and can determine a more accurate video tag.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to a method for generating a video tag, a method for determining a video, and a device. Background Art

[0002] Currently, videos can be recommended to users based on video tags. Specifically, some multimodal features (such as sound, text, etc.) included in the video can be identified, and then the identified features are spliced ​​and fused at the final voting layer, and then the relevant video tags are determined based on the spliced ​​features.

[0003] However, in actual application, different modalities will affect each other. Therefore, the accuracy of the video labels determined by the above method is not high, which affects the user experience. Summary of the invention

[0004] The present invention provides a method for generating a video tag, a method for determining a video, and a device, which solve the technical problem that the accuracy of the video tags determined in the prior art is not high, thus affecting the user experience.

[0005] The technical solution of the embodiment of the present disclosure is as follows:

[0006] According to the first aspect of the embodiment of the present disclosure, a method for generating a video tag is provided. The method may include: obtaining a video to be identified; inputting the video to be identified into an audio-video joint representation model to obtain the audio-video joint features corresponding to the video to be identified, and / or inputting the video to be identified into a text representation model to obtain the text features corresponding to the video to be identified, and the audio-video joint features are aligned with the text features; reading each topic feature included in the feature library, determining a first similarity and / or a second similarity, the first similarity being the similarity between each topic feature and the audio-video joint features, the second similarity being the similarity between each topic feature and the text features, and each topic feature included in the feature library corresponds to a topic tag; determining the topic tag corresponding to the topic feature whose first similarity or the second similarity meets the preset conditions as the target topic tag of the video to be identified.

[0007] Optionally, the above-mentioned method for generating video tags also includes: obtaining training data, which includes multiple videos within a historical time period and at least one topic tag corresponding to each of the multiple videos; based on the multiple videos and at least one topic tag corresponding to each of the multiple videos, training the first neural network model, the second neural network model and the third neural network model jointly to obtain the audio and video joint representation model, the text representation model and the topic representation model.

[0008] Optionally, the above-mentioned joint training of the first neural network model, the second neural network model and the third neural network model based on the multiple videos and at least one topic tag corresponding to each of the multiple videos to obtain the audio and video joint representation model, the text representation model and the topic representation model specifically includes: inputting the video frame of each video in the multiple videos and the audio data of each video into the first neural network model, performing feature recognition, and obtaining initial audio and video joint features; inputting each video and the first topic tag into the second neural network model and the third neural network model respectively, performing feature recognition, and obtaining initial text features and initial topic features, the first topic tag is a topic tag corresponding to each video included in the training data, and the initial audio and video joint features, the initial text features and the initial topic features are aligned; based on the initial audio and video joint features, the initial text features and the initial topic features, determining the target loss; based on the target loss, iteratively updating the parameters of the first neural network model, the parameters of the second neural network model and the parameters of the third neural network model to obtain the audio and video joint representation model, the text representation model and the topic representation model.

[0009] Optionally, the above-mentioned determination of the target loss based on the initial audio and video joint features, the initial text features and the initial topic features specifically includes: determining a third similarity and a fourth similarity, the third similarity being the similarity between the initial audio and video joint features corresponding to each video and the initial topic features corresponding to the first topic tag, and the fourth similarity being the similarity between the initial audio and video joint features corresponding to each video and the initial text features corresponding to each video; determining the target loss based on the third similarity and the fourth similarity.

[0010] Optionally, the above-mentioned video tag generation method also includes: obtaining multiple topic tags; inputting the multiple topic tags into a topic representation model to perform feature recognition to obtain topic features corresponding to each of the multiple topic tags; and storing the topic features corresponding to each of the multiple topic tags into the feature library.

[0011] According to the second aspect of the embodiment of the present disclosure, a video determination method is provided. The method may include: obtaining a topic tag to be identified; inputting the topic tag to be identified into a topic representation model, performing feature recognition, and obtaining a topic feature corresponding to the topic tag to be identified; reading each audio and video joint feature and each text feature included in the feature library, and determining the fifth similarity and / or the sixth similarity, the fifth similarity being the similarity between each audio and video joint feature and the topic feature corresponding to the topic tag to be identified, and the sixth similarity being the similarity between each text feature and the topic feature corresponding to the topic tag to be identified; determining the video corresponding to the audio and video joint feature whose fifth similarity meets the preset condition, or the video corresponding to the text feature whose sixth similarity meets the preset condition, as the target video of the topic tag to be identified.

[0012] According to the third aspect of the embodiment of the present disclosure, a device for generating video tags is provided. The device may include: an acquisition module, a processing module and a determination module; the acquisition module is configured to acquire a video to be identified; the processing module is configured to input the video to be identified into an audio-video joint representation model to obtain the audio-video joint features corresponding to the video to be identified, and / or input the video to be identified into a text representation model to obtain the text features corresponding to the video to be identified, and the audio-video joint features are aligned with the text features; the determination module is configured to read each topic feature included in the feature library, determine the first similarity and / or the second similarity, the first similarity is the similarity between each topic feature and the audio-video joint features, the second similarity is the similarity between each topic feature and the text feature, and each topic feature included in the feature library corresponds to a topic tag; the determination module is also configured to determine the topic tag corresponding to the topic feature whose first similarity or the second similarity meets the preset conditions as the target topic tag of the video to be identified.

[0013] Optionally, the acquisition module is further configured to acquire training data, which includes multiple videos within a historical time period and at least one topic tag corresponding to each of the multiple videos; the processing module is further configured to jointly train the first neural network model, the second neural network model and the third neural network model based on the multiple videos and at least one topic tag corresponding to each of the multiple videos to obtain the audio and video joint representation model, the text representation model and the topic representation model.

[0014] Optionally, the processing module is specifically configured to input the video frames of each video in the multiple videos and the audio data of each video into the first neural network model, perform feature recognition, and obtain initial audio and video joint features; the processing module is also specifically configured to input each video and the first topic tag into the second neural network model and the third neural network model respectively, perform feature recognition, and obtain initial text features and initial topic features, the first topic tag is a topic tag corresponding to each video included in the training data, and the initial audio and video joint features, the initial text features, and the initial topic features are aligned; the determination module is specifically configured to determine the target loss based on the initial audio and video joint features, the initial text features, and the initial topic features; the processing module is also specifically configured to iteratively update the parameters of the first neural network model, the parameters of the second neural network model, and the parameters of the third neural network model based on the target loss to obtain the audio and video joint representation model, the text representation model, and the topic representation model.

[0015] Optionally, the determination module is further configured to determine a third similarity and a fourth similarity, the third similarity being the similarity between the initial audio and video joint features corresponding to each video and the initial topic features corresponding to the first topic tag, and the fourth similarity being the similarity between the initial audio and video joint features corresponding to each video and the initial text features corresponding to each video; the determination module is further configured to determine the target loss based on the third similarity and the fourth similarity.

[0016] Optionally, the video tag generation device also includes a storage module; the acquisition module is also configured to acquire multiple topic tags; the processing module is also configured to input the multiple topic tags into a topic representation model, perform feature recognition, and obtain topic features corresponding to each of the multiple topic tags; the storage module is configured to store the topic features corresponding to each of the multiple topic tags into the feature library.

[0017] According to the fourth aspect of the embodiment of the present disclosure, a video determination device is provided. The device may include: an acquisition module, a processing module and a determination module; the acquisition module is configured to acquire a topic tag to be identified; the processing module is configured to input the topic tag to be identified into a topic representation model, perform feature recognition, and obtain a topic feature corresponding to the topic tag to be identified; the determination module is configured to read each audio and video joint feature and each text feature included in the feature library, and determine the fifth similarity and / or the sixth similarity, the fifth similarity being the similarity between each audio and video joint feature and the topic feature corresponding to the topic tag to be identified, and the sixth similarity being the similarity between each text feature and the topic feature corresponding to the topic tag to be identified; the determination module is also configured to determine the video corresponding to the audio and video joint feature whose fifth similarity meets the preset condition, or the video corresponding to the text feature whose sixth similarity meets the preset condition, as the target video of the topic tag to be identified.

[0018] According to the fifth aspect of an embodiment of the present disclosure, an electronic device is provided, which may include: a processor and a memory configured to store processor executable instructions; wherein the processor is configured to execute the instructions to implement any one of the optional video tag generation methods in the above-mentioned first aspect, or to implement any one of the optional video determination methods in the above-mentioned second aspect.

[0019] According to a sixth aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided, on which instructions are stored. When the instructions in the computer-readable storage medium are executed by an electronic device, the electronic device is enabled to execute any one of the optional methods for generating video tags in the above-mentioned first aspect, or execute any one of the optional methods for determining video in the above-mentioned second aspect.

[0020] According to a seventh aspect of an embodiment of the present disclosure, a computer program product is provided, which includes computer instructions. When the computer instructions are executed on an electronic device, the electronic device executes any optional method for generating a video tag as described in the first aspect, or executes any optional method for determining a video as described in the second aspect.

[0021] The technical solution provided by the embodiments of the present disclosure brings at least the following beneficial effects:

[0022] Based on any of the above aspects, in the present disclosure, an electronic device can obtain a video to be identified, and input the video to be identified into an audio-video joint representation model to obtain the audio-video joint features corresponding to the video to be identified, and / or input the video to be identified into a text representation model to obtain the text features corresponding to the video to be identified; then the electronic device reads each topic feature included in the feature library, determines the first similarity and / or the second similarity, the first similarity is the similarity between each topic feature and the audio-video joint feature, and the second similarity is the similarity between each topic feature and the text feature; and the electronic device can determine the topic tag corresponding to the topic feature whose first similarity or the second similarity meets the preset conditions as the target topic tag of the video to be identified. In the embodiment of the present disclosure, since the audio-video joint features corresponding to the video to be identified obtained by the audio-video joint representation model and the text features corresponding to the video to be identified obtained by the text representation model are aligned, that is, they have been aligned to a unified space, and each topic feature included in the feature library corresponds to a topic tag. In this way, the target topic tag of the video to be identified is determined based on the audio-video joint features, the text features and each topic feature. It can fully integrate the data features of each modality included in the video and determine more accurate video labels.

[0023] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description are used to explain the principles of the present disclosure, and do not constitute improper limitations on the present disclosure.

[0025] Figure 1 A schematic diagram of a process for generating a video tag provided by an embodiment of the present disclosure is shown;

[0026] Figure 2 A schematic diagram showing a flow chart of another method for generating a video tag provided by an embodiment of the present disclosure;

[0027] Figure 3 A training schematic diagram of a neural network model provided by an embodiment of the present disclosure is shown;

[0028] Figure 4 A schematic diagram showing a flow chart of another method for generating a video tag provided by an embodiment of the present disclosure;

[0029] Figure 5 A schematic diagram showing a flow chart of another method for generating a video tag provided by an embodiment of the present disclosure;

[0030] Figure 6 A schematic diagram showing a flow chart of another method for generating a video tag provided by an embodiment of the present disclosure;

[0031] Figure 7 A schematic diagram showing a flow chart of another method for generating a video tag provided by an embodiment of the present disclosure;

[0032] Figure 8 A schematic diagram of a process of a video determination method provided by an embodiment of the present disclosure is shown;

[0033] Fig. 9 A schematic diagram of the structure of a device for generating video tags provided by an embodiment of the present disclosure is shown;

[0034] Fig.10 A schematic diagram of the structure of another device for generating video tags provided by an embodiment of the present disclosure is shown;

[0035] Fig.11 A schematic structural diagram of a video determination device provided by an embodiment of the present disclosure is shown;

[0036] Fig.12 A schematic structural diagram of another video determination device provided in an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0037] In order to enable ordinary persons in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings.

[0038] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0039] It should also be understood that the term “comprising” indicates the presence of described features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements and / or components.

[0040] The data involved in the present disclosure may be data authorized by the user or fully authorized by all parties.

[0041] As described in the background technology, the accuracy of video tags determined by the prior art is not high, which affects the user experience. Based on this, the embodiments of the present disclosure provide a method for generating video tags, a method for determining videos, and a device, which can fully integrate the data features of each modality included in the video and determine more accurate video tags.

[0042] The video tag generation method, video determination method and device provided by the embodiments of the present disclosure are applied to video tag recommendation (and / or video recommendation) scenarios. When a video to be identified (or a topic tag to be identified) is obtained, the target topic tag of the video to be identified (or the target video of the topic tag to be identified) can be determined according to the method provided by the embodiments of the present disclosure.

[0043] The following is an exemplary description of the video tag generation method and the video determination method provided by the embodiments of the present disclosure in conjunction with the accompanying drawings:

[0044] It is understandable that the electronic device that performs the video tag generation method and video determination method provided by the embodiments of the present disclosure may be a mobile phone, tablet computer, desktop, laptop, handheld computer, notebook computer, ultra-mobile personal computer (UMPC), netbook, as well as cellular phone, personal digital assistant (PDA), augmented reality (AR)\virtual reality (VR) equipment and other equipment that can install and use content community applications, and the present disclosure does not impose any special restrictions on the specific form of the electronic device. It can interact with the user through one or more methods such as keyboard, touchpad, touch screen, remote control, voice interaction or handwriting device.

[0045] The following is an explanation by taking the application of the video tag generation method and the video determination method provided in the embodiments of the present disclosure to an electronic device as an example.

[0046] like Figure 1 As shown, the method for generating a video tag provided by the embodiment of the present disclosure may include S101-S104.

[0047] S101: Obtain a video to be identified.

[0048] S102: Input the video to be identified into an audio-video joint representation model to obtain audio-video joint features corresponding to the video to be identified, and / or input the video to be identified into a text representation model to obtain text features corresponding to the video to be identified.

[0049] Among them, the audio and video joint feature is aligned with the text feature.

[0050] Specifically, the electronic device inputs the video to be identified into the audio and video joint representation model to perform feature recognition to obtain the audio and video joint feature; the electronic device can also input the video to be identified into the text representation model to perform feature recognition to obtain the text feature.

[0051] It can be understood that text features represent text, which are some spaces in the text dimension, and audio and video joint features represent pictures and audio, which are an embedding in the network space. The original input forms of the two are different. In the disclosed embodiment, some fully connected networks can be added between the audio and video joint features and the text features, so that the audio and video joint features and the text features can be projected into the same space together (it can also be understood as putting the audio and video joint features and the text features into the same network, and letting the network fit a space in which they can coexist). In this way, the audio and video joint features and the text features can be aligned to a unified space, that is, the audio and video joint features are aligned with the text features.

[0052] S103: Read each topic feature included in the feature library, and determine the first similarity and / or the second similarity.

[0053] The first similarity is the similarity between each topic feature and the audio-video joint feature, the second similarity is the similarity between each topic feature and the text feature, and each topic feature included in the feature library corresponds to a topic tag.

[0054] In combination with the description of the above embodiment, it should be understood that the audio and video joint feature is the audio and video joint feature corresponding to the above video to be identified, and the text feature is the text feature corresponding to the video to be identified.

[0055] It can be understood that the feature library is a database that can store features (specifically, it can store topic features). When the electronic device obtains the audio and video joint features corresponding to the video to be identified and / or the text features corresponding to the video to be identified, it can determine the first similarity and / or determine the second similarity based on each topic feature included in the feature library.

[0056] Optionally, the feature library may be a Faiss library, that is, a database with a corresponding retrieval function.

[0057] S104: Determine the topic tag corresponding to the topic feature whose first similarity or second similarity meets the preset condition as the target topic tag of the video to be identified.

[0058] In one implementation of the embodiment of the present disclosure, the preset condition includes being greater than or equal to a similarity threshold, that is, when a certain similarity (for example, the first similarity) is greater than or equal to the similarity threshold, the topic tag corresponding to the first similarity (specifically, the topic tag corresponding to the topic feature corresponding to the first similarity) can be determined as the target topic tag of the video to be identified, and the number of the target topic tags can be one or more.

[0059] In another implementation of the embodiment of the present disclosure, the electronic device can also select (or determine) the M similarities with the greatest similarities from the determined first similarities and second similarities, and determine the topic tags corresponding to the M similarities (specifically, the topic tags corresponding to the corresponding topic features) as the target topic tags of the video to be identified.

[0060] The technical solution provided by the above embodiment can at least bring the following beneficial effects: From S101-S104, it can be known that the electronic device can obtain the video to be identified, and input the video to be identified into the audio and video joint representation model to obtain the audio and video joint features corresponding to the video to be identified, and / or input the video to be identified into the text representation model to obtain the text features corresponding to the video to be identified; then the electronic device reads each topic feature included in the feature library, determines the first similarity and / or the second similarity, the first similarity is the similarity between each topic feature and the audio and video joint feature, and the second similarity is the similarity between each topic feature and the text feature; and the electronic device can determine the topic tag corresponding to the topic feature whose first similarity or the second similarity meets the preset conditions as the target topic tag of the video to be identified. In the disclosed embodiment, since the audio and video joint features corresponding to the video to be identified obtained by the audio and video joint representation model and the text features corresponding to the video to be identified obtained by the text representation model are aligned, that is, they have been aligned to a unified space, and each topic feature included in the feature library corresponds to a topic tag. In this way, the target topic tag of the video to be identified is determined based on the audio and video joint feature, the text feature and each topic feature. The data features of each modality included in the video can be fully integrated, and a more accurate video tag can be determined.

[0061] Combination Figure 1 ,like Figure 2 As shown, the method for generating a video tag provided by the embodiment of the present disclosure may further include S105-S106.

[0062] S105: Obtain training data.

[0063] The training data includes multiple videos in a historical time period and at least one topic tag corresponding to each of the multiple videos.

[0064] Specifically, the electronic device can crawl videos and topic tags corresponding to the videos from websites such as Weibo and forums. In the disclosed embodiment, the electronic device can also obtain videos and topic tags corresponding to the videos from content community applications (such as Kuaishou).

[0065] Optionally, the training data may be data obtained after a data cleaning operation, that is, the electronic device may perform data cleaning on the acquired videos and the topic tags corresponding to the videos.

[0066] It should be understood that a video may include video frames (or video data) of the video, audio data of the video, and text data of the video. The text data of a video may be the title of the video and / or the letters of the video, etc.

[0067] Specifically, the electronic device can perform some operations of de-lengthening and / or de-shorting on the text data of a certain video and the topic tags (also understood as topic texts) corresponding to the video, specifically deleting longer text data (and topic tags) and / or deleting shorter text data (and topic tags). Optionally, the electronic device can also delete special characters in the text data and topic tags (such as garbled characters or emoticons, characters that will produce noise for semantic understanding, etc.).

[0068] For video frames of a video and audio data of the video, the electronic device can delete video frames with poor video quality (such as aesthetics, clarity, etc.), and can also delete some video frames and audio data that are too long.

[0069] S106. Jointly train the first neural network model, the second neural network model, and the third neural network model based on the multiple videos and at least one topic tag of each of the multiple videos to obtain an audio and video joint representation model, a text representation model, and a topic representation model.

[0070] It can be understood that, for a certain video among the multiple videos, the audio and video joint features corresponding to the video, the text features corresponding to the video, and at least one topic feature corresponding to the video (i.e., the topic feature corresponding to at least one topic tag corresponding to the video) are aligned.

[0071] It should be noted that the embodiments of the present disclosure do not limit the execution order of the above S101 and S106. For example, S101 may be executed first and then S106, that is, the video to be identified is obtained first, and then the audio and video joint representation model, text representation model and topic representation model are obtained, or S106 may be executed first and then S101, that is, the audio and video joint representation model, text representation model and topic representation model are obtained first, and then the video to be identified is obtained, or S101 and S106 may be executed at the same time. For the convenience of explanation, Figure 2In the example, S106 is executed first and then S101 is executed.

[0072] The technical solution provided by the above embodiment can at least bring the following beneficial effects: As can be seen from S105-S106, the electronic device can obtain training data, and based on the multiple videos in the historical time period included in the training data and at least one topic tag corresponding to each of the multiple videos, jointly train the first neural network model, the second neural network model and the third neural network model to obtain an audio-video joint representation model, a text representation model and a topic representation model. Since the training data includes the multiple videos and at least one topic tag corresponding to each of the multiple videos, the audio-video joint representation model, the text representation model and the topic representation model obtained after the training are neural network models after data interaction based on the video and the relevant topic tags. It can accurately reflect the relationship between the video and the topic tags corresponding to the video, and can obtain a neural network model with higher accuracy.

[0073] Combination Figure 2 ,like Figure 3 As shown, in one implementation of the embodiment of the present disclosure, the first neural network model, the second neural network model and the third neural network model are jointly trained based on multiple videos and at least one topic tag corresponding to each of the multiple videos to obtain an audio and video joint representation model, a text representation model and a topic representation model, which may specifically include S1061-S1064.

[0074] S1061. Input the video frames of each video and the audio data of each video in the multiple videos into the first neural network model to perform feature recognition to obtain initial audio and video joint features.

[0075] In combination with the description of the above embodiment, it should be understood that the multiple videos are multiple videos within the historical time period included in the above training data, and one video may include the video frame of the video and the audio data of the video.

[0076] Optionally, the first neural network model may be a network model composed of a convolutional neural network (CNN) and an attention network (i.e., CNN & attention). In the disclosed embodiment, the electronic device may input the video frame of each video in the multiple videos and the audio data of each video into the CNN & attention network model to obtain the initial audio and video joint features.

[0077] S1062. Input each video and the first topic tag into the second neural network model and the third neural network model respectively to perform feature recognition to obtain initial text features and initial topic features.

[0078] Among them, the first topic label is a topic label corresponding to each video included in the above training data, and the initial audio-visual joint feature, the initial text feature, and the initial topic feature are aligned.

[0079] Optionally, both the second neural network model and the third neural network model can be transformer network models. That is, the electronic device can input each video and the first topic label into the transformer network model respectively to obtain the initial text feature and the initial topic feature.

[0080] S1063. Determine the target loss based on the initial audio-visual joint feature, the initial text feature, and the initial topic feature.

[0081] S1064. Iteratively update the parameters of the first neural network model, the parameters of the second neural network model, and the parameters of the third neural network model based on the target loss to obtain an audio-visual joint representation model, a text representation model, and a topic representation model.

[0082] Exemplarily, as Figure 4 shown, the electronic device can input the video frame 101 of a certain video and the audio data 102 of the video into the CNN&attention network model (that is, input the video frame and the audio data of the video into the first neural network model) to obtain the initial audio-visual joint feature (Z3), and can also input a certain topic label 103 (i.e., #Food探店) corresponding to the video into the transformer network model (that is, input the above first topic label into the third neural network model) to obtain the initial topic feature (Z1), and can also input the video (specifically the text data 104, that is, the big sister of grilled cold noodles in Yinan!) into the transformer network model (that is, input the video into the second neural network model) to obtain the initial text feature (Z2).

[0083] The technical solution provided by the above embodiment can at least bring the following beneficial effects: From S1061-S1064, it can be known that the electronic device can input the video frame of each video in the multiple videos and the audio data of each video into the first neural network model, perform feature recognition, and obtain the initial audio and video joint feature; and input each video and the first topic tag (i.e., a topic tag corresponding to each video) into the second neural network model and the third neural network model respectively, perform feature recognition, and obtain the initial text feature and the initial topic feature; then, the electronic device can determine the target loss based on the initial audio and video joint feature, the initial text feature, and the initial topic feature, and iteratively update the parameters of the first neural network model, the parameters of the second neural network model, and the parameters of the third neural network model based on the target loss to obtain the audio and video joint representation model, the text representation model, and the topic representation model. In the disclosed embodiment, the electronic device can input the multiple types of data included in the training data into different neural network models, and then determine the target loss based on the features of the multiple types of data obtained, and iteratively update the parameters of the different neural network models based on the target loss to obtain the audio and video joint representation model, the text representation model, and the topic representation model. The neural network model can be trained effectively and reasonably, and the corresponding representation model can be obtained, thereby improving the training efficiency of the neural network model.

[0084] Combination Figure 3 ,like Figure 5 As shown, in one implementation of the embodiment of the present disclosure, determining the target loss based on the initial audio joint features, the initial text features and the initial topic features may specifically include: S1063a-S1063b.

[0085] S1063a: Determine a third similarity and a fourth similarity.

[0086] Among them, the third similarity is the similarity between the initial audio and video joint features corresponding to each of the above videos and the initial topic features corresponding to the above first topic tag, and the fourth similarity is the similarity between the initial audio and video joint features corresponding to each video and the initial text features corresponding to each video.

[0087] S1063b. Determine a target loss based on the third similarity and the fourth similarity.

[0088] Specifically, the electronic device can determine that the above loss satisfies the following formula:

[0089] L=1-sim1

[0090] Wherein, L represents the target loss, and sim1 represents the third similarity or the fourth similarity.

[0091] In another implementation of the embodiment of the present disclosure, the electronic device may also determine the target loss based on the third similarity, the fourth similarity, the similarity between the initial audio and video joint features corresponding to each video and the initial topic features corresponding to other topic tags (hereinafter referred to as the seventh similarity), and the similarity between the initial audio and video joint features corresponding to each video and the initial text features corresponding to other videos (hereinafter referred to as the eighth similarity), the other video is a video other than the above-mentioned multiple videos, and the other topic tag is a topic tag corresponding to the other video.

[0092] Specifically, the electronic device may determine that the loss satisfies the following formula:

[0093] L=(1-sim1)+(1-sim2)+sim3+sim4

[0094] Wherein, L represents the target loss, sim1 represents the third similarity, sim2 represents the fourth similarity, sim3 represents the seventh similarity, and sim4 represents the eighth similarity.

[0095] It should be noted that, in the disclosed embodiment, the electronic device should increase the similarity between features with the same data source (i.e., belonging to the same video) and reduce the similarity between features with different data sources. That is, a first similarity threshold and a second similarity threshold can be configured, the similarity between features with the same data source should be greater than the first similarity threshold, the similarity between features with different data sources should be less than the second similarity threshold, and the first similarity threshold is greater than the second similarity threshold.

[0096] Specifically, the third similarity and the fourth similarity are greater than the first similarity threshold; and the seventh similarity and the eighth similarity are less than the second similarity threshold.

[0097] The technical solution provided by the above embodiment can at least bring the following beneficial effects: From S1063a-S1063b, it can be known that the electronic device can determine the third similarity and the fourth similarity, and the third similarity and the fourth similarity determine the target loss. The loss of the neural network model can be accurately determined, and then the parameters in the neural network model can be accurately updated to obtain a neural network model with higher accuracy.

[0098] Combination Figure 2 ,like Figure 6 As shown, after the training data is obtained as described above, the method for generating video tags provided by the embodiment of the present disclosure may further include S107-S108.

[0099] S107: Perform a data enhancement operation on the training data to obtain data-enhanced training data.

[0100] In the disclosed embodiment, the electronic device may at least perform data enhancement on the above-mentioned video frames and text data.

[0101] Specifically, for a certain video, the electronic device can perform frame extraction processing on the video based on different frame extraction strategies (such as uniform frame extraction, fixed frame number frame extraction, and frame extraction based on the diff between frames, etc.), so that multiple video frames (or image data) corresponding to the video can be obtained, including the cover image of the video. For each of the multiple video frames, operations such as rotation, scale transformation, noise perturbation, and color transformation can be performed on each video frame to obtain more video frames (or image data).

[0102] For the text data of the video, operations such as synonym replacement, translation or inversion of sentences can be used to obtain more text data.

[0103] S108. Jointly train the first neural network model, the second neural network model, and the third neural network model based on the training data after data enhancement to obtain an audio and video joint representation model, a text representation model, and a topic representation model.

[0104] The technical solution provided by the above embodiment can at least bring the following beneficial effects: From S107-S108, it can be known that the electronic device can perform data enhancement operation on the training data to obtain the training data after data enhancement, and train the first neural network model, the second neural network model and the third neural network model based on the training data after data enhancement to obtain the audio and video joint representation model, the text representation model and the topic representation model. In the embodiment of the present disclosure, since more training data is used in the training process of the neural network model after data enhancement, covering as many video frames and text data as possible, the robustness of the neural network model can be improved.

[0105] Combination Figure 1 ,like Figure 7 As shown, in one implementation of the embodiment of the present disclosure, the above-mentioned method for generating video tags may further include S109-S111.

[0106] S109. Obtain multiple topic tags.

[0107] In combination with the description of the above embodiments, it should be understood that the electronic device can crawl (or obtain) the multiple topic tags from microblogs, forums, content community applications, etc.

[0108] S110: Input multiple topic tags into a topic representation model to perform feature recognition to obtain topic features corresponding to each of the multiple topic tags.

[0109] S111. Storing topic features corresponding to multiple topic tags in a feature library.

[0110] In combination with the description of the above embodiments, it should be understood that the electronic device stores the topic features corresponding to each of the multiple topic tags (hereinafter referred to as multiple topic features) into a feature library, so that when the electronic device obtains a certain video (for example, a video to be identified), it can determine the target topic tag of the video to be identified based on the similarity between the multiple topic features and the features of the video to be identified (specifically the audio and video joint features corresponding to the video to be identified, and / or the text features corresponding to the video to be identified).

[0111] In one implementation of the embodiment of the present disclosure, the multiple topic features may include each topic feature in the above-mentioned S103, that is, the electronic device may, after obtaining the above-mentioned topic representation model, input each of the above-mentioned topic tags into the topic representation model, perform feature recognition, and obtain the topic feature corresponding to each topic tag, and then determine the target topic tag of the video to be identified based on the topic feature.

[0112] In another implementation of the disclosed embodiment, the multiple topic features may also include other topic features (i.e., more topic features in addition to each of the above-mentioned topic features). In this way, the electronic device can further determine the target topic tags for the video to be identified based on more topic features (specifically, topic features corresponding to more topic tags), thereby improving the diversity of topic tags and topic features, and recommending more types of topic tags with higher relevance to users.

[0113] The technical solution provided by the above embodiment can at least bring the following beneficial effects: As can be seen from S109-S111, the electronic device can obtain multiple topic tags, and then input the multiple topic tags into the topic representation model, perform feature recognition, obtain the topic tags corresponding to each of the multiple topic tags, and then store the topic features corresponding to each of the multiple topic tags into the feature library. In the disclosed embodiment, the electronic device can input each of the multiple topic tags into the topic representation model to obtain the topic features corresponding to each of the topic tags, and store each of the topic features in the feature library for use in determining the target topic tag for a certain video (e.g., a video to be identified), which can improve the efficiency of determining the topic tag. In another case, the electronic device can also input other topic tags (i.e., new and more topic tags) into the topic representation model, and store the obtained new and more topic features into the feature library, which can support the determination (or detection) of more types of topic tags for a certain video (e.g., a video to be identified) and whose similarity meets the preset conditions, thereby improving the robustness of the feature library.

[0114] like Figure 8 As shown, the video determination method provided by the embodiment of the present disclosure may include S201-S204.

[0115] S201, obtaining a topic tag to be identified.

[0116] S202: Input the topic tag to be identified into the topic representation model to perform feature recognition to obtain the topic feature corresponding to the topic tag to be identified.

[0117] S203: Read each audio and video joint feature and each text feature included in the feature library to determine the fifth similarity and / or the sixth similarity.

[0118] Among them, the fifth similarity is the similarity between each audio and video joint feature and the topic feature corresponding to the topic label to be identified, and the sixth similarity is the similarity between each text feature and the topic feature corresponding to the topic label to be identified.

[0119] S204: Determine the video corresponding to the audio-video joint feature whose fifth similarity meets the preset condition, or the video corresponding to the text feature whose sixth similarity meets the preset condition, as the target video of the topic tag to be identified.

[0120] It should be noted that the explanations of the above S201-S204 are the same or similar to the descriptions of the above S101-S104, and will not be repeated here.

[0121] In combination with the description of the above embodiments, it should be understood that, for a certain video, the electronic device can input the video into the audio-video joint model and / or the text representation model, perform feature recognition, and obtain the features of the video (i.e., the audio-video joint features corresponding to the video and / or the text features corresponding to the video). For a certain topic tag, the electronic device can also input the topic tag into the topic representation model, perform feature recognition, and obtain the topic features corresponding to the topic tag.

[0122] Furthermore, when the feature library includes both topic features and video features (i.e., audio and video joint features corresponding to the video and / or text features corresponding to the video), the electronic device can determine the similarity between the features of a certain video (e.g., the video to be identified) and each topic feature included in the feature library, and can also determine the similarity between the topic features corresponding to a certain topic tag (e.g., the topic tag to be identified) and the features of each video included in the feature library. The target topic tag of the video to be identified and the target video of the topic tag to be identified are determined respectively, and then relevant topic tag recommendations and video recommendations are made respectively.

[0123] It is understandable that the above feature library can support mutual retrieval between topic tags (specifically topic features) and video features (i.e., audio and video joint features corresponding to the video and / or text features corresponding to the video). When a type of feature (e.g., topic feature) is acquired (or obtained), the corresponding target video can be determined for the topic tag corresponding to the topic feature based on other types of features included in the feature library (e.g., audio and video joint features corresponding to the video and / or text features corresponding to the video).

[0124] The technical solution provided by the above embodiment can at least bring the following beneficial effects: As can be seen from S201-S204, the electronic device can obtain the topic tag to be identified, and input the topic tag to be identified into the topic representation model to perform feature recognition to obtain the topic feature corresponding to the topic tag to be identified; then the electronic device reads each audio and video joint feature and each text feature included in the feature library, and determines the fifth similarity and / or the sixth similarity, the fifth similarity being the similarity between each audio and video joint feature and the topic feature corresponding to the topic tag to be identified, and the sixth similarity being the similarity between each text feature and the topic feature corresponding to the topic tag to be identified; then the electronic device determines the video corresponding to the audio and video joint feature whose fifth similarity meets the preset condition, or the video corresponding to the text feature whose sixth similarity meets the preset condition, as the target video of the topic to be identified. In the disclosed embodiment, since each audio and video joint feature obtained through the audio and video joint representation model is aligned with each text feature obtained through the text representation model, and the topic features corresponding to the topic tags to be identified obtained through the topic representation model, the features of the data included in each mode of the video (i.e., video frames, audio data, and text data, etc.) can be fused, and compared with the topic features for training, a network model with higher accuracy is obtained. Then, the similarity between the topic features obtained based on the topic representation model and the features of each video included in the feature library (specifically, each audio and video joint feature and / or each text feature) can be determined, and then the corresponding target video can be determined for the topic feature (specifically, the topic tag), which can more accurately determine the video corresponding to the topic tag and improve the effectiveness of video recommendation.

[0125] It is understandable that, in actual implementation, the electronic device described in the embodiment of the present disclosure may include one or more hardware structures and / or software modules for implementing the aforementioned corresponding video tag generation method and video determination method, and these execution hardware structures and / or software modules may constitute an electronic device. It should be easily appreciated by those skilled in the art that, in combination with the algorithm steps of each example described in the embodiments disclosed herein, the present disclosure can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Professional and technical personnel may use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the present disclosure.

[0126] Based on this understanding, the embodiment of the present disclosure also provides a device for generating a video tag. Fig. 9 FIG. 1 shows a schematic diagram of the structure of a device for generating video tags provided by an embodiment of the present disclosure. Fig. 9As shown, the video tag generation device 20 may include: an acquisition module 201 , a processing module 202 and a determination module 203 .

[0127] The acquisition module 201 is configured to acquire a video to be identified.

[0128] The processing module 202 is configured to input the video to be identified into an audio-video joint representation model to obtain an audio-video joint feature corresponding to the video to be identified, and / or input the video to be identified into a text representation model to obtain a text feature corresponding to the video to be identified, and the audio-video joint feature is aligned with the text feature.

[0129] The determination module 203 is configured to read each topic feature included in the feature library and determine a first similarity and / or a second similarity, wherein the first similarity is the similarity between each topic feature and the audio and video joint feature, and the second similarity is the similarity between each topic feature and the text feature. Each topic feature included in the feature library corresponds to a topic tag.

[0130] The determination module 203 is further configured to determine the topic tag corresponding to the topic feature whose first similarity or the second similarity meets a preset condition as the target topic tag of the video to be identified.

[0131] Optionally, the acquisition module 201 is further configured to acquire training data, where the training data includes a plurality of videos within a historical time period and at least one topic tag corresponding to each of the plurality of videos.

[0132] The processing module 202 is also configured to jointly train the first neural network model, the second neural network model and the third neural network model based on the multiple videos and at least one topic tag corresponding to each of the multiple videos to obtain the audio and video joint representation model, the text representation model and the topic representation model.

[0133] Optionally, the processing module 202 is specifically configured to input the video frames of each video in the multiple videos and the audio data of each video into the first neural network model to perform feature recognition to obtain initial audio and video joint features.

[0134] The processing module 202 is further specifically configured to input each video and the first topic tag into the second neural network model and the third neural network model respectively, perform feature recognition, and obtain initial text features and initial topic features. The first topic tag is a topic tag corresponding to each video included in the training data, and the initial audio and video joint features, the initial text features, and the initial topic features are aligned.

[0135] The determination module 203 is specifically configured to determine the target loss based on the initial audio and video joint feature, the initial text feature and the initial topic feature.

[0136] The processing module 202 is also specifically configured to iteratively update the parameters of the first neural network model, the parameters of the second neural network model and the eucalyptus of the third neural network model based on the target loss to obtain the audio and video joint representation model, the text representation model and the topic representation model.

[0137] Optionally, the determination module 203 is further specifically configured to determine a third similarity and a fourth similarity, the third similarity being the similarity between the initial audio and video joint features corresponding to each video and the initial topic features corresponding to the first topic tag, and the fourth similarity being the similarity between the initial audio and video joint features corresponding to each video and the initial text features corresponding to each video.

[0138] The determination module 203 is further configured to determine the target loss based on the third similarity and the fourth similarity.

[0139] Optionally, the video tag generation device 20 further includes a storage module 204 .

[0140] The acquisition module 201 is further configured to acquire a plurality of topic tags.

[0141] The processing module 202 is further configured to input the multiple topic tags into a topic representation model to perform feature recognition to obtain topic features corresponding to each of the multiple topic tags.

[0142] The storage module 204 is configured to store the topic features corresponding to each of the plurality of topic tags into the feature library.

[0143] As described above, the embodiments of the present disclosure can divide the functional modules of the device for generating video tags according to the above method examples. Among them, the above integrated modules can be implemented in the form of hardware or in the form of software functional modules. In addition, it should be noted that the division of modules in the embodiments of the present disclosure is schematic and is only a logical functional division. There may be other division methods in actual implementation. For example, each functional module can be divided according to each function, or two or more functions can be integrated into one processing module.

[0144] Regarding the device for generating video tags in the above embodiment, the specific manner in which each module performs operations and the beneficial effects thereof have been described in detail in the above method embodiment and will not be repeated here.

[0145] Fig.10FIG. 1 is a schematic diagram of the structure of another apparatus for generating video tags provided by the present disclosure. Fig.10 The video tag generation device 30 may include at least one processor 301 and a memory 303 for storing processor executable instructions. The processor 301 is configured to execute the instructions in the memory 303 to implement the video tag generation method in the above embodiment.

[0146] In addition, the video tag generating device 30 may further include a communication bus 302 and at least one communication interface 304 .

[0147] The processor 301 may be a central processing unit (CPU), a microprocessing unit, an ASIC, or one or more integrated circuits for controlling the execution of the program of the disclosed solution.

[0148] Communication bus 302 may include a pathway for transmitting information between the above-mentioned components.

[0149] The communication interface 304 uses any transceiver or other device for communicating with other devices or communication networks, such as Ethernet, radio access network (RAN), wireless local area networks (WLAN), etc.

[0150] The memory 303 may be a read-only memory (ROM) or other types of static storage devices that can store static information and instructions, a random access memory (RAM) or other types of dynamic storage devices that can store information and instructions, or an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical disc, laser disc, optical disc, digital versatile disc, Blu-ray disc, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store the desired program code in the form of an instruction or data structure and can be accessed by a computer, but is not limited thereto. The memory may exist independently and be connected to the processing unit through a bus. The memory may also be integrated with the processing unit.

[0151] The memory 303 is used to store instructions for executing the solution of the present disclosure, and the execution is controlled by the processor 301. The processor 301 is used to execute the instructions stored in the memory 303, so as to realize the functions in the method of the present disclosure.

[0152] In a specific implementation, as an embodiment, the processor 301 may include one or more CPUs, such as Fig.10 CPU0 and CPU1 in.

[0153] In a specific implementation, as an embodiment, the video tag generation device 30 may include multiple processors, such as Fig.10 301 and processor 307 in the embodiment of the present invention. Each of these processors may be a single-CPU processor or a multi-CPU processor. The processor herein may refer to one or more devices, circuits, and / or processing cores for processing data (e.g., computer program instructions).

[0154] In a specific implementation, as an embodiment, the video tag generation device 30 may also include an output device 305 and an input device 306. The output device 305 communicates with the processor 301 and can display information in a variety of ways. For example, the output device 305 may be a liquid crystal display (LCD), a light emitting diode (LED) display device, a cathode ray tube (CRT) display device, or a projector. The input device 306 communicates with the processor 301 and can accept user input in a variety of ways. For example, the input device 306 may be a mouse, a keyboard, a touch screen device, or a sensor device.

[0155] Fig.11 FIG. 1 is a structural example diagram of a video determination device provided by the present disclosure. Fig.11 As shown, the video determination device 40 may include: an acquisition module 401 , a processing module 402 and a determination module 403 .

[0156] The acquisition module 401 is configured to acquire a topic tag to be identified.

[0157] The processing module 402 is configured to input the topic tag to be identified into a topic representation model to perform feature recognition to obtain a topic feature corresponding to the topic tag to be identified.

[0158] The determination module 403 is configured to read each audio and video joint feature and each text feature included in the feature library, and determine the fifth similarity and / or the sixth similarity, wherein the fifth similarity is the similarity between each audio and video joint feature and the topic feature corresponding to the topic label to be identified, and the sixth similarity is the similarity between each text feature and the topic feature corresponding to the topic label to be identified.

[0159] The determination module 403 is further configured to determine the video corresponding to the audio-video joint feature whose fifth similarity meets the preset condition, or the video corresponding to the text feature whose sixth similarity meets the preset condition, as the target video of the topic tag to be identified.

[0160] Fig.12 FIG. 1 is a schematic diagram of the structure of another video determination device provided by the present disclosure. Fig.12 The video determination device 50 may include at least one processor 501 and a memory 503 for storing processor executable instructions. The processor 501 is configured to execute instructions in the memory 503 to implement the video determination method in the above embodiment.

[0161] In addition, the video determination device 50 may further include a communication bus 502 and at least one communication interface 504 .

[0162] Processor 501 may be a CPU, a microprocessing unit, an ASIC, or one or more integrated circuits for controlling the execution of the program of the disclosed solution.

[0163] Communication bus 502 may include a pathway for transmitting information between the above-mentioned components.

[0164] The communication interface 504 uses any transceiver or other device for communicating with other devices or communication networks, such as Ethernet, RAN, WLAN, etc.

[0165] The memory 503 may be a ROM or other types of static storage devices that can store static information and instructions, a RAM or other types of dynamic storage devices that can store information and instructions, or an EEPROM, a CD-ROM or other optical disk storage, an optical disk storage (including a compressed optical disk, a laser disk, an optical disk, a digital versatile disk, a Blu-ray disk, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store the desired program code in the form of an instruction or data structure and can be accessed by a computer, but is not limited thereto. The memory may be independent and connected to the processing unit via a bus. The memory may also be integrated with the processing unit.

[0166] The memory 503 is used to store instructions for executing the solution of the present disclosure, and the execution is controlled by the processor 501. The processor 501 is used to execute the instructions stored in the memory 503, so as to realize the functions in the method of the present disclosure.

[0167] In a specific implementation, as an embodiment, the processor 501 may include one or more CPUs, such as Fig.12 CPU0 and CPU1 in.

[0168] In a specific implementation, as an embodiment, the video determination device 50 may include multiple processors, such as Fig.12 501 and processor 507 in the embodiment of the present invention. Each of these processors may be a single-CPU processor or a multi-CPU processor. The processor herein may refer to one or more devices, circuits, and / or processing cores for processing data (e.g., computer program instructions).

[0169] In a specific implementation, as an embodiment, the video determination device 50 may further include an output device 505 and an input device 506. The output device 505 communicates with the processor 501 and may display information in a variety of ways. For example, the output device 505 may be an LCD, LED display device, CRT display device, or a projector. The input device 506 communicates with the processor 501 and may receive user input in a variety of ways. For example, the input device 506 may be a mouse, a keyboard, a touch screen device, or a sensor device.

[0170] Those skilled in the art will appreciate that the above Fig.10 The structure shown in the figure does not constitute a limitation on the video tag generation device 30, and the above Fig.12 The structure shown in the figure does not constitute a limitation on the video determination device 50, and may include more or less components than those shown in the figure, or combine some components, or adopt a different component arrangement.

[0171] In addition, the present disclosure also provides a computer-readable storage medium, including instructions, which, when executed by an electronic device, enable the electronic device to execute the method for generating a video tag as provided in the above embodiment, or execute the method for determining a video as provided in the above embodiment.

[0172] In addition, the present disclosure also provides a computer program product, including instructions, which, when executed by an electronic device, enable the electronic device to execute the method for generating a video tag as provided in the above embodiment, or execute the method for determining a video as provided in the above embodiment.

[0173] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. The present disclosure is intended to cover any variations, uses or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The description and examples are to be considered exemplary only, and the true scope and spirit of the present disclosure are indicated by the claims.

Claims

1. A method for generating a video tag, characterized in that: include: Get the video to be identified; Input the video to be identified into an audio-video joint representation model to obtain an audio-video joint feature corresponding to the video to be identified, input the video to be identified into a text representation model to obtain a text feature corresponding to the video to be identified, the audio-video joint feature is aligned with the text feature, a fully connected network corresponds to the audio-video joint feature and the text feature, and the fully connected network is used to project the audio-video joint feature and the text feature into the same space, so as to align the audio-video joint feature and the text feature into a unified space; Read each topic feature included in the feature library, determine a first similarity and a second similarity, wherein the first similarity is the similarity between each topic feature and the audio-video joint feature, and the second similarity is the similarity between each topic feature and the text feature, and each topic feature included in the feature library corresponds to a topic tag; The topic tag corresponding to the topic feature whose first similarity or the second similarity meets the preset condition is determined as the target topic tag of the video to be identified.

2. The method for generating a video tag according to claim 1, characterized in that: The method further comprises: Acquire training data, where the training data includes a plurality of videos within a historical time period and at least one topic tag corresponding to each of the plurality of videos; The first neural network model, the second neural network model and the third neural network model are jointly trained based on the multiple videos and at least one topic tag corresponding to each of the multiple videos to obtain the audio and video joint representation model, the text representation model and the topic representation model.

3. The method for generating a video tag according to claim 2, characterized in that: The jointly training the first neural network model, the second neural network model, and the third neural network model based on the multiple videos and at least one topic tag corresponding to each of the multiple videos to obtain the audio and video joint representation model, the text representation model, and the topic representation model includes: Inputting a video frame of each video in the multiple videos and audio data of each video into the first neural network model to perform feature recognition to obtain an initial audio and video joint feature; Input each of the videos and the first topic label into the second neural network model and the third neural network model respectively to perform feature recognition to obtain initial text features and initial topic features, wherein the first topic label is a topic label corresponding to each of the videos included in the training data, and the initial audio-video joint features, the initial text features, and the initial topic features are aligned; Determining a target loss based on the initial audio and video joint features, the initial text features, and the initial topic features; The parameters of the first neural network model, the parameters of the second neural network model, and the parameters of the third neural network model are iteratively updated based on the target loss to obtain the audio and video joint representation model, the text representation model, and the topic representation model.

4. The method for generating a video tag according to claim 3, characterized in that: The determining the target loss based on the initial audio and video joint feature, the initial text feature, and the initial topic feature includes: Determine a third similarity and a fourth similarity, wherein the third similarity is the similarity between the initial audio-video joint feature corresponding to each video and the initial topic feature corresponding to the first topic tag, and the fourth similarity is the similarity between the initial audio-video joint feature corresponding to each video and the initial text feature corresponding to each video; The target loss is determined based on the third similarity and the fourth similarity.

5. The method for generating a video tag according to any one of claims 1 to 4, characterized in that: The method further comprises: Get multiple hashtags; Inputting the multiple topic tags into a topic representation model to perform feature recognition to obtain topic features corresponding to each of the multiple topic tags; The topic features corresponding to each of the multiple topic tags are stored in the feature library.

6. A video determination method, characterized in that: include: Get the topic tag to be identified; Input the topic tag to be identified into a topic representation model to perform feature recognition to obtain a topic feature corresponding to the topic tag to be identified; Read each audio and video joint feature and each text feature included in the feature library, and determine a fifth similarity and a sixth similarity, wherein the fifth similarity is the similarity between each audio and video joint feature and the topic feature corresponding to the topic tag to be identified, and the sixth similarity is the similarity between each text feature and the topic feature corresponding to the topic tag to be identified; The video corresponding to the audio and video joint feature whose fifth similarity meets the preset condition, or the video corresponding to the text feature whose sixth similarity meets the preset condition, is determined as the target video of the topic tag to be identified, the audio and video joint feature is aligned with the text feature, and a fully connected network corresponds to the audio and video joint feature and the text feature. The fully connected network is used to project the audio and video joint feature and the text feature into the same space together, so as to align the audio and video joint feature and the text feature into a unified space.

7. A device for generating a video tag, characterized in that: include: Acquisition module, processing module and determination module; The acquisition module is configured to acquire the video to be identified; The processing module is configured to input the video to be identified into an audio-video joint representation model to obtain an audio-video joint feature corresponding to the video to be identified, and input the video to be identified into a text representation model to obtain a text feature corresponding to the video to be identified, wherein the audio-video joint feature is aligned with the text feature, and a fully connected network corresponds to the audio-video joint feature and the text feature, and the fully connected network is used to project the audio-video joint feature and the text feature into the same space, so as to align the audio-video joint feature and the text feature into a unified space; The determination module is configured to read each topic feature included in the feature library, determine a first similarity and a second similarity, wherein the first similarity is the similarity between each topic feature and the audio-video joint feature, and the second similarity is the similarity between each topic feature and the text feature, and each topic feature included in the feature library corresponds to a topic tag; The determination module is further configured to determine the topic tag corresponding to the topic feature whose first similarity or the second similarity meets a preset condition as the target topic tag of the video to be identified.

8. The device for generating a video tag according to claim 7, characterized in that: The acquisition module is further configured to acquire training data, wherein the training data includes a plurality of videos within a historical time period and at least one topic tag corresponding to each of the plurality of videos; The processing module is also configured to jointly train the first neural network model, the second neural network model and the third neural network model based on the multiple videos and at least one topic tag corresponding to each of the multiple videos to obtain the audio and video joint representation model, the text representation model and the topic representation model.

9. The device for generating a video tag according to claim 8, characterized in that: The processing module is specifically configured to input the video frame of each video in the multiple videos and the audio data of each video into the first neural network model, perform feature recognition, and obtain initial audio and video joint features; The processing module is further configured to input each video and the first topic label into the second neural network model and the third neural network model respectively, perform feature recognition, and obtain initial text features and initial topic features, wherein the first topic label is a topic label corresponding to each video included in the training data, and the initial audio and video joint features, the initial text features, and the initial topic features are aligned; The determination module is specifically configured to determine a target loss based on the initial audio and video joint feature, the initial text feature, and the initial topic feature; The processing module is specifically configured to iteratively update the parameters of the first neural network model, the parameters of the second neural network model, and the parameters of the third neural network model based on the target loss to obtain the audio and video joint representation model, the text representation model, and the topic representation model.

10. The device for generating a video tag according to claim 9, characterized in that: The determination module is further configured to determine a third similarity and a fourth similarity, wherein the third similarity is the similarity between the initial audio-video joint feature corresponding to each video and the initial topic feature corresponding to the first topic tag, and the fourth similarity is the similarity between the initial audio-video joint feature corresponding to each video and the initial text feature corresponding to each video; The determination module is further specifically configured to determine the target loss based on the third similarity and the fourth similarity.

11. The device for generating a video tag according to any one of claims 7 to 10, characterized in that: The device also includes a storage module; The acquisition module is further configured to acquire a plurality of topic tags; The processing module is further configured to input the multiple topic tags into a topic representation model to perform feature recognition to obtain topic features corresponding to each of the multiple topic tags; The storage module is configured to store the topic features corresponding to each of the multiple topic tags into the feature library.

12. A video determination device, characterized in that: include: Acquisition module, processing module and determination module; The acquisition module is configured to acquire a topic tag to be identified; The processing module is configured to input the topic tag to be identified into a topic representation model, perform feature recognition, and obtain a topic feature corresponding to the topic tag to be identified; The determination module is configured to read each audio and video joint feature and each text feature included in the feature library, and determine a fifth similarity and a sixth similarity, wherein the fifth similarity is the similarity between each audio and video joint feature and the topic feature corresponding to the topic tag to be identified, and the sixth similarity is the similarity between each text feature and the topic feature corresponding to the topic tag to be identified; The determination module is also configured to determine the video corresponding to the audio and video joint feature whose fifth similarity meets the preset condition, or the video corresponding to the text feature whose sixth similarity meets the preset condition, as the target video of the topic tag to be identified, the audio and video joint feature is aligned with the text feature, and there is a fully connected network corresponding to the audio and video joint feature and the text feature, and the fully connected network is used to project the audio and video joint feature and the text feature into the same space together, so as to align the audio and video joint feature and the text feature into a unified space.

13. An electronic device, characterized in that: The electronic device comprises: processor; a memory configured to store instructions executable by the processor; The processor is configured to execute the instructions to implement the method for generating a video tag according to any one of claims 1 to 5, or to implement the video determination method according to claim 6.

14. A computer-readable storage medium having instructions stored thereon, characterized in that: When the instructions in the computer-readable storage medium are executed by an electronic device, the electronic device is enabled to execute the method for generating a video tag as described in any one of claims 1 to 5, or execute the video determination method as described in claim 6.

15. A computer program product, characterized in that The computer program product includes computer instructions, and when the computer instructions are executed on an electronic device, the electronic device executes the method for generating a video tag according to any one of claims 1 to 5, or executes the video determination method according to claim 6.

Images