AI-based comment generation methods, devices, equipment, and storage media

Abstract

This application provides a method, apparatus, electronic device, and computer-readable storage medium for generating comments based on artificial intelligence; it relates to artificial intelligence technology; the method includes: encoding multimedia information of a video to obtain multimedia features of the video; fusing the multimedia information of the video to obtain fused features of the video; encoding the fused features of the video to obtain encoded features of the video; classifying the video plot based on the encoded features of the video to obtain plot type information of the video; and generating text based on the plot type information and the encoded features of the video to obtain comment information that matches the plot of the video. This application can improve the accuracy of video comment generation.

Description

Technical Field

[0001] This application relates to artificial intelligence technology, and more particularly to an AI-based comment generation method, apparatus, electronic device, and computer-readable storage medium. Background Technology

[0002] Artificial Intelligence (AI) is a comprehensive technology within computer science that studies the design principles and implementation methods of various intelligent machines, enabling them to possess perception, reasoning, and decision-making capabilities. AI technology is a multidisciplinary field encompassing a wide range of areas, including natural language processing and machine learning / deep learning. With technological advancements, AI will be applied in more fields and play an increasingly important role.

[0003] Video comment generation is an important application of artificial intelligence. It can guide users to comment on videos after they are published, thereby increasing user engagement.

[0004] Currently, there is a lack of effective solutions for generating video comments in related technologies, with most relying on encoding the text of the video to generate comments. However, this approach produces inaccurate video comments and wastes significant computational resources. Summary of the Invention

[0005] This application provides an artificial intelligence-based comment generation method, apparatus, electronic device, and computer-readable storage medium, which can improve the accuracy of video comment generation.

[0006] The technical solution of this application embodiment is implemented as follows:

[0007] This application provides an artificial intelligence-based comment generation method, including:

[0008] Feature extraction processing is performed on the video to obtain features of multiple modalities of the video;

[0009] The fusion features of the video are obtained by fusing features from multiple modalities of the video.

[0010] The fusion features of the video are encoded to obtain the encoded features of the video;

[0011] Based on the encoding features of the video, video plot classification processing is performed to obtain the plot type information of the video;

[0012] Text generation processing is performed based on the plot type information and encoding features of the video to obtain comment information that matches the plot of the video.

[0013] In the above technical solution, the step of performing feature extraction processing on the video to obtain features of multiple modalities of the video includes:

[0014] The text information of the video is processed by text encoding to obtain the text features of the video;

[0015] The image information of the video is subjected to image encoding processing to obtain the image features of the video;

[0016] The text features and image features of the video are used as features of multiple modalities of the video.

[0017] In the above technical solution, the step of fusing features based on multiple modalities of the video to obtain the fused features of the video includes:

[0018] The image features of the video are mapped to obtain the image mapping features of the video;

[0019] The text features and image mapping features of the video are concatenated to obtain the fused features of the video.

[0020] In the above technical solution, the step of performing text encoding processing on the text information of the video to obtain the text features of the video includes:

[0021] The text information of the video is segmented into words to obtain multiple words from the text information;

[0022] Multiple words in the text information are concatenated to obtain a word sequence of the text information;

[0023] The word sequence of the text information is subjected to vector transformation processing to obtain the text features of the video.

[0024] In the above technical solution, the step of performing vector transformation processing on the word sequence of the text information to obtain the text features of the video includes:

[0025] The word sequence of the text information is vector encoded to obtain the word vector sequence of the text information;

[0026] Obtain the position vectors and type vectors of multiple word vectors in the word vector sequence;

[0027] The text features of the video are obtained by encoding based on the position vectors and type vectors of the multiple word vectors.

[0028] In the above technical solution, the encoding process based on the position vectors and type vectors of the multiple word vectors to obtain the text features of the video includes:

[0029] For any one of the plurality of word vectors, the following processing is performed: the position vector of the word vector, the type vector of the word vector, and the word vector are summed to obtain the sum vector of the word vectors;

[0030] The sum of the multiple word vectors is concatenated to obtain the text features of the video.

[0031] This application provides an artificial intelligence-based comment generation device, including:

[0032] The feature extraction module is used to perform feature extraction processing on the video to obtain features of multiple modalities of the video;

[0033] The fusion module is used to perform fusion processing based on the features of multiple modalities of the video to obtain the fused features of the video;

[0034] An encoding module is used to encode the fusion features of the video to obtain the encoded features of the video;

[0035] The classification module is used to perform video plot classification processing based on the encoded features of the video to obtain the plot type information of the video;

[0036] The generation module is used to perform text generation processing based on the plot type information and encoding features of the video to obtain comment information that matches the plot of the video.

[0037] In the above technical solution, the feature extraction module is further used to perform text encoding processing on the text information of the video to obtain the text features of the video;

[0038] The image information of the video is subjected to image encoding processing to obtain the image features of the video;

[0039] The text features and image features of the video are used as features of multiple modalities of the video.

[0040] In the above technical solution, the fusion module is further used to perform mapping processing on the image features of the video to obtain the image mapping features of the video;

[0041] The text features and image mapping features of the video are concatenated to obtain the fused features of the video.

[0042] In the above technical solution, the feature extraction module is further used to perform word segmentation on the text information of the video to obtain multiple words of the text information;

[0043] Multiple words in the text information are concatenated to obtain a word sequence of the text information;

[0044] The word sequence of the text information is subjected to vector transformation processing to obtain the text features of the video.

[0045] In the above technical solution, the feature extraction module is further used to perform vector encoding processing on the word sequence of the text information to obtain the word vector sequence of the text information;

[0046] Obtain the position vectors and type vectors of multiple word vectors in the word vector sequence;

[0047] The text features of the video are obtained by encoding based on the position vectors and type vectors of the multiple word vectors.

[0048] In the above technical solution, the feature extraction module is further configured to perform the following processing on any one of the plurality of word vectors: summing the position vector of the word vector, the type vector of the word vector, and the word vector to obtain the sum vector of the word vector;

[0049] The sum of the multiple word vectors is concatenated to obtain the text features of the video.

[0050] In the above technical solution, the encoding process is implemented through an encoding model, which includes multiple cascaded encoding layers; the encoding module is further used to perform attention-based encoding processing on the fusion features of the video through the first encoding layer of the multiple cascaded encoding layers.

[0051] The encoding result of the first encoding layer is output to the subsequent concatenated encoding layers, and the encoding processing and encoding result output are continued through the subsequent concatenated encoding layers;

[0052] The encoding result output from the last encoding layer is used as the encoding feature of the video.

[0053] In the above technical solution, the encoding module is further configured to perform the following processing through the j-th encoding layer of the plurality of cascaded encoding layers:

[0054] The encoding result of the (j-1)th encoding layer is subjected to self-attention processing to obtain the jth self-attention feature;

[0055] The j-th self-attention feature and the encoding result of the (j-1)-th encoder are subjected to residual connection processing to obtain the j-th residual feature;

[0056] The j-th residual feature is mapped to obtain the j-th mapped feature;

[0057] Perform a residual connection operation on the j-th mapping feature and the j-th residual feature to obtain the encoding result of the j-th encoding layer;

[0058] Output the encoding result of the j-th encoding layer through the j-th encoding layer;

[0059] Among them, j is an increasing natural number and its value range is 1 < j ≤ N, N is the number of encoding layers, and N is a positive integer greater than 1.

[0060] In the above technical solution, the generation module is further configured to perform decoding processing on the plot type information of the video and the encoding features of the video to generate word vectors that conform to the plot of the video;

[0061] According to the order of the generated word vectors, perform combination processing on the words corresponding to the word vectors to obtain comment information that conforms to the plot of the video.

[0062] In the above technical solution, the generation module is further configured to perform decoding processing on the plot type information of the video and the encoding features of the video to generate the first word vector that conforms to the plot of the video;

[0063] Based on the plot type information of the video, the encoding features of the video, and the previously generated i - 1 word vectors, perform decoding processing to generate the i-th word vector that conforms to the plot of the video;

[0064] Among them, i is an increasing natural number and its value range is 1 < i ≤ M, M is the number of word vectors that conform to the plot of the video, and M is a positive integer greater than 1.

[0065] In the above technical solution, the decoding processing is implemented by a decoder, and the decoder includes multiple cascaded decoding layers; the generation module is further configured to perform decoding processing on the plot type information of the video, the encoding features of the video, and the previously generated i - 1 word vectors through the first decoding layer of the multiple cascaded decoding layers;

[0066] Output the decoding result of the first decoding layer to the subsequent cascaded decoding layers, and continue to perform decoding processing and decoding result output through the subsequent cascaded decoding layers;

[0067] Take the decoding result output by the last decoding layer as the i-th word vector that conforms to the plot of the video.

[0068] In the above technical solution, the generation module is further configured to perform the following processing through the k-th decoding layer of the multiple cascaded decoding layers:

[0069] Perform self-attention-based mapping processing on the decoding result of the (k - 1)-th decoding layer to obtain the k-th mapping feature;

[0070] Perform decoding processing based on multi-head attention on the k-th mapping feature and the encoded feature of the video to obtain the k-th decoded feature;

[0071] Perform non-linear mapping processing on the k-th decoded feature to obtain the k-th non-linear mapping feature;

[0072] Perform residual connection processing on the k-th non-linear mapping feature and the k-th decoded feature to obtain the decoding result of the k-th decoding layer;

[0073] Output the decoding result of the k-th decoding layer through the k-th decoding layer;

[0074] Where k is an increasing natural number and the value range is 1 < k ≤ H, H is the number of decoding layers, and H is a positive integer greater than 1.

[0075] In the above technical solution, the generation module is further configured to perform self-attention processing on the decoding result of the (k - 1)-th decoding layer to obtain the k-th self-attention feature;

[0076] Perform residual connection processing on the k-th self-attention feature and the decoding result of the (k - 1)-th decoding layer to obtain the k-th mapping feature.

[0077] In the above technical solution, the generation module is further configured to perform multi-head attention processing on the k-th mapping feature and the encoded feature of the video to obtain the k-th multi-head attention feature;

[0078] Perform residual connection processing on the k-th multi-head attention feature and the k-th mapping feature to obtain the k-th decoded feature.

[0079] In the above technical solution, the classification module is further configured to perform dot product processing on the encoded feature of the video and multiple plot type vectors respectively to obtain the dot product scores corresponding to each plot type;

[0080] Perform mapping processing on the dot product scores corresponding to each plot type to obtain the probabilities corresponding to each plot type;

[0081] Perform screening processing based on the probabilities corresponding to each plot type, and use the screened plot type as the plot type information of the video.

[0082] An embodiment of the present application provides an electronic device for comment generation, and the electronic device includes:

[0083] A memory for storing executable instructions;

[0084] The processor, when executing executable instructions stored in the memory, implements the AI-based comment generation method provided in the embodiments of this application.

[0085] This application provides a computer-readable storage medium storing executable instructions for inducing a processor to execute and implement the AI-based comment generation method provided in this application.

[0086] The embodiments of this application have the following beneficial effects:

[0087] By fusing features from multiple video modalities to obtain diverse coding features, and classifying video plots based on these coding features to obtain plot type information, text generation is performed based on the plot type information and the video's coding features. This approach leverages the plot type information to improve the rationality and accuracy of video comment generation. Compared to schemes that generate video comments based on video text information, this method saves computational resources by sharing coding features in both the video plot classification and text generation tasks. Attached Figure Description

[0088] Figure 1 This is a schematic diagram illustrating an application scenario of the video playback system provided in this application embodiment;

[0089] Figure 2 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application;

[0090] Figures 3A-3C This is a flowchart illustrating the AI-based comment generation method provided in an embodiment of this application;

[0091] Figure 4 This is a schematic diagram of the overall model structure provided in the embodiments of this application;

[0092] Figure 5 This is a schematic diagram of the encoder provided in an embodiment of this application;

[0093] Figure 6 This is a schematic diagram of the structure of the video comment generation model provided in the embodiments of this application;

[0094] Figure 7 This is a schematic diagram of the encoding-decoding structure provided in the embodiments of this application. Detailed Implementation

[0095] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0096] In the following description, the terms "first" and "second" are used merely to distinguish similar objects and do not represent a specific ordering of objects. It is understood that "first" and "second" may be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0097] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0098] Before providing a further detailed description of the embodiments of this application, the nouns and terms involved in the embodiments of this application will be explained, and the nouns and terms involved in the embodiments of this application shall be interpreted as follows.

[0099] 1) Artificial Intelligence (AI): A comprehensive technology in computer science that studies the design principles and implementation methods of various intelligent machines, enabling them to have perception, reasoning, and decision-making functions. AI technology is a multidisciplinary field, involving a wide range of areas, such as natural language processing, machine learning / deep learning, autonomous driving, and intelligent transportation. With technological advancements, AI technology will be applied in more fields and play an increasingly important role.

[0100] 2) Client: An application running on a terminal that provides various services, such as a video client or a game client.

[0101] 3) Response: used to indicate the conditions or states on which the operation is performed. When the conditions or states on which the operation is performed are met, one or more operations may be performed in real time or with a set delay. Unless otherwise specified, there is no restriction on the order in which the multiple operations are performed.

[0102] 4) Modality: Every source or form of information can be called a modality, such as information in the form of speech, video, and text. Each form of information can be called a modality of information. Modality can be divided into unimodality and multimodality; unimodality represents information as a numerical vector that a computer can process or further abstracts it into a higher-level feature vector, while multimodality learns better feature representations by utilizing the complementarity between multiple modalities and eliminating redundancy between modalities.

[0103] 5) Sequence-to-Sequence Model: A neural network structure consisting of an encoder and decoder made up of a bidirectional recurrent neural network (RNN), which enables the conversion between one sequence and another when the lengths of the input and output sequences are different.

[0104] 6) Optical Character Recognition (OCR): The process by which electronic devices (such as scanners or digital cameras) examine characters printed on paper and then use character recognition methods to translate the shapes into computer text. In other words, it is the process of scanning text data and then analyzing and processing image files to obtain text and layout information.

[0105] 7) Automatic Speech Recognition (ASR): A technology that converts human speech into text. Its goal is to convert the lexical content of human speech into computer-readable input, such as keystrokes, binary codes, or character sequences.

[0106] 8) Blockchain: It consists of a series of blocks that are sequentially generated. Once a new block is added to the blockchain, it will not be removed. The blocks record the data submitted by the nodes in the blockchain system.

[0107] This application provides an artificial intelligence-based comment generation method, apparatus, electronic device, and computer-readable storage medium, which can improve the accuracy of video comment generation.

[0108] The AI-based comment generation method provided in this application can be implemented by a terminal / server alone; or it can be implemented collaboratively by a terminal and a server. For example, the terminal can independently undertake the AI-based comment generation method described below, or the terminal can send a playback request for a video to the server, and the server can execute the AI-based comment generation method according to the received playback request, classify the video plot based on the video's encoding features to obtain the video's plot type information, and perform text generation processing based on the video's plot type information and the video's encoding features to obtain comment information that matches the video's plot. This utilizes the video's plot type information to improve the rationality and accuracy of video comment generation.

[0109] The electronic device for video processing provided in this application can be various types of terminals or servers. The server can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms. The terminal can be a smartphone, tablet, laptop, desktop computer, smart speaker, smartwatch, smart TV, etc., but is not limited to these. The terminal and server can be directly or indirectly connected via wired or wireless communication, which is not limited herein.

[0110] Taking servers as an example, such as server clusters deployed in the cloud, AI as a Service (AIaaS) is provided to users. The AIaaS platform breaks down several common AI services and provides them as independent or packaged services in the cloud. This service model is similar to an AI-themed marketplace. All users can access and use one or more artificial intelligence services provided by the AIaaS platform through application programming interfaces.

[0111] For example, one type of AI cloud service could be a comment generation service, where a cloud server encapsulates the comment generation program provided in this application embodiment. Users invoke the comment generation service in the cloud service through a terminal (running a client, such as a video client), causing the cloud-deployed server to call the encapsulated comment generation program. This program classifies the video's encoded features to obtain the video's plot type information, and then performs text generation processing based on the plot type information and the video's encoded features to obtain comment information that matches the video's plot. This utilizes the video's plot type information to improve the rationality and accuracy of video comment generation.

[0112] See Figure 1 , Figure 1 This is a schematic diagram of the application scenario of the video playback system 10 provided in the embodiments of this application. The terminal 200 is connected to the server 100 through the network 300. The network 300 can be a wide area network or a local area network, or a combination of the two.

[0113] A terminal (running a client, such as a video client) can be used to obtain playback requests for videos. For example, when a user opens a video client running on a terminal and clicks to play a video, the terminal automatically obtains playback requests for the video (including videos used for generating comments).

[0114] In some embodiments, a comment generation plugin may be embedded in the client running on the terminal 200 to implement an AI-based comment generation method locally on the client. For example, the terminal 200 calls the comment generation plugin to implement an AI-based comment generation method, which performs feature extraction on the video to obtain features of multiple modalities, fuses these features to obtain fused features, encodes these fused features to obtain coded features, classifies the video plot based on these coded features to obtain plot type information, and generates text based on the plot type information and the coded features to obtain comment information that matches the video plot. This utilizes the plot type information to improve the rationality and accuracy of video comment generation, thereby guiding users to comment on the video after its release and increasing user engagement.

[0115] In some embodiments, after receiving a playback request for a video, the terminal 200 calls the comment generation interface of the server 100 (which can be provided as a cloud service, i.e., a comment generation service). Based on the playback request, the server 100 performs feature extraction processing on the video to obtain features of multiple modalities of the video. Based on the features of multiple modalities of the video, it performs fusion processing to obtain fused features of the video. It performs encoding processing on the fused features of the video to obtain encoded features of the video. Based on the encoded features of the video, it performs video plot classification processing to obtain plot type information of the video. Based on the plot type information and the encoded features of the video, it performs text generation processing to obtain comment information that matches the plot of the video. The comment information is then sent to the terminal 200 so that the terminal 200 can display the generated comment information when playing the video. This utilizes the plot type information of the video to improve the rationality and accuracy of video comment generation. As a result, after the video is published, the accurately generated video comments can guide users to comment on the video, thereby increasing user engagement.

[0116] In some embodiments, the terminal or server can implement the AI-based comment generation method provided in this application by running a computer program, wherein the computer program is such as... Figure 1 The client running in the terminal 200 shown can be, for example, a computer program that is a native program or software module in an operating system; it can be a native application (APP), i.e., a program that needs to be installed in the operating system to run; it can also be a small program, i.e., a program that only needs to be downloaded to a browser environment to run; or it can be a small program that can be embedded in any APP. In short, the above-mentioned computer program can be any form of application, module or plugin.

[0117] In some embodiments, multiple servers can form a blockchain, and server 100 is a node on the blockchain. Information connections can exist between each node in the blockchain, and information can be transmitted between nodes through these connections. Data related to the AI-based comment generation method provided in this application (e.g., comment generation logic, video comment information) can be stored on the blockchain.

[0118] The structure of the electronic device provided in the embodiments of this application is described below. See also... Figure 2 , Figure 2 This is a schematic diagram of the structure of the electronic device 500 provided in an embodiment of this application. The electronic device 500 can be a terminal or a server. This embodiment of the application uses a server as an example for illustration. Figure 2 The illustrated electronic device 500 includes at least one processor 510, a memory 550, at least one network interface 520, and a user interface 530. The various components in the electronic device 500 are coupled together via a bus system 540. It is understood that the bus system 540 is used to implement communication between these components. In addition to a data bus, the bus system 540 also includes a power bus, a control bus, and a status signal bus. However, for clarity, ... Figure 2 The general labeled all buses as Bus System 540.

[0119] The processor 510 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor, etc.

[0120] Memory 550 may include volatile memory or non-volatile memory, or both. The non-volatile memory may be read-only memory (ROM), and the volatile memory may be random access memory (RAM). The memory 550 described in this application embodiment is intended to include any suitable type of memory. Memory 550 may optionally include one or more storage devices physically located away from processor 510.

[0121] In some embodiments, memory 550 is capable of storing data to support various operations, examples of which include programs, modules, and data structures or subsets or supersets thereof, as illustrated below.

[0122] Operating system 551 includes system programs for handling various basic system services and performing hardware-related tasks, such as the framework layer, core library layer, driver layer, etc., for implementing various basic business functions and handling hardware-based tasks;

[0123] The network communication module 552 is used to reach other electronic devices via one or more (wired or wireless) network interfaces 520, exemplary network interfaces 520 including: Bluetooth, WiFi, and Universal Serial Bus (USB), etc.

[0124] In some embodiments, the AI-based comment generation apparatus provided in this application can be implemented in software. Figure 2 An AI-based comment generation device 555, stored in memory 550, is shown. This device can be software in the form of programs and plugins, and includes the following software modules: a feature extraction module 5551, a fusion module 5552, an encoding module 5553, a classification module 5554, and a generation module 5555. These modules are logically connected and can therefore be arbitrarily combined or further separated according to their implemented functions. The functions of each module will be described below.

[0125] As mentioned above, the AI-based comment generation method provided in this application can be implemented by various types of electronic devices. See also Figure 3A , Figure 3A This is a flowchart illustrating the AI-based comment generation method provided in this application embodiment, combined with... Figure 3A The steps shown are explained.

[0126] In the following steps, the features of multiple modalities of the video include: text representation features (i.e., text features, text modal features) and image representation features (i.e., image features, image modal features). Of course, the features of multiple modalities of the video can also include audio representation features (i.e., audio features, audio modal features).

[0127] In step 101, feature extraction processing is performed on the video to obtain features of multiple modalities of the video.

[0128] For example, when uploading a video, users fill in text information such as the video title and tags, and then upload the video. After the video is published, it's necessary to generate video comments to encourage user engagement. Before generating video comments, feature extraction processing is required to obtain features from multiple modalities of the video, which will then be used for subsequent tasks such as video plot type classification and video comment generation based on these features.

[0129] In some embodiments, feature extraction processing is performed on the video to obtain features of multiple modalities of the video, including: performing text encoding processing on the text information of the video to obtain text features of the video; performing image encoding processing on the image information of the video to obtain image features of the video; and using the text features and image features of the video as features of multiple modalities of the video.

[0130] For example, each image frame in a video can be encoded in image form to obtain a vector representation of each image frame. These vector representations can then be fused to obtain the video's image features. Similarly, each text frame in a video can be encoded in text form to obtain a vector representation of each text frame. These vector representations can then be fused to obtain the video's text features. Likewise, each audio frame in a video can be encoded in audio form to obtain a vector representation of each audio frame. These vector representations can then be fused to obtain the video's audio features.

[0131] In some embodiments, text encoding processing is performed on the text information of the video to obtain text features of the video, including: segmenting the text information of the video into words to obtain multiple words of the text information; concatenating the multiple words of the text information to obtain a word sequence of the text information; and performing vector transformation processing on the word sequence of the text information to obtain text features of the video.

[0132] The text information includes at least one of the following types: video title, video tags, text generated by OCR, text generated by ASR, and operational category type. For example, the text information such as video title, video tags, text generated by OCR, text generated by ASR, and operational category type is segmented into multiple words. These multiple words are then concatenated using the "[SEP]" separator to obtain a word sequence, with the initial symbol "[CLS]". The length of the word sequence can be controlled; if the length exceeds a set threshold, the portion exceeding the threshold is truncated; if the length is less than the threshold, the "[PAD]" symbol is added. Finally, the word sequence is converted into a vector sequence to obtain the text features of the video.

[0133] In some embodiments, vector transformation processing is performed on the word sequence of text information to obtain the text features of the video, including: performing vector encoding processing on the word sequence of text information to obtain a word vector sequence of text information; obtaining the position vectors and type vectors of multiple word vectors in the word vector sequence; and performing encoding processing based on the position vectors and type vectors of multiple word vectors to obtain the text features of the video.

[0134] For example, an encoder (such as the Lightweight Bidirectional Encoder Representations from Transformers (ALBERT) model) transforms a sequence of words in text information into a sequence of word vectors. This involves converting words in the sequence into vectors and generating a position vector and a type vector for each word vector. Encoding is then performed based on the position and type vectors of multiple word vectors to obtain the text features of the video. These text features contain both positional and type information, allowing for an accurate representation of the video's textual features. The type vector represents the type of the word vectors, such as the video title, video tags, text generated by OCR, text generated by ASR, or operational classification types.

[0135] In some embodiments, encoding is performed based on the position vectors and type vectors of multiple word vectors to obtain the text features of the video, including: performing the following processing on any word vector among the multiple word vectors: summing the position vector, type vector, and word vector of the word vector to obtain a summed vector of word vectors; concatenating the summed vectors of multiple word vectors to obtain the text features of the video.

[0136] Following the example above, after generating a position vector and a type vector for each word vector in the word vector sequence using ALBERT, the position vector and type vector of any word vector are added to any word vector to obtain the sum vector of any word vector. The sum vectors of all word vectors are then concatenated to obtain the text features of the video.

[0137] In step 102, features from multiple modalities of the video are fused to obtain fused features of the video.

[0138] For example, after obtaining the features of multiple modalities of a video, the features of the multiple modalities of the video are fused to obtain the fused features of the video. This allows for subsequent tasks such as video comment generation and video plot type classification based on the fused features of the video, thereby improving the rationality and diversity of video comment generation and the accuracy of video plot type classification.

[0139] In some embodiments, the fusion processing based on the features of multiple modalities of the video to obtain the fused features of the video includes: mapping the image features of the video to obtain the image mapping features of the video; and concatenating the text features of the video and the image mapping features of the video to obtain the fused features of the video.

[0140] For example, when the features of multiple modalities of a video include both image features and text features, a linear transformation is first performed on the image features of the video to obtain the image mapping features of the video. Then, the text features and the image mapping features of the video are concatenated to obtain the fused features of the video. For example, if the first position of the text feature is a vector of “[CLS]”, then the image mapping features of the video are added to the vector of “[CLS]” to achieve the concatenation operation.

[0141] In step 103, the fusion features of the video are encoded to obtain the encoded features of the video.

[0142] For example, after obtaining the fusion features of the video, the fusion features of the video can be encoded by the encoder in the sequence-to-sequence model to obtain the encoded features of the video. Subsequently, the encoded features can be shared by video comment generation tasks and video plot type classification tasks, which can greatly reduce the consumption of computing resources.

[0143] In some embodiments, the encoding process is implemented through an encoding model, which includes multiple cascaded encoding layers. Encoding the fusion features of the video to obtain the encoded features of the video includes: performing attention-based encoding on the fusion features of the video through the first encoding layer in the multiple cascaded encoding layers; outputting the encoding result of the first encoding layer to subsequent cascaded encoding layers, continuing the encoding process and outputting the encoding result through subsequent cascaded encoding layers; and using the encoding result output by the last encoding layer as the encoded features of the video.

[0144] like Figure 7 As shown, the coding model (i.e., the encoder) can adopt the coding structure in the sequence-to-sequence structure (e.g., the transformer encoder), and the coding model includes multiple cascaded coding layers. Figure 7 The diagram shows two transformer coding layers. The first coding layer in a series of cascaded coding layers performs attention-based coding on the fused features of the video. The coding result of the first coding layer is then passed to the second coding layer, where attention-based coding continues. The output of the second coding layer is used as the coded features of the video. With each coding layer, the understanding of the coding result deepens. Through multiple coding layers, the model gradually learns lexical, syntactic, and semantic information.

[0145] In some embodiments, encoding processing and encoding result output are continued through subsequent cascaded encoding layers, including: performing the following processing through the j-th encoding layer of multiple cascaded encoding layers: performing self-attention processing on the encoding result of the (j - 1)-th encoding layer to obtain the j-th self-attention feature; performing residual connection processing on the j-th self-attention feature and the encoding result of the (j - 1)-th encoder to obtain the j-th residual feature; performing mapping processing on the j-th residual feature to obtain the j-th mapping feature; performing residual connection processing on the j-th mapping feature and the j-th residual feature to obtain the encoding result of the j-th encoding layer; outputting the encoding result of the j-th encoding layer through the j-th encoding layer; where j is an increasing natural number and the value range is 1 < j ≤ N, N is the number of encoding layers, and N is a positive integer greater than 1.

[0146] As Figure 5 shown, each encoder layer contains two parts, a self-attention module and a feed forward module. Among them, the self-attention module contains specific self-attention operations and residual connection operations (including add operation and normalize operation (add&normalize operation), that is, adding bit by bit first and then normalizing), and the feed forward module contains specific feed forward operations (that is, dimension increasing first and then dimension decreasing) and add&normalize operations. After one layer of encoder layer calculation, the deep meaning understanding of each vector for the whole sentence is deepened by one step. After multiple layers of encoder layer calculation, the model gradually learns information such as morphology, grammar, and semantics.

[0147] For example, the specific encoding process for each layer is as follows: performing self-attention operation on the encoding result of the (j - 1)-th encoding layer to obtain the j-th self-attention feature; first adding the j-th self-attention feature and the encoding result of the (j - 1)-th encoder bit by bit, and then normalizing the sum result to obtain the j-th residual feature; increasing the dimension first and then decreasing the dimension of the j-th residual feature to obtain the j-th mapping feature; adding the j-th mapping feature and the j-th residual feature bit by bit, and normalizing the sum result to obtain the encoding result of the j-th encoding layer.

[0148] The Self-attention operation calculates the similarity between any two vectors, normalizes the similarity of each vector to all other vectors into weights that sum to 1, and then weights these weights and sums them with the other vectors to obtain a new vector for each vector. Vectors without self-attention only contain information about themselves and do not carry the meaning of the entire sentence. After self-attention, each vector can acquire the information carried by other vectors, thus obtaining the full sentence information relevant to itself. For example, if the input to the encoding layer includes three vectors, a1, a2, and a3, and the similarity between a1 and a2 is 0.7, and the similarity between a1 and a3 is also 0.7, then after normalizing the similarities, the weights of a2 and a3 are 0.5. Therefore, after the Self-attention operation, a1 contains the vector 0.5*a2 + 0.5*a3, thus obtaining the full sentence information relevant to itself.

[0149] In step 104, video plot classification is performed based on the video's encoding features to obtain the video's plot type information.

[0150] For example, after obtaining the encoded features of a video, the encoded features can be classified into video plots using a video plot type classification model to obtain the plot type information of the video. This information can then be used to generate comments, thereby improving the rationality and accuracy of video comments.

[0151] See Figure 3B , Figure 3B This is an optional flowchart illustrating an artificial intelligence-based evaluation generation method provided in an embodiment of this application. Figure 3B Show Figure 3A Step 104 can be implemented through steps 1041-1043: In step 1041, the encoded features of the video are multiplied by multiple plot type vectors to obtain the dot product score corresponding to each plot type; in step 1042, the dot product score corresponding to each plot type is mapped to obtain the probability corresponding to each plot type; in step 1043, a filtering process is performed based on the probability corresponding to each plot type, and the filtered plot types are used as the plot type information of the video.

[0152] For example, the video plot type classification model uses the encoded features output by the encoder as input, pre-sets multiple video plot type vectors, multiplies the encoded features of the video with all the plot type vectors. At this time, each plot type obtains a dot product score, and through the sigmoid function, a probability value in the range of 0-1 is obtained. This probability value represents the probability that the video belongs to this plot type. After obtaining the probability values of all plot types, the video plot types greater than the threshold are used as the plot type information of the video. Or, based on the probability values of the plot types, the plot types are sorted in descending order, and the top k plot types in the sorting result are used as the plot type information of the video.

[0153] In step 105, text generation processing is performed based on the plot type information of the video and the encoded features of the video to obtain comment information that conforms to the plot of the video.

[0154] For example, by combining the comment generation task with the video plot type classification task, the rationality and diversity of video comment generation and the accuracy of video plot type classification can be improved. By sharing the encoder, the consumption of computing resources can be greatly reduced.

[0155] See Figure 3C , Figure 3C which is an optional process schematic diagram of the evaluation generation method based on artificial intelligence provided by the embodiments of the present application. Figure 3C shows Figure 3A Step 105 in can be implemented through steps 1051-step 1052: In step 1051, decoding processing is performed on the plot type information of the video and the encoded features of the video to generate word vectors that conform to the plot of the video; in step 1052, according to the order of the generated word vectors, the words corresponding to the word vectors are combined to obtain comment information that conforms to the plot of the video.

[0156] Continuing the above example, the solution for decoding the plot type information of the video and the encoded features of the video is as follows: Decoding processing is performed on the plot type information of the video and the encoded features of the video to generate the first word vector that conforms to the plot of the video; based on the plot type information of the video, the encoded features of the video, and the previously generated i-1 word vectors, decoding processing is performed to generate the i-th word vector that conforms to the plot of the video; where i is an increasing natural number and the value range is 1 < i ≤ M, and M is the number of word vectors that conform to the plot of the video, and M is a positive integer greater than 1.

[0157] For example, the structure of a video comment generation model adopts a decoding structure within a sequence-to-sequence structure (e.g., a transformer decoder structure). The vector corresponding to the plot type information (i.e., the video plot vector) is used as the initialization vector for the transformer decoder, and decoding begins. First, the first word vector (token) is predicted based on the initialization vector and the video's encoding features. Then, the second token is predicted based on the initialization vector and the first token, and so on, until the predicted token is a terminator or the prediction length reaches a threshold. Based on the order of the generated word vectors, the words corresponding to the word vectors are combined to obtain comment information that matches the video's plot. For instance, after predicting the probability distribution of the first token based on the initialization vector and the video's encoding features, the probability distribution is sorted in descending order. The first k vectors are used as the first token and input together with the video plot vector into the transformer decoder to predict the second token, and so on. The final output resembles a tree structure, where each path from the root node to a leaf node represents a comment. The score for each comment is the product of the probabilities of each token along the path. The scores of the comments are sorted in descending order, and the top k comments are selected. Thus, each video plot type can obtain k comments.

[0158] In some embodiments, the decoding process is implemented by a decoder, which includes multiple cascaded decoding layers. The decoding process, based on the video's plot type information, the video's encoding features, and the generated first i-1 word vectors, generates the i-th word vector that matches the video's plot. This includes: decoding the video's plot type information, the video's encoding features, and the generated first i-1 word vectors through the first decoding layer of the multiple cascaded decoding layers; outputting the decoding result of the first decoding layer to subsequent cascaded decoding layers, continuing the decoding process and outputting the decoding result through these subsequent cascaded decoding layers; and using the decoding result output by the last decoding layer as the i-th word vector that matches the video's plot.

[0159] like Figure 7 As shown, the decoding model (i.e., the decoder) can adopt the decoding structure in the sequence-to-sequence structure (e.g., a transformer decoder). The decoding model includes multiple cascaded decoding layers. Figure 7Two transformer decoder layers are shown. Through the first decoder layer of multiple cascaded decoder layers, the plot type information of the video, the encoded features of the video, and the generated first i - 1 word vectors are decoded. The decoding result of the first decoder layer is output to the second decoder layer, and decoding processing continues through the second decoder layer. The decoding result of the second decoder layer is used as the i-th word vector that conforms to the plot of the video. Among them, after one layer of decoder layer calculation, the understanding of the decoding result is deepened by one step. After multiple layers of decoder layer calculation, the model gradually learns information such as morphology, grammar, and semantics.

[0160] In some embodiments, continuing the decoding processing and decoding result output through subsequent cascaded decoder layers includes: performing the following processing through the k-th decoder layer of multiple cascaded decoder layers: performing self-attention-based mapping processing on the decoding result of the k - 1-th decoder layer to obtain the k-th mapping feature; performing multi-head attention-based decoding processing on the k-th mapping feature and the encoded features of the video to obtain the k-th decoding feature; performing non-linear mapping processing on the k-th decoding feature to obtain the k-th non-linear mapping feature; performing residual connection processing on the k-th non-linear mapping feature and the k-th decoding feature to obtain the decoding result of the k-th decoder layer; outputting the decoding result of the k-th decoder layer through the k-th decoder layer; where k is an increasing natural number and the value range is 1 < k ≤ H, H is the number of decoder layers, and H is a positive integer greater than 1.

[0161] As Figure 6 shown, each decoder layer includes three parts, a self-attention module, a feed forward module, and an encoder-decoder attention module. Among them, the self-attention module includes specific self-attention operations and residual connection (add&normalize) operations (that is, add bit by bit first, and then normalize), the feed forward module includes specific feed forward operations (that is, increase the dimension first and then reduce the dimension) and add&normalize operations, and the encoder-decoder attention module includes specific encoder-decoder attention operations and add&normalize operations.

[0162] Following the example above, the decoding result of the (k-1)th decoding layer is processed based on self-attention to obtain the kth mapping feature. This includes: performing self-attention processing on the decoding result of the (k-1)th decoding layer to obtain the kth self-attention feature; and performing residual connection processing on the kth self-attention feature and the decoding result of the (k-1)th decoding layer to obtain the kth mapping feature.

[0163] Following the example above, the k-th mapping feature and the encoded video feature are subjected to multi-head attention-based decoding to obtain the k-th decoded feature. This includes: performing multi-head attention processing on the k-th mapping feature and the encoded video feature to obtain the k-th multi-head attention feature; and performing residual connection processing on the k-th multi-head attention feature and the k-th mapping feature to obtain the k-th decoded feature.

[0164] For example, the specific decoding process for each layer is as follows: Perform a self-attention operation on the decoding result of the (k-1)th decoding layer to obtain the kth self-attention feature. Add the kth self-attention feature bitwise to the decoding result of the (k-1)th decoding layer, and then normalize the sum to obtain the kth mapping feature. Perform multi-head attention processing on the kth mapping feature and the encoded features of the video to obtain the kth multi-head attention feature. Add the kth multi-head attention feature bitwise to the kth mapping feature, and then normalize the sum to obtain the kth decoding feature. First, increase the dimensionality of the kth decoding feature and then decrease it to obtain the kth non-linear mapping feature. Add the kth non-linear mapping feature bitwise to the kth decoding feature, and then normalize the sum to obtain the decoding result of the kth decoding layer. Finally, output the decoding result of the kth decoding layer through the kth decoding layer.

[0165] In summary, the AI-based comment generation method provided in this application has the following beneficial effects: by fusing features from multiple modalities of a video to obtain diverse coding features, and classifying the video's coding features into video plots to obtain plot type information, and then performing text generation processing based on the video's plot type information and coding features, the rationality and accuracy of video comment generation are improved by utilizing the video's plot type information. Compared to schemes that generate video comments based on video text information, this method saves related computing resources by sharing coding features in both the video plot classification task and the text generation task.

[0166] The following will describe an exemplary application of the embodiments of this application in a real-world application scenario.

[0167] Typically, after a video is published, the platform needs to generate comments to encourage user engagement and increase user time spent. After generating a number of comments, suitable ones are selected and published.

[0168] In related technologies, video comment generation mainly adopts two methods: 1) Encoding video information into a vector and then decoding the vector into a text sequence; 2) Using a sequence-to-sequence structure, the encoder encodes the input text information into a vector, and the decoder uses these encoded vectors to output the comment sequence one by one.

[0169] However, the relevant technologies do not fully combine video and text information, and using a single generation task cannot fully learn the information provided by the input features. That is, the video comment generation task and the video plot type classification task run independently, wasting a lot of computing resources.

[0170] To address the aforementioned issues, this application combines the comment generation task with the video plot type classification task, and integrates video information (i.e., image information) and text information such as video title, tags, OCR-generated text, ASR-generated text, and operational classification type, thereby further improving the rationality and diversity of video comment generation and the accuracy of video plot type classification. By using a shared encoder, the consumption of computing resources can be greatly reduced.

[0171] As mentioned above, the video comment generation scheme provided in this application embodiment can be implemented by various types of electronic devices, such as terminals or servers. The video comment generation scheme of this application embodiment is described in detail below, and its process mainly consists of the following four steps:

[0172] Step 11: Receive information such as the video title filled in by the user when uploading the video.

[0173] Users upload videos and fill in video titles on the application platform. The backend receives the video and title information and inputs it into the video resource review pool to await review.

[0174] Step 12: Video feature extraction.

[0175] Relevant video features are extracted, including video titles, tags, text generated by OCR, text generated by ASR, and video vector information generated by existing video models.

[0176] Step 13: Video comment generation and video plot type classification.

[0177] like Figure 4 As shown, the overall model includes a comment generation model and a video plot type classification model, with both models sharing an encoder. The overall model structure is explained in detail below:

[0178] a) Regarding model input

[0179] First, the text information, including video title, tags, operational categories (e.g., variety shows, TV dramas), text generated by OCR, and text generated by ASR, is segmented into words. These words are then concatenated using the "[SEP]" delimiter to obtain a word sequence. The word sequence begins with the symbol "[CLS]". The length of the word sequence is controlled to be 512 characters; if the length is greater than 512, it is truncated; if it is less than 512, the symbol "[PAD]" is added. The concatenated word sequence is then converted into a vector sequence. An encoder (e.g., a lightweight bidirectional encoder representations from transformers (ALBERT) model) generates a position vector and a type vector for each word vector in the vector sequence. These position and type vectors are then added to the word vectors to obtain the video's text vector. The type vector represents what the vector at that position is (e.g., title, tag, operational category, OCR, ASR).

[0180] Then, the obtained image vector (the features obtained by encoding the image frames of the video) is subjected to a linear transformation and added to the vector of "[CLS]" (i.e. the first symbol in the text vector) to obtain a vector sequence, which is then used as the input of the encoder.

[0181] b) About the encoder

[0182] The encoder in this embodiment can employ an encoding structure within a sequence-to-sequence structure (e.g., a transformer encoder). Through multi-layer attention operations, it transforms the original vectors (i.e., the vector sequence) into vector representations with abstract semantics. Each vector in this representation contains both the semantic information of the words and the contextual information of the current vector. To improve training convergence speed and effectiveness, a pre-trained model like ALBERT can be used as the encoder.

[0183] like Figure 5As shown, the encoder in this embodiment includes a multi-layer transformer encoder layer. The structure of the transformer encoder layer comprises two parts: a self-attention module and a feed-forward module. The self-attention module includes specific self-attention operations and residual connection operations (including add and normalize operations), while the feed-forward module includes specific feed-forward operations and add and normalize operations. After computation by one layer of the transformer encoder layer, each vector deepens the understanding of the full sentence's meaning. Through multi-layer transformer encoder layer computation, the model gradually learns lexical, syntactic, and semantic information.

[0184] The self-attention operation calculates the similarity between any two vectors, normalizes the similarity of each vector to all other vectors into weights that sum to 1, and then weights these weights and sums them with the other vectors to obtain a new vector for each vector. Vectors without self-attention only contain information about themselves and do not carry the meaning of the entire sentence. After self-attention, each vector can acquire information from other vectors, thus obtaining the full sentence information relevant to itself.

[0185] c) On the video plot type classification model

[0186] The video plot type classification model uses the vector at the first position of the encoder's output vector as input. A vector is randomly generated for each video plot type for training. The input vector of the video plot type classification model is multiplied by all plot type vectors, resulting in a score for each plot type. This score is then processed by a sigmoid function to obtain a probability value in the range of 0-1, representing the probability that the video belongs to that plot type. After obtaining the probability values ​​for all video plot types, the video plot types with probabilities greater than a threshold are selected for output.

[0187] During training, the label vector corresponding to the video plot type uses a multi-hot vector, that is, the corresponding position is set to 1 if the video contains a certain video plot type, and 0 otherwise. The loss function is cross-entropy.

[0188] d) Regarding video comment generation models

[0189] The video comment generation model in this embodiment adopts a decoding structure (e.g., a transformer decoder structure) within a sequence-to-sequence structure. The model input includes all vector sequences output by the encoder and the label vectors corresponding to the plot types output by the video plot type classification model. The decoder's initial state uses the aforementioned label vectors corresponding to the plot types to generate comments related to that video plot type. Therefore, multiple comments related to different plot types can be generated for the video. The vector sequences output by the encoder are used for the decoder's attention operation, facilitating the decoder to obtain more relevant information and making the comment generation results more reasonable and coherent. After the model prediction is complete, k comments are selected from each video plot type.

[0190] like Figure 6 As shown, the video comment generation model in this embodiment also includes a multi-layer transformer decoder layer. Compared to the transformer encoder layer, it has an additional encoder-decoder attention module. Self-attention is for sentences with the same input channels, while encoder-decoder attention is for sentences with different input channels. In actual computation, the inputs to the encoder-decoder attention module are: the output vector of the same self-attention layer and the final output vector from the transformer encoder. This allows the semantic and syntactic knowledge carried by the transformer encoder output vector to be integrated into the transformer decoder, making the transformer decoder output a vector that is strongly related to the input sentence.

[0191] like Figure 7 As shown, the specific decoding process will be described below with reference to the transformer decoder and transformer encoder of the embodiments of this application:

[0192] First, the input statement is encoded using a transformer encoder, and a classification task is performed. During decoding, the n video plot vectors with the highest probabilities are selected. These n video plot vectors are used as the initialization vectors for the transformer decoder, and encoding on the decoder side begins. The first word vector (token) is predicted based on the initialization vectors. The second token is predicted based on the initialization vectors and the first token, and so on, until the predicted token is a terminator or the prediction length reaches a threshold. The decoding steps are explained in detail below:

[0193] First, the video plot vector is linearly mapped. The similarity between the mapped video plot vector and the vector sequence output by the transformer encoder is calculated. The similarity score and the vector sequence are used to calculate the vector related to the video plot. This vector is then multiplied by the vocabulary matrix to obtain the probability distribution of the first token. The probability distribution is sorted in descending order, and the top k vectors are used as the first token, inputting them along with the video plot vector into the transformer decoder to predict the second token. This process continues, resulting in an output structure resembling a tree. Each path from the root node to a leaf node represents a comment. The score of each comment is the product of the probabilities of each token along the path. The comment scores are sorted in descending order, and the top k comments are selected, thus obtaining k comments for each video plot type.

[0194] Therefore, after obtaining the comments predicted by the model, a set number of comments are selected from all the output comments according to the application requirements for comment publication.

[0195] In summary, the video comment generation scheme provided in this application has the following beneficial effects:

[0196] 1) By making full use of video information and text information such as video titles, OCR, ASR, and operational classification types, we can not only improve the diversity of video comment generation, but also improve the accuracy of video plot type classification;

[0197] 2) Combining the video comment generation model and the video plot type classification model improves the rationality of comment generation and greatly reduces the consumption of computing resources by reusing the encoder.

[0198] The present application has thus far illustrated the artificial intelligence-based comment generation method provided by the embodiments of this application through exemplary applications and implementations of the electronic devices provided in the embodiments of this application. The embodiments of this application also provide an artificial intelligence-based comment generation apparatus. In practical applications, the functional modules in the artificial intelligence-based comment generation apparatus can be collaboratively implemented by the hardware resources of the electronic device (such as a terminal, server, or server cluster), such as computing resources like processors, communication resources (such as those used to support various communication methods like optical fiber and cellular), and memory. Figure 2 An AI-based comment generation device 555 stored in memory 550 is shown. It can be software in the form of programs and plug-ins, such as software modules designed in programming languages ​​such as C / C++ and Java, application software designed in programming languages ​​such as C / C++ and Java, or dedicated software modules, application programming interfaces, plug-ins, cloud services, etc. in large software systems. Examples of different implementation methods are given below.

[0199] The AI-based comment generation device 555 includes a series of modules, including a feature extraction module 5551, a fusion module 5552, an encoding module 5553, a classification module 5554, and a generation module 5555. The following description further illustrates how the various modules in the AI-based comment generation device 555 provided in this embodiment work together to implement the comment generation scheme.

[0200] The feature extraction module 5551 is used to perform feature extraction processing on the video to obtain features of multiple modalities of the video; the fusion module 5552 is used to perform fusion processing based on the features of multiple modalities of the video to obtain fused features of the video; the encoding module 5553 is used to encode the fused features of the video to obtain encoded features of the video; the classification module 5554 is used to perform video plot classification processing based on the encoded features of the video to obtain plot type information of the video; and the generation module 5555 is used to perform text generation processing based on the plot type information and the encoded features of the video to obtain comment information that matches the plot of the video.

[0201] In some embodiments, the feature extraction module 5551 is further configured to perform text encoding processing on the text information of the video to obtain text features of the video; perform image encoding processing on the image information of the video to obtain image features of the video; and use the text features and image features of the video as features of multiple modalities of the video.

[0202] In some embodiments, the fusion module 5552 is further configured to perform mapping processing on the image features of the video to obtain the image mapping features of the video; and to perform splicing processing on the text features of the video and the image mapping features of the video to obtain the fusion features of the video.

[0203] In some embodiments, the feature extraction module 5551 is further configured to perform word segmentation on the text information of the video to obtain multiple words in the text information; concatenate the multiple words in the text information to obtain a word sequence in the text information; and perform vector transformation on the word sequence in the text information to obtain the text features of the video.

[0204] In some embodiments, the feature extraction module 5551 is further configured to perform vector encoding processing on the word sequence of the text information to obtain the word vector sequence of the text information; obtain the position vectors and type vectors of multiple word vectors in the word vector sequence; and perform encoding processing based on the position vectors and type vectors of the multiple word vectors to obtain the text features of the video.

[0205] In some embodiments, the feature extraction module 5551 is further configured to perform the following processing on any one of the plurality of word vectors: summing the position vector of the word vector, the type vector of the word vector, and the word vector to obtain a summed vector of the word vector; and concatenating the summed vectors of the plurality of word vectors to obtain the text features of the video.

[0206] In some embodiments, the encoding process is implemented through an encoding model, which includes multiple cascaded encoding layers; the encoding module 5553 is further configured to perform attention-based encoding processing on the fusion features of the video through the first encoding layer of the multiple cascaded encoding layers; output the encoding result of the first encoding layer to the subsequent cascaded encoding layers, and continue encoding processing and output encoding results through the subsequent cascaded encoding layers; and use the encoding result output by the last encoding layer as the encoded feature of the video.

[0207] In some embodiments, the encoding module 5553 is further configured to perform the following processing through the j-th encoding layer of the plurality of cascaded encoding layers: perform self-attention processing on the encoding result of the (j - 1)-th encoding layer to obtain the j-th self-attention feature; perform residual connection processing on the j-th self-attention feature and the encoding result of the (j - 1)-th encoder to obtain the j-th residual feature; perform mapping processing on the j-th residual feature to obtain the j-th mapping feature; perform residual connection processing on the j-th mapping feature and the j-th residual feature to obtain the encoding result of the j-th encoding layer; output the encoding result of the j-th encoding layer through the j-th encoding layer; where j is an increasing natural number and the value range is 1 < j ≤ N, N is the number of encoding layers, and N is a positive integer greater than 1.

[0208] In some embodiments, the generation module 5555 is further configured to perform decoding processing on the plot type information of the video and the encoding features of the video to generate word vectors conforming to the plot of the video; perform combination processing on the words corresponding to the word vectors according to the order of the generated word vectors to obtain comment information conforming to the plot of the video.

[0209] In some embodiments, the generation module 5555 is further configured to perform decoding processing on the plot type information of the video and the encoding features of the video to generate the first word vector conforming to the plot of the video; perform decoding processing based on the plot type information of the video, the encoding features of the video, and the previously generated (i - 1) word vectors to generate the i-th word vector conforming to the plot of the video; where i is an increasing natural number and the value range is 1 < i ≤ M, M is the number of word vectors conforming to the plot of the video, and M is a positive integer greater than 1.

[0210] In some embodiments, the decoding processing is implemented by a decoder, and the decoder includes a plurality of cascaded decoding layers; the generation module 5555 is further configured to perform decoding processing on the plot type information of the video, the encoding features of the video, and the previously generated (i - 1) word vectors through the first decoding layer of the plurality of cascaded decoding layers; output the decoding result of the first decoding layer to the subsequent cascaded decoding layers, and continue to perform decoding processing and decoding result output through the subsequent cascaded decoding layers; use the decoding result output by the last decoding layer as the i-th word vector conforming to the plot of the video.

[0211] In some embodiments, the generating module 5555 is further configured to perform the following processing through the k-th decoding layer of the plurality of cascaded decoding layers: perform self-attention-based mapping processing on the decoding result of the (k - 1)-th decoding layer to obtain the k-th mapping feature; perform multi-head attention-based decoding processing on the k-th mapping feature and the encoded feature of the video to obtain the k-th decoding feature; perform non-linear mapping processing on the k-th decoding feature to obtain the k-th non-linear mapping feature; perform residual connection processing on the k-th non-linear mapping feature and the k-th decoding feature to obtain the decoding result of the k-th decoding layer; output the decoding result of the k-th decoding layer through the k-th decoding layer; where k is an increasing natural number and the value range is 1 < k ≤ H, H is the number of decoding layers, and H is a positive integer greater than 1.

[0212] In some embodiments, the generating module 5555 is further configured to perform self-attention processing on the decoding result of the (k - 1)-th decoding layer to obtain the k-th self-attention feature; perform residual connection processing on the k-th self-attention feature and the decoding result of the (k - 1)-th decoding layer to obtain the k-th mapping feature.

[0213] In some embodiments, the generating module 5555 is further configured to perform multi-head attention processing on the k-th mapping feature and the encoded feature of the video to obtain the k-th multi-head attention feature; perform residual connection processing on the k-th multi-head attention feature and the k-th mapping feature to obtain the k-th decoding feature.

[0214] In some embodiments, the classification module 5554 is further configured to perform dot product processing on the encoded feature of the video and a plurality of plot type vectors respectively to obtain the dot product score corresponding to each plot type; perform mapping processing on the dot product score corresponding to each plot type to obtain the probability corresponding to each plot type; perform screening processing based on the probability corresponding to each plot type, and use the screened plot type as the plot type information of the video.

[0215] In summary, the comment generation device based on artificial intelligence provided by the embodiments of the present application has the following beneficial effects: by fusing features of multiple modalities of the video to obtain diversified encoded features, performing video plot classification on the encoded features of the video to obtain the plot type information of the video, and performing text generation processing based on the plot type information of the video and the encoded features of the video, thereby using the plot type information of the video to improve the rationality and accuracy of video comment generation. Compared with the solution of generating video comments based on the text information of the video, by sharing encoded features in the video plot classification task and the text generation task, relevant computing resources are saved.

[0216] This application provides a computer program product or computer program that includes computer instructions stored in a computer-readable storage medium. An electronic device's processor reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the electronic device to perform the AI-based comment generation method described above in this application.

[0217] This application provides a computer-readable storage medium storing executable instructions. When these executable instructions are executed by a processor, they cause the processor to execute the AI-based comment generation method provided in this application. For example... Figures 3A-3C The example shown is an AI-based comment generation method.

[0218] In some embodiments, the computer-readable storage medium may be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or it may be a variety of devices including one or any combination of the above-mentioned memories.

[0219] In some embodiments, executable instructions may take the form of a program, software, software module, script, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a standalone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.

[0220] As an example, executable instructions may, but do not necessarily, correspond to files in a file system. They may be stored as part of a file that holds other programs or data, for example, in one or more scripts in a Hyper Text Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple collaborating files (e.g., a file that stores one or more modules, subroutines, or code sections).

[0221] As an example, executable instructions can be deployed to execute on a single electronic device, or on multiple electronic devices located in one location, or on multiple electronic devices distributed across multiple locations and interconnected via a communication network.

[0222] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, and improvements made within the spirit and scope of this application are included within the scope of protection of this application.

Claims

1. A comment generation method based on artificial intelligence, characterized in that, The method includes: Performing feature extraction processing on the video to obtain features of multiple modalities of the video; Performing fusion processing based on the features of multiple modalities of the video to obtain the fusion features of the video; Performing encoding processing on the fusion features of the video to obtain the encoded features of the video; Performing video plot classification processing based on the encoded features of the video to obtain the plot type information of the video; Performing text generation processing based on the plot type information of the video and the encoded features of the video to obtain comment information conforming to the plot of the video.

2. The method according to claim 1, characterized in that, The performing feature extraction processing on the video to obtain features of multiple modalities of the video includes: Performing text encoding processing on the text information of the video to obtain the text features of the video; Performing image encoding processing on the image information of the video to obtain the image features of the video; Taking the text features of the video and the image features of the video as the features of multiple modalities of the video.

3. The method according to claim 2, characterized in that, The performing fusion processing based on the features of multiple modalities of the video to obtain the fusion features of the video includes: Performing mapping processing on the image features of the video to obtain the image mapping features of the video; Performing splicing processing on the text features of the video and the image mapping features of the video to obtain the fusion features of the video.

4. The method according to claim 1, wherein The encoding processing is implemented by an encoding model, and the encoding model includes multiple cascaded encoding layers; The performing encoding processing on the fusion features of the video to obtain the encoded features of the video includes: Performing attention-based encoding processing on the fusion features of the video through the first encoding layer in the multiple cascaded encoding layers; Outputting the encoding result of the first encoding layer to the subsequent cascaded encoding layers, and continuously performing encoding processing and encoding result output through the subsequent cascaded encoding layers; Taking the encoding result output by the last encoding layer as the encoded features of the video.

5. The method according to claim 4, characterized in that, The continuously performing encoding processing and encoding result output through the subsequent cascaded encoding layers includes: Performing the following processing through the j-th encoding layer in the multiple cascaded encoding layers: Performing self-attention processing on the encoding result of the (j - 1)-th encoding layer to obtain the j-th self-attention feature; Performing residual connection processing on the j-th self-attention feature and the encoding result of the (j - 1)-th encoder to obtain the j-th residual feature; Performing mapping processing on the j-th residual feature to obtain the j-th mapping feature; Performing residual connection processing on the j-th mapping feature and the j-th residual feature to obtain the encoding result of the j-th encoding layer; Outputting the encoding result of the j-th encoding layer through the j-th encoding layer; where j is an increasing natural number and the value range is 1 < j ≤ N, N is the number of the encoding layers, and N is a positive integer greater than 1.

6. The method according to claim 1, characterized in that, The performing text generation processing based on the plot type information of the video and the encoded features of the video to obtain comment information conforming to the plot of the video includes: Decode the plot type information of the video and the encoding features of the video to generate word vectors that conform to the plot of the video; According to the order of the generated word vectors, perform combination processing on the words corresponding to the word vectors to obtain comment information that conforms to the plot of the video.

7. The method according to claim 6, characterized in that, The decoding the plot type information of the video and the encoding features of the video to generate word vectors that conform to the plot of the video includes: Decode the plot type information of the video and the encoding features of the video to generate the first word vector that conforms to the plot of the video; Based on the plot type information of the video, the encoding features of the video, and the previously generated i-1 word vectors, perform decoding processing to generate the i-th word vector that conforms to the plot of the video; where i is an increasing natural number and the value range is 1 < i ≤ M, M is the number of word vectors that conform to the plot of the video, and M is a positive integer greater than 1.

8. The method according to claim 7, wherein The decoding processing is implemented by a decoder, and the decoder includes multiple cascaded decoding layers; The based on the plot type information of the video, the encoding features of the video, and the previously generated i-1 word vectors, perform decoding processing to generate the i-th word vector that conforms to the plot of the video includes: Through the first decoding layer of the multiple cascaded decoding layers, decode the plot type information of the video, the encoding features of the video, and the previously generated i-1 word vectors; Output the decoding result of the first decoding layer to the subsequent cascaded decoding layers, and continue decoding processing and decoding result output through the subsequent cascaded decoding layers; Take the decoding result output by the last decoding layer as the i-th word vector that conforms to the plot of the video.

9. The method according to claim 8, characterized in that, The through the subsequent cascaded decoding layers continue decoding processing and decoding result output includes: Execute the following processing through the k-th decoding layer of the multiple cascaded decoding layers: Perform self-attention-based mapping processing on the decoding result of the k-1-th decoding layer to obtain the k-th mapping feature; Perform multi-head attention-based decoding processing on the k-th mapping feature and the encoding features of the video to obtain the k-th decoding feature; Perform non-linear mapping processing on the k-th decoding feature to obtain the k-th non-linear mapping feature; Perform residual connection processing on the k-th non-linear mapping feature and the k-th decoding feature to obtain the decoding result of the k-th decoding layer; Output the decoding result of the k-th decoding layer through the k-th decoding layer; where k is an increasing natural number and the value range is 1 < k ≤ H, H is the number of decoding layers, and H is a positive integer greater than 1.

10. The method according to claim 9, characterized in that, The perform self-attention-based mapping processing on the decoding result of the k-1-th decoding layer to obtain the k-th mapping feature includes: Perform self-attention processing on the decoding result of the k-1-th decoding layer to obtain the k-th self-attention feature; Perform residual connection processing on the k-th self-attention feature and the decoding result of the k-1-th decoding layer to obtain the k-th mapping feature.

11. The method according to claim 9, characterized in that, The step of performing multi-head attention-based decoding on the k-th mapping feature and the encoded features of the video to obtain the k-th decoded feature includes: Multi-head attention processing is performed on the k-th mapping feature and the encoded features of the video to obtain the k-th multi-head attention feature; The kth multi-head attention feature and the kth mapping feature are subjected to residual connection processing to obtain the kth decoding feature.

12. The method according to claim 1, characterized in that, The video plot classification process based on the encoded features of the video to obtain the plot type information of the video includes: The encoded features of the video are multiplied by multiple plot type vectors to obtain the dot product score for each plot type. The dot product score corresponding to each plot type is mapped to obtain the probability corresponding to each plot type; The filtering process is performed based on the probability corresponding to each plot type, and the filtered plot types are used as the plot type information of the video.

13. A comment generation device based on artificial intelligence, characterized in that, The device includes: The feature extraction module is used to perform feature extraction processing on the video to obtain features of multiple modalities of the video; The fusion module is used to perform fusion processing based on the features of multiple modalities of the video to obtain the fused features of the video; An encoding module is used to encode the fusion features of the video to obtain the encoded features of the video; The classification module is used to perform video plot classification processing based on the encoded features of the video to obtain the plot type information of the video; The generation module is used to perform text generation processing based on the plot type information and encoding features of the video to obtain comment information that matches the plot of the video.

14. An electronic device, characterized in that, The electronic device includes: Memory, used to store executable instructions; A processor, when executing executable instructions stored in the memory, implements the AI-based comment generation method according to any one of claims 1 to 12.

15. A computer-readable storage medium, characterized in that, It stores executable instructions for implementing the AI-based comment generation method according to any one of claims 1 to 12 when executed by a processor.

Images