Dialogue generation method, video comment method, device and equipment, and storage medium

Abstract

The application provides an artificial intelligence-based dialogue processing method, an artificial intelligence-based video comment method, a device, an electronic device and a computer readable storage medium; the method comprises: respectively performing encoding processing on an input sentence and multimedia information to obtain an encoding vector corresponding to the input sentence and an encoding vector corresponding to the multimedia information; performing encoding processing on the encoding vector of the input sentence and the encoding vector of the multimedia information to obtain a feature vector of the multimedia information; performing fusion processing on the encoding vector of the input sentence and the feature vector of the multimedia information to obtain a fusion vector; and performing decoding processing on the fusion vector to obtain a reply sentence used to reply to the input sentence. Through the application, a dialogue can be automatically and accurately generated, and the attraction of interaction with a video is enhanced.

Description

TECHNICAL FIELD

[0001] The present application relates to artificial intelligence technology, and in particular to a dialogue processing method based on artificial intelligence, a video comment method based on artificial intelligence, an apparatus, an electronic device, and a computer readable storage medium. BACKGROUND

[0002] Artificial intelligence (AI) is a comprehensive technology of computer science, which enables machines to have the functions of perception, reasoning and decision-making by studying the design principles and implementation methods of various intelligent machines. Artificial intelligence technology is a comprehensive discipline, involving a wide range of fields, such as natural language processing technology and machine learning / deep learning. With the development of technology, artificial intelligence technology will be applied in more fields and play an increasingly important role.

[0003] In related technologies, there is a lack of effective solutions to simulate real users' dialogue on multimedia information, which mainly relies on manual solutions, such as replying to input sentences discussing multimedia information according to the content of the multimedia information, so that the reply sentences can be connected to the input sentences. However, this way is too inefficient and cannot be applied to online applications. SUMMARY

[0004] The embodiments of the present application provide a dialogue processing method based on artificial intelligence, a video comment method based on artificial intelligence, an apparatus, an electronic device, and a computer readable storage medium, which can automatically and accurately and efficiently form a dialogue for multimedia information.

[0005] The technical solutions of the embodiments of the present application are as follows:

[0006] The embodiments of the present application provide a dialogue generation method based on artificial intelligence, comprising:

[0007] The input sentence and the multimedia information are respectively encoded to obtain an encoding vector corresponding to the input sentence and an encoding vector corresponding to the multimedia information;

[0008] The encoding vectors of the input sentence and the encoding vectors of the multimedia information are encoded to obtain a feature vector of the multimedia information;

[0009] The encoding vectors of the input sentence and the feature vectors of the multimedia information are fused to obtain a fusion vector;

[0010] The fusion vector is decoded to obtain a reply sentence for replying to the input sentence.

[0011] In the technical solution, the decoding processing of the fusion vector is performed to obtain a reply sentence for replying to the input sentence, which comprises:

[0012] The fusion vector and the generated word are linearly mapped to obtain a probability distribution of the next word to be generated;

[0013] The word corresponding to the maximum probability in the probability distribution is determined as the next generated word;

[0014] According to the order of the generated words, the generated words are combined to form a reply sentence for replying to the input sentence.

[0015] In the technical solution, the method further comprises:

[0016] The encoding vector of the input sentence and the encoding vector of the multimedia information in the training sample are restored by the dialogue generation model to obtain the probability of each restored word corresponding to the input sentence in the training sample;

[0017] According to the probability of each restored word corresponding to the input sentence in the training sample, an encoding loss function of the dialogue generation model is constructed;

[0018] The fusion vector in the training sample is predicted by the dialogue generation model to obtain the probability of each generated word;

[0019] According to the probability of each generated word, a decoding loss function of the dialogue generation model is constructed;

[0020] The encoding loss function and the decoding loss function are added to obtain an overall loss function of the dialogue generation model;

[0021] The parameters of the dialogue generation model are updated until the overall loss function converges, and the updated parameters of the dialogue generation model when the overall loss function converges are used as the parameters of the trained dialogue generation model.

[0022] The embodiment of the application provides a video comment method based on artificial intelligence, which comprises:

[0023] Presenting the content in the video;

[0024] Generating comment information in the form of dialogue about the content according to the video information and audio information of the content;

[0025] Presenting the comment information in the form of dialogue.

[0026] The embodiment of the application provides a dialogue generation device based on artificial intelligence, which comprises:

[0027] The encoding module is configured to encode the input sentence and the multimedia information respectively to obtain an encoding vector corresponding to the input sentence and an encoding vector corresponding to the multimedia information.

[0028] The auto-encoding module is configured to encode the encoding vector of the input sentence and the encoding vector of the multimedia information to obtain a feature vector of the multimedia information.

[0029] The decoding module is configured to fuse the encoding vector of the input sentence and the feature vector of the multimedia information to obtain a fusion vector.

[0030] The generating module is configured to decode the fusion vector to obtain a reply sentence for replying to the input sentence.

[0031] In the technical solution, the encoding module is further configured to obtain a semantic vector and a position vector of the input sentence according to input content of the input sentence.

[0032] The semantic vector and the position vector of the input sentence are fused to obtain the encoding vector of the input sentence.

[0033] In the technical solution, the encoding module is further configured to perform word encoding on a word sequence of the input sentence to obtain the semantic vector of the input sentence.

[0034] The positions of words in the input sentence are encoded to obtain the position vector of the input sentence.

[0035] In the technical solution, the multimedia information includes video information and audio information; the encoding module is further configured to obtain a semantic vector and a position vector of the video information according to image content of the video information, and fuse the semantic vector and the position vector of the video information to obtain an encoding vector of the video information.

[0036] A semantic vector and a position vector of the audio information are obtained according to audio content of the audio information, and the semantic vector and the position vector of the audio information are fused to obtain an encoding vector of the audio information.

[0037] In the technical solution, the encoding module is further configured to perform feature extraction on each image frame in the video information to obtain a feature vector of each image frame, and combine the feature vectors of the image frames to obtain the feature vector of the video information.

[0038] The feature vector of the video information is subjected to nonlinear mapping to obtain an intermediate feature vector of the video information.

[0039] position encoding processing is performed on a position of each image frame in the video information to obtain a position vector of the video information;

[0040] The intermediate feature vector of the video information is added to the position vector to obtain an encoded vector of the video information.

[0041] In the technical solution, the encoded vector of the multimedia information includes an encoded vector of the video information and an encoded vector of the audio information; the self-encoding module is further configured to perform self-attention processing on the encoded vector of the input sentence to obtain an attention vector of the input sentence;

[0042] The attention vector of the input sentence and the encoded vector of the video information are subjected to multi-head attention processing to obtain a feature vector of the video information, and

[0043] The feature vector of the video information and the encoded vector of the audio information are subjected to multi-head attention processing to obtain a feature vector of the audio information.

[0044] In the technical solution, the self-encoding module is further configured to perform self-attention processing on the attention vector of the input sentence output by the i-1th encoding layer of the self-encoder through the i th encoding layer of the self-encoder to obtain an attention vector of the input sentence of the i th encoding layer;

[0045] wherein, 1 < i ≤ N, i and N are natural numbers, and N is the total number of encoding layers of the self-encoder;

[0046] The attention vector of the input sentence of the i th encoding layer and the feature vector of the video information output by the i-1th encoding layer are subjected to multi-head attention processing through the i th encoding layer of the self-encoder to obtain a feature vector of the video information of the i th encoding layer;

[0047] The feature vector of the video information of the i th encoding layer and the encoded vector of the audio information output by the i-1th encoding layer are subjected to multi-head attention processing through the i th encoding layer of the self-encoder to obtain a feature vector of the audio information of the i th encoding layer.

[0048] In the technical solution, the encoding module is further configured to perform encoding processing on text information to obtain an encoded vector of the text information;

[0049] The decoding module is further configured to perform decoding processing on the encoded vector of the input sentence, the encoded vector of the text information and the feature vector of the multimedia information through a decoder respectively to obtain a decoding vector corresponding to the input sentence, a decoding vector corresponding to the text information and a decoding vector corresponding to the multimedia information;

[0050] The decoding vector of the input sentence, the decoding vector of the text information, and the decoding vector of the multimedia information are spliced to obtain the fusion vector.

[0051] In the technical solution, the decoding module is further configured to perform self-attention processing on the encoding vector of the text information by using the decoder to obtain the decoding vector of the text information.

[0052] The decoding vector of the text information and the encoding vector of the input sentence are subjected to multi-head attention processing to obtain the decoding vector of the input sentence, and

[0053] The decoding vector of the input sentence and the feature vector of the multimedia information are subjected to multi-head attention processing to obtain the decoding vector of the multimedia information.

[0054] In the technical solution, the feature vector of the multimedia information includes a feature vector of video information and a feature vector of audio information; and the decoding module is further configured to perform multi-head attention processing on the decoding vector of the input sentence and the feature vector of the video information to obtain the decoding vector of the video information.

[0055] The decoding vector of the video information and the feature vector of the audio information are subjected to multi-head attention processing to obtain the decoding vector of the audio information.

[0056] In the technical solution, the generating module is further configured to perform linear mapping on the fusion vector and the generated word to obtain a probability distribution of the next word to be generated.

[0057] The word corresponding to the maximum probability in the probability distribution is determined as the next generated word.

[0058] According to the order of the generated words, the generated words are combined to form a reply sentence for replying to the input sentence.

[0059] In the technical solution, the device further includes:

[0060] The training module is configured to perform restoration processing on the encoding vector of the input sentence and the encoding vector of the multimedia information in the training sample by using the dialogue generation model to obtain the probability of each restored word corresponding to the input sentence in the training sample.

[0061] According to the probability of each restored word corresponding to the input sentence in the training sample, an encoding loss function of the dialogue generation model is constructed.

[0062] The fusion vector in the training sample is subjected to prediction processing by using the dialogue generation model to obtain the probability of each generated word.

[0063] construct a decoding loss function of the dialogue generation model according to the probability of each generated word;

[0064] add the encoding loss function and the decoding loss function to obtain an overall loss function of the dialogue generation model;

[0065] update parameters of the dialogue generation model until the overall loss function converges, and use the updated parameters of the dialogue generation model when the overall loss function converges as the parameters of the trained dialogue generation model.

[0066] An embodiment of the present application provides a video comment device based on artificial intelligence, which comprises:

[0067] a presentation module configured to present content in a video;

[0068] a processing module configured to generate comment information in the form of dialogue about the content according to video information and audio information of the content;

[0069] the presentation module is further configured to present the comment information in the form of dialogue.

[0070] In the technical solution, the processing module is further configured to, when the video is a live video, acquire an input sentence of a comment on the content made by a host;

[0071] invoke a dialogue generation model to perform the following operation: generate one or more reply sentences of virtual audiences for replying to the input sentence according to the video information and the audio information of the content.

[0072] In the technical solution, the processing module is further configured to, when the video is a live video, acquire an input sentence of a comment on the content made by a first audience;

[0073] invoke a dialogue generation model to perform the following operation: generate a reply sentence of a second audience for replying to the input sentence according to the video information and the audio information of the content;

[0074] At least one of the first audience and the second audience is a virtual audience.

[0075] An embodiment of the present application provides an electronic device for dialogue generation, which comprises:

[0076] a memory configured to store executable instructions;

[0077] a processor configured to execute the executable instructions stored in the memory to implement the dialogue generation method based on artificial intelligence provided by an embodiment of the present application.

[0078] The embodiment of the present application provides an electronic device for video comment, and the electronic device comprises:

[0079] a memory for storing executable instructions;

[0080] a processor for executing the executable instructions stored in the memory to realize the video comment method based on artificial intelligence provided by the embodiment of the present application.

[0081] The embodiment of the present application provides a computer readable storage medium, which stores executable instructions for causing a processor to execute to realize the dialogue generation method based on artificial intelligence provided by the embodiment of the present application.

[0082] The embodiment of the present application provides a computer readable storage medium, which stores executable instructions for causing a processor to execute to realize the video comment method based on artificial intelligence provided by the embodiment of the present application.

[0083] The embodiment of the present application has the following beneficial effects:

[0084] By encoding and decoding the input sentence and the multimedia information, a reply sentence for replying to the input sentence is obtained, so that the dialogue for the multimedia information can be automatically generated, the efficiency of the dialogue generation for the multimedia information is improved, and the online application is suitable; Furthermore, the input sentence and the multimedia information are fused, and the obtained fusion vector is decoded to obtain a reply sentence that can accept the input sentence, that is, the knowledge of the multimedia information can be fully utilized, the input sentence for the multimedia information is accurately generated, the accuracy of the reply sentence is improved, and the coherence of the dialogue is improved. BRIEF DESCRIPTION OF DRAWINGS

[0085] Figure 1 is an application scenario schematic diagram of a video comment system 10 provided by the embodiment of the present application;

[0086] Figure 2 is a structural schematic diagram of an electronic device for dialogue generation provided by the embodiment of the present application;

[0087] Figures 3A-3B is a flow schematic diagram of the dialogue generation method based on artificial intelligence provided by the embodiment of the present application;

[0088] Figure 4 is a structural schematic diagram of an electronic device for video comment provided by the embodiment of the present application;

[0089] Figure 5 is a flow schematic diagram of the image recommendation method provided by the embodiment of the present application;

[0090] Figure 6 is an interface schematic diagram of the video live provided by the embodiment of the present application;

[0091] Figure 7 is a structural schematic diagram of a multi-modal Transformer model provided by an embodiment of the present application;

[0092] Figure 8 is a structural schematic diagram of a text sequence encoder provided by an embodiment of the present application;

[0093] Figure 9 is a structural schematic diagram of a video encoder provided by an embodiment of the present application. DETAILED DESCRIPTION

[0094] In order to make the objects, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the drawings, and the described embodiments should not be regarded as limiting the present application, and all other embodiments obtained by those skilled in the art without making creative efforts fall within the scope of protection of the present application.

[0095] In the following description, the terms "first\second" are merely to distinguish similar objects, and do not represent a specific order of the objects. Understandably, the "first\second" can be interchanged in a specific order or sequence as allowed, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0096] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present application belongs. The terms used herein are only for the purpose of describing the embodiments of the present application and are not intended to limit the present application.

[0097] Before the embodiments of the present application are further described in detail, the terms and phrases involved in the embodiments of the present application are explained, and the terms and phrases involved in the embodiments of the present application are applicable to the following explanations.

[0098] 1) Modality: Each source or form of information can be referred to as a modality, for example, information in the form of voice, video, text, etc., each form of information can be referred to as information of a modality. Modalities can be distinguished into single modalities and multi-modalities; single modalities are information represented as numerical vectors that can be processed by computers or further abstracted as higher-level feature vectors, while multi-modalities are better feature representations learned by eliminating the redundancy between modalities through the complementarity between multi-modalities.

[0099] For example, multimedia information can include information of the following modalities: image information (i.e. information of a video modality), voice information (i.e. information of an audio modality); and can also include text information, i.e. information in the form of text.

[0100] 2) Recurrent Neural Network (RNN): a class of recurrent neural networks with sequence data as input, recursion in the evolution direction of sequence, and all nodes (recurrent units) connected in chain. The recurrent neural network has the characteristics of memory, parameter sharing and Turing completeness, so it has certain advantages in learning the nonlinear characteristics of sequence. Recurrent neural networks have applications in natural language processing (NLP), such as speech recognition, language modeling, machine translation, etc.

[0101] The embodiment of the application provides a dialogue processing method and device based on artificial intelligence, a video comment method and device based on artificial intelligence, an electronic device and a computer readable storage medium, which can automatically and accurately generate a reply sentence for replying to an input sentence, thereby improving the efficiency of dialogue generation.

[0102] The following describes an exemplary application of the electronic device for dialogue processing provided by the embodiment of the application.

[0103] The electronic device for dialogue processing provided by the embodiment of the application can be various types of terminal devices or servers. The server can be a standalone physical server, a server cluster or a distributed system composed of multiple physical servers, or a cloud server providing cloud computing services. The terminal can be a smartphone, a tablet computer, a notebook computer, a desktop computer, a smart speaker, a smart watch, etc., but is not limited thereto. The terminal and the server can be directly or indirectly connected through wired or wireless communication, which is not limited in the application. Taking the server as an example, after receiving the input sentence and the multimedia information (video information and audio information), the server calls the encapsulated dialogue generation program, generates a reply sentence for replying to the input sentence according to the input sentence and the multimedia information, thereby automatically generating the reply sentence to form a dialogue, and feeds back the reply sentence to the corresponding client to display the reply sentence in the client, so that the user can see the dialogue about the video in the video and improve the popularity and attention of the video through the generated dialogue.

[0104] Referring to Figure 1 , Figure 1 is a schematic diagram of an application scenario of a video comment system 10 provided by the embodiment of the application. The terminal 200 is connected to the server 100 through the network 300. The network 300 can be a wide area network or a local area network, or a combination of the two.

[0105] The terminal 200 can be used to obtain the input sentence and multimedia information. For example, after a user opens a video through the terminal, the terminal automatically obtains the input sentence and multimedia information (video information and audio information) of the video.

[0106] In some embodiments, the terminal 200 locally executes the AI-based video comment method provided by the embodiments of the present application to automatically generate the comment information in the form of a dialogue. For example, after a user opens a video in a video application (APP) installed on the terminal 200, the terminal 200 presents the content in the video and calls a dialogue generation program to generate comment information in the form of a dialogue about the content according to the video information and audio information of the video content, and displays the comment information on the display interface 210 of the terminal 200, so that the user can see the comments about the video in the video and improve the popularity and attention of the video through the generated comment information.

[0107] In some embodiments, the terminal 200 can also send the address of the video opened on the terminal 200 to the server 100 in the cloud through the network 300 and call the dialogue generation function (encapsulated dialogue generation program) provided by the server 100. The server 100 generates comment information in the form of a dialogue through the AI-based dialogue generation method provided by the embodiments of the present application. For example, after a user opens a video in a video application installed on the terminal 200, the terminal 200 sends the address of the video to the server 100 through the network 300. After the server 100 receives the address of the video, it obtains the video information and audio information of the video content according to the address of the video, calls the encapsulated dialogue generation program, generates comment information in the form of a dialogue about the content according to the video information and audio information of the video content, and returns the comment information to the video application. The comment information is displayed on the display interface 210 of the terminal 200, so that the user can see the comments about the video in the video and improve the popularity and attention of the video through the generated comment information.

[0108] The structure of the electronic device for dialogue generation provided by the embodiments of the present application is described below. The electronic device for dialogue generation can be various terminals such as a mobile phone, a computer, etc., or a server 100 as shown. Figure 1

[0109] Referring to Figure 2 , Figure 2 is a structural schematic diagram of the electronic device 500 for dialogue generation provided by the embodiments of the present application, which is taken as an example of a server, Figure 2 ​The electronic device 500 shown for dialogue generation includes at least one processor 510, a memory 550, at least one network interface 520, and a user interface 530. The various components in the electronic device 500 are coupled together by a bus system 540. It can be understood that the bus system 540 is used to realize the connection communication between the components. In addition to including a data bus, the bus system 540 also includes a power bus, a control bus, and a status signal bus. However, for the sake of clarity, all the buses are marked as the bus system 540 in the Figure 2

[0110] The processor 510 can be an integrated circuit chip with processing capability, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic device, discrete gate or transistor logic device, discrete hardware component, etc., wherein the general-purpose processor can be a microprocessor or any conventional processor.

[0111] The memory 550 includes a volatile memory or a non-volatile memory, and can also include both volatile and non-volatile memories. The non-volatile memory can be a read-only memory (ROM), and the volatile memory can be a random access memory (RAM). The memory 550 described in the embodiments of the present application is intended to include any suitable type of memory. The memory 550 can optionally include one or more storage devices that are physically remote from the processor 510.

[0112] In some embodiments, the memory 550 can store data to support various operations, examples of which include programs, modules, and data structures or subsets or supersets thereof, which are exemplarily illustrated below.

[0113] The operating system 551 includes system programs for processing various basic system services and performing hardware-related tasks, such as a framework layer, a core library layer, a driver layer, etc., for implementing various basic services and processing hardware-based tasks;

[0114] The network communication module 552 is used to reach other computing devices via one or more (wired or wireless) network interfaces 520, and exemplary network interfaces 520 include Bluetooth, wireless compatibility certification (WiFi), and universal serial bus (USB), etc.

[0115] In some embodiments, the dialogue generation device based on artificial intelligence provided by the embodiments of the present application can be realized in a software manner, Figure 2 ​An artificial intelligence-based dialogue generation apparatus 555 stored in the memory 550 is shown, which can be software in the form of programs and plug-ins, and includes a series of modules, including an encoding module 5551, a self-encoding module 5552, a decoding module 5553, a generation module 5554, and a training module 5555; wherein the encoding module 5551, the self-encoding module 5552, the decoding module 5553, and the generation module 5554 are used to implement the functions of dialogue generation provided by the embodiments of the present application, and the training module 5555 is used to implement the training of the dialogue generation model.

[0116] According to the above, it can be understood that the artificial intelligence-based dialogue generation method provided by the embodiments of the present application can be implemented by various types of electronic devices for dialogue generation, such as smart terminals and servers, etc.

[0117] The artificial intelligence-based dialogue generation method provided by the embodiments of the present application will be described below in conjunction with an exemplary application and implementation of a server provided by the embodiments of the present application. Referring to Figure 3A , Figure 3A is a flowchart of the artificial intelligence-based dialogue generation method provided by the embodiments of the present application, which will be described in conjunction with Figure 3A the steps shown.

[0118] In step 101, the input sentence and the multimedia information are respectively encoded to obtain an encoding vector corresponding to the input sentence and an encoding vector corresponding to the multimedia information.

[0119] As an example of obtaining the input sentence and the multimedia information, the audience can open a certain video, such as a certain variety show, on the terminal, and the terminal can forward the address of the video to the server, so that the server obtains the input sentence and the multimedia information (video information and audio information, such as video and audio) about the video according to the address of the video, so as to obtain the encoding vector corresponding to the input sentence and the encoding vector corresponding to the multimedia information according to the input sentence and the multimedia information, and generate a reply sentence for replying to the input sentence according to the encoding vector of the input sentence and the encoding vector of the multimedia information.

[0120] In some embodiments, in order to improve the accuracy of the encoding vector of the input sentence, the encoding processing of the input sentence to obtain the encoding vector of the input sentence includes: obtaining a semantic vector and a position vector of the input sentence according to the input content of the input sentence; and fusing the semantic vector and the position vector of the input sentence to obtain the encoding vector of the input sentence.

[0121] For example, when the server obtains an input sentence, for example, the input sentence is "Do you think the brother is very sporty?", according to the content of the input sentence and the position of each word in the content, the semantic vector and the position vector of each word in the input sentence are obtained, and the semantic vector and the position vector of each word are fused to obtain the encoding vector of the input sentence, so that the content of the input sentence and the position information of each word in the content are integrated into the encoding vector of the input sentence, so that the encoding vector of the input sentence can also represent the position sequence information in the input information, so as to improve the accuracy of the encoding vector of the input sentence, so as to subsequently perform fusion processing and decoding processing according to the accurate encoding vector of the input sentence, and obtain an accurate reply sentence.

[0122] In some embodiments, according to the input content of the input sentence, the semantic vector and the position vector of the input sentence are obtained, including: performing word encoding processing on the word sequence of the input sentence to obtain the semantic vector of the input sentence; and performing position encoding processing on the position of each word in the input sentence to obtain the position vector of the input sentence.

[0123] For example, the word sequence of the input sentence is word-encoded by the word encoder to obtain the semantic vector of the input sentence. The position of each word in the input sentence is position-encoded by the position encoder to obtain the position vector of the input sentence. After obtaining the semantic vector and the position vector of the input sentence, the semantic vector and the position vector of the input sentence are added to obtain the encoding vector of the input sentence, so that the position information of each word in the input sentence is integrated into the encoding vector of the input sentence.

[0124] In some embodiments, the multimedia information includes video information and audio information. In order to improve the accuracy of the encoding vector of the multimedia information, the encoding processing is performed on the multimedia information to obtain the encoding vector of the multimedia information, including: according to the image content of the video information, obtaining the semantic vector and the position vector of the video information, and performing fusion processing on the semantic vector and the position vector of the video information to obtain the encoding vector of the video information; according to the audio content of the audio information, obtaining the semantic vector and the position vector of the audio information, and performing fusion processing on the semantic vector and the position vector of the audio information to obtain the encoding vector of the audio information.

[0125] For example, after the server obtains the video information (image frame sequence) and the audio information (audio frame sequence) of the video, the semantic vector and the position vector of each image frame are obtained according to the image content of the video information and the position of each image frame, and the semantic vector and the position vector of each image frame are fused to obtain the encoding vector of the video information, so that the semantic vector and the position vector of each image frame are integrated into the encoding vector of the video information, so that the encoding vector of the video information can also represent the position sequence information of the image frame, thereby improving the accuracy of the encoding vector of the video information. According to the audio content of the audio information and the position of each audio frame, the semantic vector and the position vector of each audio frame are obtained, and the semantic vector and the position vector of each audio frame are fused to obtain the encoding vector of the audio information, so that the semantic vector and the position vector of each audio frame are integrated into the encoding vector of the audio information, so that the encoding vector of the audio information can also represent the position sequence information of the audio frame, thereby improving the accuracy of the encoding vector of the audio information. In order to subsequently perform subsequent fusion processing and decoding processing according to the accurate encoding vector of the video information and the accurate encoding vector of the audio information, and obtain an accurate reply sentence.

[0126] In some embodiments, according to the image content of the video information, the semantic vector and the position vector of the video information are obtained, and the semantic vector and the position vector of the video information are fused to obtain the encoding vector of the video information, comprising: performing feature extraction processing on each image frame in the video information to obtain a feature vector of each image frame, and combining the feature vector of each image frame to obtain a feature vector of the video information; performing nonlinear mapping processing on the feature vector of the video information to obtain an intermediate feature vector of the video information; performing position encoding processing on the position of each image frame in the video information to obtain a position vector of the video information; and adding the intermediate feature vector and the position vector of the video information to obtain the encoding vector of the video information.

[0127] For example, after the feature vector of each image frame in the video information is extracted, the feature vectors of each image frame are combined to obtain a feature vector of the video information, and the feature vector of the video information is subjected to nonlinear mapping processing to obtain an intermediate feature vector of the video information, thereby changing the vector dimension of the feature vector of the video information to adapt to the vector dimension of the position vector. The position of each image frame in the video information is position encoded by the position encoder to obtain a position vector of the video information. After obtaining the intermediate feature vector and the position vector of the video information, the intermediate feature vector and the position vector of the video information are added to obtain the encoding vector of the video information, so that the position information of each image frame is integrated into the encoding vector of the video information.

[0128] In some embodiments, according to the audio content of the audio information, the semantic vector and the position vector of the audio information are obtained, and the semantic vector and the position vector of the audio information are fused to obtain the encoding vector of the audio information, including: performing feature extraction processing on each audio frame in the audio information to obtain a feature vector of each audio frame, and combining the feature vector of each audio frame to obtain a feature vector of the audio information; performing nonlinear mapping processing on the feature vector of the audio information to obtain an intermediate feature vector of the audio information; performing position encoding processing on the position of each audio frame in the audio information to obtain a position vector of the audio information; and adding the intermediate feature vector of the audio information and the position vector to obtain the encoding vector of the audio information.

[0129] For example, after extracting the feature vector of each audio frame in the audio information, the feature vectors of each audio frame are combined to obtain the feature vector of the audio information, the feature vector of the audio information is subjected to nonlinear mapping processing to obtain the intermediate feature vector of the audio information, so as to change the vector dimension of the feature vector of the audio information to adapt to the vector dimension of the position vector. The position of each audio frame in the audio information is subjected to position encoding by the position encoder to obtain the position vector of the audio information. After obtaining the intermediate feature vector of the audio information and the position vector, the intermediate feature vector of the audio information and the position vector are added to obtain the encoding vector of the audio information, so as to integrate the position information of each audio frame in the encoding vector of the audio information.

[0130] In step 102, the encoding vector of the input sentence and the encoding vector of the multimedia information are subjected to encoding processing to obtain a feature vector of the multimedia information.

[0131] In order to fuse the input sentence and the multimedia information, after the server obtains the encoding vector of the input sentence and the encoding vector of the multimedia information, the encoding vector of the input sentence and the encoding vector of the multimedia information are encoded to obtain a feature vector of the multimedia information integrated with the input sentence, so as to subsequently generate an accurate reply sentence according to the feature vector of the multimedia information.

[0132] In some embodiments, the encoding vector of the multimedia information includes an encoding vector of video information and an encoding vector of audio information; the encoding vector of the input sentence and the encoding vector of the multimedia information are subjected to encoding processing to obtain a feature vector of the multimedia information, including: performing self-attention processing on the encoding vector of the input sentence to obtain an attention vector of the input sentence; performing multi-head attention processing on the attention vector of the input sentence and the encoding vector of the video information to obtain a feature vector of the video information, and performing multi-head attention processing on the feature vector of the video information and the encoding vector of the audio information to obtain a feature vector of the audio information.

[0133] The feature vector of the multimedia information includes a feature vector of video information and a feature vector of audio information. The self-attention mechanism is used to perform self-attention processing on the encoding vector of the input sentence to obtain an attention vector of the input sentence, so that the attention vector of the input sentence pays more attention to important parts in the input sentence. The multi-head attention mechanism is used to perform multi-head attention processing on the attention vector of the input sentence and the encoding vector of the video information to obtain the feature vector of the video information, so that the feature vector of the video information pays more attention to important parts in the input sentence and the image frame. The multi-head attention mechanism is used to perform multi-head attention processing on the feature vector of the video information and the encoding vector of the audio information to obtain the feature vector of the audio information, so that the feature vector of the audio information pays more attention to important parts in the input sentence, the image frame, and the audio frame. That is, the attention mechanism is used to make the feature vector of the video information and the feature vector of the audio information pay more attention to important information, so as to avoid omission of important information of the video information and the audio information by the feature vector of the video information and the feature vector of the audio information.

[0134] In some embodiments, the self-attention processing on the encoding vector of the input sentence to obtain the attention vector of the input sentence includes: performing self-attention processing on the attention vector of the input sentence output by the i-1th encoding layer of the self-encoder by the i th encoding layer of the self-encoder to obtain the attention vector of the input sentence of the i th encoding layer; wherein 1

[0135] For example, the input sentence, the video information and the audio information are encoded by the autoencoder to obtain the feature vector of the video information and the feature vector of the audio information. The autoencoder includes N layers of the same encoding layers. The attention vector of the input sentence output by the i-1th encoding layer of the autoencoder is self-attention processed by the i th encoding layer of the autoencoder to obtain the attention vector of the input sentence of the i th encoding layer, wherein the input of the 1st encoding layer is the encoding vector of the input sentence. The attention vector of the input sentence of the i th encoding layer and the feature vector of the video information output by the i-1th encoding layer are multi-head attention processed by the i th encoding layer of the autoencoder to obtain the feature vector of the video information of the i th encoding layer, wherein the input of the 1st encoding layer is the attention vector of the input sentence of the 1st encoding layer and the encoding vector of the video information. The feature vector of the video information of the i th encoding layer and the encoding vector of the audio information output by the i-1th encoding layer are multi-head attention processed by the i th encoding layer of the autoencoder to obtain the feature vector of the audio information of the i th encoding layer, wherein the input of the 1st encoding layer is the feature vector of the video information of the 1st encoding layer and the encoding vector of the audio information. Through the N layers of encoding layers, the important information of the input sentence, the video information and the audio information is continuously refined, so that the feature vector of the video information and the feature vector of the audio information avoid missing the important information of the video information and the audio information.

[0136] In step 103, the encoding vector of the input sentence and the feature vector of the multimedia information are fused to obtain a fusion vector.

[0137] In order to obtain a reply sentence for replying to the input sentence according to the input sentence and the multimedia information, after obtaining the encoding vector of the input sentence and the feature vector of the multimedia information on the server, the encoding vector of the input sentence and the feature vector of the multimedia information are fused to obtain a fusion vector, which can be decoded to generate a reply sentence.

[0138] In some embodiments, in order to fuse various modal information of the dialogue, the text information can also be fused in the fusion vector. Therefore, the text information is encoded to obtain an encoding vector of the text information before obtaining the fusion vector. Thus, the encoding vector of the input sentence and the feature vector of the multimedia information are fused to obtain a fusion vector, including: the encoding vector of the input sentence, the encoding vector of the text information and the feature vector of the multimedia information are decoded by the decoder respectively to obtain the decoding vector corresponding to the input sentence, the decoding vector corresponding to the text information and the decoding vector corresponding to the multimedia information; the decoding vector of the input sentence, the decoding vector of the text information and the decoding vector of the multimedia information are spliced to obtain the fusion vector.

[0139] For example, the text information includes a content summary of the video, a historical conversation about the video, and a title of the video, etc. The word sequence of the text information is word-encoded by a word encoder to obtain a semantic vector of the text information. The position of each word in the text information is position-encoded by a position encoder to obtain a position vector of the text information. After obtaining the semantic vector and the position vector of the text information, the semantic vector and the position vector of the text information are added to obtain an encoding vector of the text information, so that the position information of each word in the input sentence is integrated into the encoding vector of the text information.

[0140] Wherein, after the server obtains the encoding vector of the text information, the encoding vector of the input sentence, the encoding vector of the text information and the feature vector of the multimedia information are respectively decoded by the decoder to obtain the decoding vector corresponding to the input sentence, the decoding vector corresponding to the text information and the decoding vector corresponding to the multimedia information, and then the decoding vector of the input sentence, the decoding vector of the text information and the decoding vector of the multimedia information are spliced to obtain the fusion vector.

[0141] In some embodiments, the decoding vector corresponding to the input sentence, the decoding vector corresponding to the text information and the decoding vector corresponding to the multimedia information are obtained by respectively decoding the encoding vector of the input sentence, the encoding vector of the text information and the feature vector of the multimedia information by the decoder, including: the decoding vector of the text information is obtained by self-attention processing of the encoding vector of the text information by the decoder; the decoding vector of the input sentence is obtained by multi-head attention processing of the decoding vector of the text information and the encoding vector of the input sentence, and the decoding vector of the multimedia information is obtained by multi-head attention processing of the decoding vector of the input sentence and the feature vector of the multimedia information.

[0142] The encoding vector of the text information is self-attention processed through the self-attention mechanism to obtain the attention vector of the input sentence, so that the decoding vector of the text information pays more attention to important parts in the encoding vector of the text information. The decoding vector of the text information and the encoding vector of the input sentence are multi-head attention processed through the multi-head attention mechanism to obtain the decoding vector of the text information, so that the decoding vector of the text information pays more attention to important parts in the decoding vector of the text information and the encoding vector of the input sentence. The decoding vector of the input sentence and the feature vector of the multimedia information are multi-head attention processed through the multi-head attention mechanism to obtain the decoding vector of the multimedia information, so that the decoding vector of the multimedia information pays more attention to important parts in the decoding vector of the input sentence and the feature vector of the multimedia information. That is, through the attention mechanism, the decoding vector of the text information, the decoding vector of the input sentence and the decoding vector of the multimedia information pay more attention to important information, thereby avoiding missing important information of the encoding vector of the text information, the encoding vector of the input sentence and the feature vector of the multimedia information.

[0143] For example, the encoding vector of the input sentence and the feature vector of the multimedia information are fused by the decoder to obtain a fusion vector. The decoder includes N layers of the same decoding layer. The attention vector of the text information output by the i-1th decoding layer is self-attention processed by the i th decoding layer of the decoder to obtain the attention vector of the text information of the i th decoding layer, wherein the input of the 1st decoding layer is the encoding vector of the text information, and the output of the N th decoding layer is the decoding vector of the text information. The attention vector of the text information of the i th decoding layer and the decoding vector of the input sentence output by the i-1th decoding layer are multi-head attention processed by the i th decoding layer of the decoder to obtain the decoding vector of the input sentence of the i th decoding layer, wherein the input of the 1st decoding layer is the attention vector of the text information of the 1st decoding layer and the encoding vector of the input sentence. The decoding vector of the input sentence of the i th decoding layer and the decoding vector of the multimedia information output by the i-1th decoding layer are multi-head attention processed by the i th decoding layer of the decoder to obtain the decoding vector of the multimedia information of the i th decoding layer, wherein the input of the 1st decoding layer is the decoding vector of the input sentence of the 1st decoding layer and the feature vector of the multimedia information. Through the N decoding layers, important information of the input sentence, video information and audio information is constantly refined, thereby avoiding missing important information of the text information and the multimedia information.

[0144] When the feature vector of the multimedia information comprises a feature vector of video information and a feature vector of audio information, the decoding vector of the input sentence and the feature vector of the multimedia information are subjected to multi-head attention processing to obtain a decoding vector of the multimedia information, comprising: subjecting the decoding vector of the input sentence and the feature vector of the video information to multi-head attention processing to obtain a decoding vector of the video information; and subjecting the decoding vector of the video information and the feature vector of the audio information to multi-head attention processing to obtain a decoding vector of the audio information.

[0145] In step 104, the fusion vector is decoded to obtain a reply sentence for replying to the input sentence.

[0146] After the server obtains the fusion vector of the video, the fusion vector can be predicted by the generation network to obtain a reply sentence for replying to the input sentence. The server feeds back the reply sentence to the terminal, and after the terminal receives the reply sentence, the reply sentence is presented on the interface of playing the video, which can be presented in the form of a barrage or in the form of a list.

[0147] In some embodiments, the decoding of the fusion vector obtains a reply sentence for replying to the input sentence, comprising: linearly mapping the fusion vector and the generated word to obtain a probability distribution of the next generated word; determining the word corresponding to the maximum probability in the probability distribution as the next generated word; and according to the order of the generated words, the generated words are combined to form the reply sentence for replying to the input sentence.

[0148] For example, when the first word needs to be generated, the fusion vector and the start symbol are linearly mapped to the word library to obtain a probability distribution of the first generated word, and the word corresponding to the maximum probability in the probability distribution of the first generated word is determined as the first generated word, and the first generated word is used to continue to generate the next word. The fusion vector and the generated word are linearly mapped to the word library to obtain a probability distribution of the next generated word, and the word corresponding to the maximum probability in the probability distribution is determined as the next generated word, and the generated words are combined to form the reply sentence for replying to the input sentence.

[0149] Referring to Figure 3B , Figure 3B is an optional flowchart of the dialogue generation method based on artificial intelligence provided by the embodiments of the present application. In order to generate a reply sentence by using the trained dialogue generation model, the dialogue generation model needs to be trained, and then Figure 3BFurther comprising steps 105-110: in step 105, by the dialogue generation model, the encoding vector of the input sentence in the training sample and the encoding vector of the multimedia information are restored to obtain the probability of each restored word of the input sentence corresponding to the training sample; in step 106, according to the probability of each restored word of the input sentence corresponding to the training sample, the encoding loss function of the dialogue generation model is constructed; in step 107, by the dialogue generation model, the fusion vector in the training sample is predicted to obtain the probability of each generated word; in step 108, according to the probability of each generated word, the decoding loss function of the dialogue generation model is constructed; in step 109, the encoding loss function and the decoding loss function are added to obtain the overall loss function of the dialogue generation model; in step 110, the parameters of the dialogue generation model are updated until the overall loss function converges, and the updated parameters of the dialogue generation model when the overall loss function converges are used as the parameters of the trained dialogue generation model.

[0150] Wherein, steps 105-110 and steps 101 to step 104 have no obvious order. When the server generates each restored word of the input sentence in the training sample (the restored word corresponds to the original input sentence, and the more similar the restored word is to the original input sentence, the better the restoration effect is, and the more accurate the feature vector of the generated multimedia information is) and the generated reply sentence, according to the probability of each restored word of the input sentence in the training sample and the probability of each generated word in the generated reply sentence, the value of the overall loss function of the dialogue generation model is determined. When the value of the overall loss function exceeds the preset threshold, the error signal of the dialogue generation model is determined based on the overall loss function, the error information is back propagated in the dialogue generation model, and the model parameters of each layer are updated in the propagation process.

[0151] Here, the back propagation is explained. The training sample data is input to the input layer of the neural network model, passes through the hidden layer, finally reaches the output layer and outputs the result, which is the forward propagation process of the neural network model. Since there is an error between the output result of the neural network model and the actual result, the error between the output result and the actual value is calculated, and the error is back propagated from the output layer to the hidden layer until it propagates to the input layer. In the process of back propagation, the value of the model parameter is adjusted according to the error; the above process is iterated continuously until convergence, wherein the dialogue generation model belongs to the neural network model.

[0152] The video comment method based on artificial intelligence provided by the embodiment of the application is described below in combination with the exemplary application and implementation of the terminal provided by the embodiment of the application. Referring to Figure 4 , Figure 4 is a structural schematic diagram of an electronic device 600 for video comment provided by the embodiment of the application, Figure 4The electronic device 600 for video comment shown includes at least one processor 610, a memory 650, at least one network interface 620 and a user interface 630. Among them, the functions of the processor 610, the memory 650, the at least one network interface 620 and the user interface 630 are similar to those of the processor 510, the memory 550, the at least one network interface 520 and the user interface 530, that is, the functions of the operating system 651 and the network communication module 652 are similar to those of the operating system 551 and the network communication module 552, and details are not repeated.

[0153] In some embodiments, the image recommendation device provided by the embodiments of the present application can be realized in a software manner, Figure 4 An artificial intelligence-based video comment device 655 stored in the memory 650 is shown, which can be software in the form of programs and plug-ins, and includes a series of modules, including a presentation module 6551 and a processing module 6552; wherein the presentation module 6551 and the processing module 6552 are used to realize the artificial intelligence-based video comment method provided by the embodiments of the present application.

[0154] The image recommendation method provided by the embodiments of the present application will be described below in conjunction with exemplary applications and implementations of the terminal provided by the embodiments of the present application. Referring to Figure 5 , Figure 5 is a flowchart of the artificial intelligence-based video comment method provided by the embodiments of the present application, which is described in conjunction with Figure 5 the steps shown.

[0155] In step 201, the content in the video is presented.

[0156] For example, the audience can open a certain video, such as a certain variety show video or live video, on a terminal (including a client, such as a video client), and then present the content in the video, such as the variety show program of the variety show video or the live content of the live video, on the display interface of the terminal.

[0157] In step 202, according to the video information and the audio information of the content, the comment information in the form of dialogue about the content is generated.

[0158] For example, while displaying video content on the terminal, the terminal can invoke a dialogue generation model. This model integrates video and audio information from the content to generate a dialogue-style comment about the content. The input statement (the current dialogue input, which can be a blank statement or a predefined statement) and multimedia information (video and audio information from the content) are encoded separately to obtain the corresponding encoding vectors for the input statement and the multimedia information. These encoding vectors are then further encoded to obtain the feature vectors for the multimedia information. Finally, the encoded vectors for the input statement and the feature vectors for the multimedia information are fused together, and the resulting fused vector is decoded to produce a response statement to the input statement, thus forming a dialogue-style comment about the content.

[0159] like Figure 5 As shown, step 202 can be implemented through steps 2021-2022. In step 2021, when the video is a live video, the input statement of the comment posted by the host is obtained; in step 2022, the dialogue generation model is invoked to perform the following operations: based on the video and audio information of the content, one or more virtual viewers are generated to respond to the input statement.

[0160] For example, when the video is a live video, the input statement of the comment made by the host in the current live video can be obtained, and the video information and audio information of the content can be integrated through the dialogue generation model to generate one or more virtual viewers to reply to the input statement, thereby simulating that there are many viewers in the live video room, thus increasing the popularity of the live video.

[0161] In this process, after the dialogue generation model generates one or more virtual viewers to respond to the input statement, the response statement is used as the new input statement. Combined with the content presented in real time in the live video (video information and audio information), a new response statement is generated. This allows the generated dialogue-style comment information to continue, greatly increasing the attention of new or less popular streamers and preventing the live video from becoming dull.

[0162] In some embodiments, generating comment information in the form of a dialogue about the content based on the video and audio information of the content includes: when the video is a live video, obtaining the input statement of the comment content posted by a first viewer; invoking a dialogue generation model to perform the following operations: generating a reply statement for a second viewer to reply to the input statement based on the video and audio information of the content; wherein at least one of the first viewer and the second viewer is a virtual viewer.

[0163] For example, when the video is a live video, the input sentence of the comment content published by the first audience in the current live video can be obtained, and the video information and the audio information of the content are fused by the dialogue generation model to generate a reply sentence of the second audience for replying to the input sentence, so that it is simulated that many audiences are watching the live video in the live room, thereby improving the popularity of the live video.

[0164] After the dialogue generation model generates the reply sentence of the one or more virtual audiences for replying to the input sentence, the reply sentence is taken as a new input sentence, and the new reply sentence is generated by combining the content (video information and audio information) presented in real time by the live video, so that the generated comment information in the form of dialogue continues, greatly improves the attention of the new anchor or the anchor with low popularity, and avoids the cold field of the live video.

[0165] As an example of the input sentence of the comment content published by the first audience, the video information and the audio information of the content can be encoded by the dialogue generation model to obtain an encoded vector of the video information and an encoded vector of the audio information, the encoded vector of the video information and the encoded vector of the audio information are encoded to obtain a feature vector of the video information and a feature vector of the audio information, the feature vector of the video information and the feature vector of the audio information are fused, and the obtained fusion vector is decoded to obtain the input sentence of the comment content published by the first virtual audience.

[0166] After the dialogue generation model generates the reply sentence of the second audience for replying to the input sentence, the reply sentence of the second audience for replying to the input sentence is taken as a new input sentence, and the reply sentence of the first audience for replying to the new input sentence is generated by combining the content (video information and audio information) presented in real time by the live video, so that the dialogue between the first audience and the second audience continues, greatly improves the attention of the new anchor or the anchor with low popularity, and avoids the cold field of the live video.

[0167] In step 203, the comment information in the form of dialogue is presented.

[0168] After the dialogue generation model simulates the comment information in the form of dialogue, the comment in the form of dialogue can be presented in the video in the form of a list or a barrage, so that it is simulated that a large number of virtual audiences are watching the video, thereby improving the popularity of the video.

[0169] In the following, an exemplary application of the embodiment of the present application in an actual application scenario will be described.

[0170] Video live broadcast as a new entertainment way attracts a large number of users in recent years. During the video live broadcast, the user can interact with the host or other users in the live room through the way of text comments, which can increase the activity of the live room. However, the new host or the live work with low popularity often lacks comments and attention. By automatically generating comments (reply information for input information) of the live video, the attention of the new host or the low-popularity host can be effectively improved.

[0171] Therefore, the embodiment of the present application can be applied to the application scenario of video live broadcast, such as Figure 1 As shown in the figure, the terminal 200 is connected to the server 100 deployed in the cloud through the network 300. The video live broadcast application is installed on the terminal 200. The audience opens a certain live room in the video live broadcast application. The terminal 200 sends the address of the live room to the server 100 through the network 300. After the server 100 receives the address of the live room, the related information of the live room is determined according to the address of the live room, such as the audio and video currently played in the live room, the historical comment information of the live room, the summary of the live video, etc. According to the related information of the live room, the reply information corresponding to the current comment information of the live room is determined, and the reply information is returned to the video live broadcast application and displayed on the display interface 210 of the terminal 200, so that the audience can see various comment information about the live room, so as to improve the popularity and attention of the live room.

[0172] As shown in the figure, Figure 6In the live broadcast example shown, interface 601 displays a summary or title of the live broadcast, embodying the core content of the live broadcast, for example, according to the title "Handsome young man shows you dynamic outfit, fashion sports, wait for you to choose and purchase", it can be known that the core content of the live broadcast is related to the outfit of men, interface 602 displays a continuous time picture during the live broadcast, and interface 603 displays a question and answer comment about the live broadcast. Among them, the comments generated in the embodiment of the application can be simulated question and answer comments between users, can also be reply information to the anchor, and can also be reply information to the user, for example, "Q1: Wow, XX's pants feel very youthful!" in interface 603 is input information of the anchor, in response to the input information of the anchor, the reply information "Yes, XX's pants use very comfortable micro-pulmonary trendy fabric, which is the main push" is generated; "Q1: Wow, XX's pants feel very youthful!" in interface 603 is simulated input information of the user, and in response to the simulated input information of the user, the reply information "Yes, XX's pants use very comfortable micro-pulmonary trendy fabric, which is the main push" is automatically generated; "Q1: Wow, XX's pants feel very youthful!" in interface 603 is input information of a user, and in response to the input information of the user, the reply information "Yes, XX's pants use very comfortable micro-pulmonary trendy fabric, which is the main push" is automatically generated. Therefore, by automatically generating reply information of the live broadcast video, the activity level of the live broadcast room can be effectively increased to improve the attention of new anchors or anchors with low popularity. Thus, other audiences can be attracted, and the anchor can be encouraged to publish more excellent works through popular comments.

[0173] In the related art, three methods are mainly used to automatically generate comments, which are: a neural encoder-decoder model for visual comment generation can automatically generate comments, the neural encoder-decoder model includes three encoders, namely a late fusion encoder, a hierarchical recurrent encoder and a memory network, and two decoders (a generation network and a discriminator network); a video comment generation model based on supervised learning is extended, and a deep reinforcement learning technique is used to optimize the comment generation strategy of the video comment generation model, so as to automatically generate comments through the video comment generation model; an input perception attention mechanism is adopted, so that the video comment generation model can focus on the most relevant part of the dialogue context, so as to automatically generate comments through the video comment generation model.

[0174] Wherein, the generation of the video comment needs to generate a comment according to the visual (image) and voice of a video. Compared with the comment generation task based on pictures or text alone, the video-based comment generation task is more challenging. The difficulties mainly have two aspects: 1) the feature space of the video spans multiple pictures, and it is difficult to obtain semantic information; 2) the video comment generation must be able to process information from different modalities (audio, video, subtitles, etc.) to obtain comprehensive understanding.

[0175] Although the related art can automatically generate comments. However, due to the complexity of the long sequence information of multiple modalities, the related art will lose a lot of original information when processing the long sequence information of the video frame. In order to solve the above problem, the embodiment of the application proposes a multi-modal transformer (Transformer) model (dialog generation model), which can model the complex sequence information in the video frame, and can also fuse different modal information, and an attention mechanism is designed in the autoencoder for extracting features related to the input text from non-text mode. In the task of generating video comments, the multi-modal Transformer model can perform complex reasoning on multi-modal information by jointly processing information in different feature spaces, so as to more easily fuse information from different modalities, avoid losing original information, and improve the accuracy of the generated comment information.

[0176] Wherein, the video live room mainly contains the visual (image, video) part, the voice (sound, audio) part and the user's comment (text) of a video. In addition, there is some additional information, for example, the content introduction or summary of the live video. Among them, the user's comment is composed of multiple texts, and each text is a user's comment text. Multiple user comments constitute the context history of the dialogue. Therefore, the embodiment of the application can generate a user comment at the current time according to the above given information, that is, given a video input V, a video title C, a context of a dialogue (historical dialogue) {(Q1, A1),..., (Q t-1 ,A t-1 )} and the input text Q t at the current time, a reply A tFor example, the context of the dialogue here can be two ends of the text {Q1: Wow, the pants of XX feel very young! A1: Yes, XX uses very comfortable and fashionable fabric with a little stretch, which is the main promotion.}, the input text of the current time is "Q2: Do you think the brother is very sporty?", and the task of the multi-modal Transformer model is to output the current time comment "A2: Yes, I also want to be so energetic!" The embodiment of the application proposes a new architecture multi-modal Transformer network based on the Transformer network for fusing multi-modal features. The multi-modal Transformer network can realize complex reasoning on video long sequence information by focusing on the features of different modalities. In the embodiment of the application, the multi-modal Transformer model mainly considers three modalities of text, speech and image.

[0177] The multi-modal Transformer model in the embodiment of the application includes three modules: an encoding layer, a decoding layer and a self-encoding layer. Among them, the encoding layer is mainly used for encoding the video input (text input, speech input and image input) to obtain the continuous vector representation of the input information, and also encodes the positions of the video sequence and the text sequence through a position encoder, so as to obtain the position information of different time sequences; the decoding layer is responsible for generating the target sequence, and in the decoding process, the decoding layer will use a multi-head attention mechanism to reason on multiple encoding features; the self-encoding layer uses an attention mechanism with input perception to increase the visual and speech features of the video. As shown in the overall architecture of the multi-modal Transformer model, Figure 7 in the test mode, the multi-modal Transformer model first encodes the dialogue history, the video title, the input text, the visual information of the video and the speech information of the video through the encoding layer to obtain the corresponding five vector representations, that is, the vector representation z his of the dialogue history, the vector representation z cap of the video title, the vector representation z s of the input text, the vector representation f v of the visual information of the video and the vector representation f a of the speech information of the video. Subsequently, the self-encoding layer of the multi-modal Transformer model uses three attention modules respectively to take z s , f v and f a as inputs to obtain the visual feature vector and the speech feature vector of the video. Finally, the decoding layer takes z his , z cap , z s , and At the same time as input, and output the final vector representation m s , the subsequent generation network generates the comment A s according to m t (reply information). In addition, during the training of the multi-modal Transformer model, the auto-encoding layer restores the input text according to the vector representation of the video feature (f v and f a ) and the vector representation of the input text (z s ) so that the video feature vectors f v and f a can introduce more information of the input text.

[0178] The encoding layer, the decoding layer, the auto-encoding layer and the generation network in the multi-modal Transformer model will be introduced in detail below:

[0179] A) Encoding layer

[0180] The encoding layer includes two modules: text sequence encoder and video encoder. Among them, the dialogue history encoder, the video summary encoder and the input text encoder are text sequence encoders, and the video (vision / image) encoder and the video (speech) encoder are video encoders.

[0181] Among them, the text sequence encoder is used to encode the dialogue history, the video title and the input text to obtain the corresponding vector representation of the dialogue history z his , the vector representation of the video title z cap and the vector representation of the input text z s . The text sequence encoder maps each input text sequence (x1,...,x n ) (for example, the text sequence is (Q2: Do you think the brother is very sporty?)) to a continuous vector representation z=(z1,...,z n )∈R d . For example, the text sequence encoder can be a pre-trained BERT model. Figure 8The structure of the text sequence encoder shown is composed of three parts: a word encoder, a position encoder, and a layer normalization module. The word encoder is used to obtain the semantic information of the input text, the position encoder is used to fuse the sequence information of the source input, and the layer normalization module is used to accelerate model convergence and improve the feature vector representation capability. Both the word encoder and the position encoder use the same dimensional vector representation. Finally, the information of the text sequence is obtained by adding the vector representations output by the word encoder and the position encoder. The position encoder is used to calculate the position embedding (PE), and the calculation formula is shown in formula (1):

[0182] PE(pos,2i)=sin(pos / 10000 2i (1)

[0183] Here, sin() represents the sine trigonometric function, pos represents the position of the word, and i represents the dimension of the position vector.

[0184] The video encoder is used to encode the visual and audio information of the video to obtain a vector representation f of the visual information of the video. v The vector representation f of the audio information of the video a For a given video input V, this embodiment of the invention uses a sliding window of length v frames to extract video features. f m This represents a feature vector of a video sequence with length v frames, where m can represent either visual (v) or speech (a). This embodiment of the invention considers features from both visual and speech perspectives. For example... Figure 9 The overall structure of the video encoder shown is for a set of video sequences s1,...,s v The feature vector of mode m obtained by the video encoder is f m =(f1,...,f v The video encoder first uses a linear network and the ReLU activation function to process the data obtained after feature extraction from the video. m The 1-dimensional feature vector is transformed into a d-dimensional feature vector. Subsequently, the video encoder also uses a position encoder to obtain sequence information and adds this sequence information to the d-dimensional feature vector to obtain the feature vector f. m .

[0185] B) Autoencoder layer

[0186] like Figure 7As shown, since the multi-head attention mechanism selects the more important part of the input information through dynamic programming, the video features obtained by using the attention mechanism may not be optimal. Therefore, the embodiment of the present application uses an independent attention calculation module (i.e., an input text self-encoder) for the input text to strengthen the video feature representation. The input text self-encoder can focus on encoding the information of the input text, and the input text self-encoder is implemented in an unsupervised manner. The self-encoding layer includes N layers of networks in total, each layer of network includes an input text self-attention module and a video attention module for the input text (a visual attention module and a speech attention module), i.e., the number of sub-modules of each layer of network is 3. For the self-attention module, the parameters q, k and v are the same, and the output of the previous layer is used, and the calculation formula of the multi-head self-attention mechanism is as shown in formulas (2) and (3):

[0187]

[0188]

[0189] wherein, is the output of the self-attention module of the n-th layer of network in the self-encoding layer, Attn() represents a multi-head self-attention function, h i,n+1 is the output of the word attention module of the n+1-th layer of network in the self-encoding layer.

[0190] For the video attention module (the visual attention module and the speech attention module) for the input text, the parameters q, k and v are not the same, q is the output of the input text self-attention module of the current layer, k and v are the outputs of the video attention module of the previous layer, and the calculation formula of the multi-head self-attention mechanism is as shown in formula (4):

[0191]

[0192] wherein, is the output of the self-attention module of the n+1-th layer of network in the self-encoding layer, Attn() represents a multi-head self-attention function, is the output of the video attention module of the n-th layer of network in the self-encoding layer, h i,n+1 is the output of the video attention module of the n+1-th layer of network in the self-encoding layer. The output of the video attention module of the N-th layer of network in the self-encoding layer is the visual feature vector of the video and the speech feature vector

[0193] C) Decoding layer

[0194] When a vector representation z s of an input text is given, the decoding layer (decoder) generates a video representation xs An output sequence y1,...,y m The decoder comprises N identical modules, each of which comprises five sub-modules, namely a dialogue history attention module, a video summary attention module, an input text attention module, a visual attention module, and a semantic attention module. Each of the sub-modules uses an independent attention mechanism to encode the vector representation z his of the video title z cap , the vector representation z s of the input text z , the visual feature vector of the video , and the speech feature vector . The dialogue history attention module comprises a multi-head self-attention mechanism, and the calculation formula for encoding the vector z his is shown in equation (5):

[0195]

[0196] wherein, is the output of the dialogue history attention module of the n-th network in the decoding layer, Attn() represents a multi-head self-attention function, h i,n+1 represents the output of the dialogue history attention module of the n+1-th network in the self-encoding layer.

[0197] The video summary attention module, the input text attention module, the visual attention module, and the semantic attention module each comprise a multi-head self-attention mechanism, and the calculation formula for encoding the vector z(z cap , z s , or ) is shown in equation (6):

[0198]

[0199] wherein, Attn() represents a multi-head self-attention function, z m,n represents the output of the current attention module of the n-th network in the decoding layer, is the output of the previous attention module of the n+1-th network in the decoding layer, h i,n+1 represents the output of the current attention module of the n+1-th network in the decoding layer, for example is the output of the input text attention module of the n+1-th network in the decoding layer, then z m,n represents the output of the visual attention module of the n-th network in the decoding layer, h i,n+1This represents the output of the visual attention module in the (n+1)th layer of the decoding layer. Specifically, the outputs of the dialogue history attention module, video summarization attention module, input text attention module, visual attention module, and semantic attention module in the Nth layer of the decoding layer are: h1, h2, h3, h4, and h5.

[0200] The formula for calculating the fusion vector output by the decoding layer is shown in formula (7):

[0201] m s =Concat(h1,...,h5)W O (7)

[0202] in, Concat() represents a concatenation operation, m s This represents the final vector representation of the decoding layer, and then m... s Input the generator network to produce text output.

[0203] In this invention, residual networks can be added to the decoder and encoder to improve model performance. The Transformer network can be a CNN network, and the autoencoder can be a variational autoencoder. The embodiments of this invention are not limited to the structures of the decoder, encoder, Transformer network, and autoencoder.

[0204] D) Generating Network

[0205] like Figure 7 As shown, the generator network corresponds to the linear transformation layer and activation function (Linear & Softmax) modules in the multimodal Transformer model structure diagram. Based on the vector m obtained from the decoding layer... s The probability distribution for generating the next word is calculated sequentially.

[0206] When training a multimodal Transformer model, given the vector representation z of the input text for the training samples... s and the vector representation of the target output z t (Tags) The Linear & Softmax module ultimately generates an output sequence y1,...,y m In the autoencoder layer, the Linear & Softmax module is also used to reconstruct the input text based on the vectors of the input text and video features. Therefore, the loss function of the multimodal Transformer model during training is the sum of the logarithmic losses of the decoding and encoding layers, as shown in formula (8):

[0207]

[0208] Wherein, L represents the overall loss, L(T) represents the decoding loss of the decoding layer, L(Q) represents the encoding loss of the auto-encoding layer, y1,..., y m-1 ,y m represent words generated in sequence during training, x1,..., x n-1 ,x n represent restored words restored in sequence during training.

[0209] To sum up, the embodiment of the present application proposes a multi-modal Transformer model, which can generate an output sequence, i.e., a comment sentence or a reply sentence, according to an input text, so as to automatically generate user comments in a live room and increase the activity of the live room.

[0210] Thus, the exemplary application and implementation of the server provided by the embodiment of the present application have been combined to describe the conversation generation method based on artificial intelligence provided by the embodiment of the present application, and the scheme of cooperation of each module in the conversation generation device 555 based on artificial intelligence provided by the embodiment of the present application will be described below.

[0211] The encoding module 5551 is configured to encode the input sentence and the multimedia information respectively to obtain an encoding vector corresponding to the input sentence and an encoding vector corresponding to the multimedia information; the auto-encoding module 5552 is configured to encode the encoding vector of the input sentence and the encoding vector of the multimedia information to obtain a feature vector of the multimedia information; the decoding module 5553 is configured to fuse the encoding vector of the input sentence and the feature vector of the multimedia information to obtain a fusion vector; and the generation module 5554 is configured to decode the fusion vector to obtain a reply sentence for replying to the input sentence.

[0212] In some embodiments, the encoding module 5551 is further configured to obtain a semantic vector and a position vector of the input sentence according to input content of the input sentence; and fuse the semantic vector and the position vector of the input sentence to obtain the encoding vector of the input sentence.

[0213] In some embodiments, the encoding module 5551 is further configured to perform word encoding processing on a word sequence of the input sentence to obtain a semantic vector of the input sentence; and perform position encoding processing on positions of each word in the input sentence to obtain a position vector of the input sentence.

[0214] In some embodiments, the multimedia information includes video information and audio information; the encoding module 5551 is further configured to obtain a semantic vector and a position vector of the video information according to image content of the video information, and perform fusion processing on the semantic vector and the position vector of the video information to obtain an encoding vector of the video information; obtain a semantic vector and a position vector of the audio information according to audio content of the audio information, and perform fusion processing on the semantic vector and the position vector of the audio information to obtain an encoding vector of the audio information.

[0215] In some embodiments, the encoding module 5551 is further configured to perform feature extraction processing on each image frame in the video information to obtain a feature vector of each image frame, and combine the feature vector of each image frame to obtain a feature vector of the video information; perform nonlinear mapping processing on the feature vector of the video information to obtain an intermediate feature vector of the video information; perform position encoding processing on the position of each image frame in the video information to obtain a position vector of the video information; and add the intermediate feature vector of the video information and the position vector to obtain an encoding vector of the video information.

[0216] In some embodiments, the encoding vector of the multimedia information includes an encoding vector of video information and an encoding vector of audio information; the self-encoding module 5552 is further configured to perform self-attention processing on the encoding vector of the input sentence to obtain an attention vector of the input sentence; perform multi-head attention processing on the attention vector of the input sentence and the encoding vector of the video information to obtain a feature vector of the video information, and perform multi-head attention processing on the feature vector of the video information and the encoding vector of the audio information to obtain a feature vector of the audio information.

[0217] In some embodiments, the self-encoding module 5552 is further configured to perform self-attention processing on an attention vector of an input sentence output by an i-1 layer encoding layer of a self-encoder through an i layer encoding layer of the self-encoder to obtain an attention vector of the input sentence of the i layer encoding layer; wherein 1

[0218] In some embodiments, the encoding module 5551 is further configured to perform encoding processing on the text information to obtain an encoding vector of the text information; the decoding module 5553 is further configured to perform decoding processing on the encoding vector of the input sentence, the encoding vector of the text information, and the feature vector of the multimedia information by using a decoder respectively to obtain a decoding vector corresponding to the input sentence, a decoding vector corresponding to the text information, and a decoding vector corresponding to the multimedia information; and perform splicing processing on the decoding vector of the input sentence, the decoding vector of the text information, and the decoding vector of the multimedia information to obtain the fusion vector.

[0219] In some embodiments, the decoding module 5553 is further configured to perform self-attention processing on the encoding vector of the text information by using the decoder to obtain a decoding vector of the text information; perform multi-head attention processing on the decoding vector of the text information and the encoding vector of the input sentence to obtain a decoding vector of the input sentence, and perform multi-head attention processing on the decoding vector of the input sentence and the feature vector of the multimedia information to obtain a decoding vector of the multimedia information.

[0220] In some embodiments, the feature vector of the multimedia information includes a feature vector of video information and a feature vector of audio information; the decoding module 5553 is further configured to perform multi-head attention processing on the decoding vector of the input sentence and the feature vector of the video information to obtain a decoding vector of the video information; and perform multi-head attention processing on the decoding vector of the video information and the feature vector of the audio information to obtain a decoding vector of the audio information.

[0221] In some embodiments, the generating module 5554 is further configured to perform linear mapping on the fusion vector and the generated words to obtain a probability distribution of the next word to be generated; determine a word corresponding to a maximum probability in the probability distribution as the next generated word; and according to the order of the generated words, group the generated words to form a reply sentence used to reply to the input sentence.

[0222] In some embodiments, the artificial intelligence-based dialogue generation apparatus 555 further includes a training module 5555 configured to: perform restoration processing on the encoding vector of the input sentence and the encoding vector of the multimedia information in a training sample by a dialogue generation model to obtain probabilities of respective restored words corresponding to the input sentence in the training sample; construct an encoding loss function of the dialogue generation model according to the probabilities of the respective restored words corresponding to the input sentence in the training sample; perform prediction processing on the fusion vector in the training sample by the dialogue generation model to obtain probabilities of each generated word; construct a decoding loss function of the dialogue generation model according to the probabilities of each generated word; add the encoding loss function and the decoding loss function to obtain an overall loss function of the dialogue generation model; update parameters of the dialogue generation model until the overall loss function converges, and use the updated parameters of the dialogue generation model when the overall loss function converges as parameters of the dialogue generation model after training.

[0223] So far, the exemplary application and implementation of the terminal provided by the embodiments of the present application have been described, and the artificial intelligence-based video comment method provided by the embodiments of the present application will be described below. The scheme of realizing video comment by cooperation of various modules in the artificial intelligence-based video comment apparatus 655 provided by the embodiments of the present application will be described below.

[0224] The presentation module 6551 is configured to present content in a video; the processing module 6552 is configured to generate comment information in the form of dialogue about the content according to video information and audio information of the presented content in the video; and the presentation module 6551 is further configured to present the comment information in the form of dialogue.

[0225] In some embodiments, the processing module 6552 is further configured to, when the video is a live video, acquire an input sentence of a comment on the content made by a host; and invoke a dialogue generation model to perform the following operation: generating one or more reply sentences of virtual audiences for replying to the input sentence according to the video information and the audio information of the content.

[0226] In some embodiments, the processing module 6552 is further configured to, when the video is a live video, acquire an input sentence of a comment on the content made by a first audience; and invoke a dialogue generation model to perform the following operation: generating a reply sentence of a second audience for replying to the input sentence according to the video information and the audio information of the content; and wherein at least one of the first audience and the second audience is a virtual audience.

[0227] The embodiment of the present application further provides a computer readable storage medium storing executable instructions, wherein the executable instructions, when executed by a processor, cause the processor to execute the artificial intelligence based dialogue generation method or the artificial intelligence based video comment method provided by the embodiment of the present application, for example, the artificial intelligence based dialogue generation method as shown in Figures 3A-3B the artificial intelligence based video comment method as shown in Figure 5 .

[0228] In some embodiments, the computer readable storage medium can be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, flash memory, magnetic surface memory, optical disc, or CD-ROM, etc.; or various devices including one or any combination of the above memories.

[0229] In some embodiments, the executable instructions can be in the form of programs, software, software modules, scripts or codes, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and can be deployed in any form, including being deployed as independent programs or as modules, components, subroutines or other units suitable for use in a computing environment.

[0230] As an example, the executable instructions can but not necessarily correspond to files in a file system, can be stored in part of a file storing other programs or data, for example, stored in one or more scripts in a Hyper Text Markup Language (HTML) document, stored in a single file dedicated to the program in question, or stored in multiple cooperating files (for example, files storing one or more modules, subroutines or code portions).

[0231] As an example, the executable instructions can be deployed to execute on one computing device (including devices such as smart terminals and servers), or on multiple computing devices located at one site, or on multiple computing devices distributed at multiple sites and interconnected through a communication network.

[0232] The above only describes the embodiments of the present application, and is not used to limit the protection scope of the present application. Any modification, equivalent replacement and improvement within the spirit and scope of the present application shall be included in the protection scope of the present application.

Claims

1. An artificial intelligence-based dialogue generation method, characterized by, The method comprises: encoding an input sentence for multimedia information to obtain an encoded vector of the input sentence, and encoding the multimedia information to obtain an encoded vector of the multimedia information, the multimedia information comprising video information and audio information, and the encoded vector of the multimedia information comprising an encoded vector of the video information and an encoded vector of the audio information; the encoding of the multimedia information to obtain the encoded vector of the multimedia information comprises: obtaining a semantic vector and a position vector of the video information according to image content of the video information, and fusing the semantic vector and the position vector of the video information to obtain the encoded vector of the video information, the position vector of the video information being determined based on positions of each image frame in the video information; obtaining a semantic vector and a position vector of the audio information according to audio content of the audio information, and fusing the semantic vector and the position vector of the audio information to obtain the encoded vector of the audio information, the position vector of the audio information being determined based on positions of each audio frame in the audio information; performing self-attention processing on the encoded vector of the input sentence to obtain an attention vector of the input sentence, and performing multi-head attention processing on the attention vector of the input sentence and the encoded vector of the multimedia information to obtain a feature vector of the multimedia information; fusing the encoded vector of the input sentence and the feature vector of the multimedia information to obtain a fused vector; performing decoding processing on the fused vector to obtain a reply sentence for replying to the input sentence.

2. The method of claim 1, wherein, The encoding of the input sentence to obtain the encoded vector of the input sentence comprises: obtaining a semantic vector and a position vector of the input sentence according to input content of the input sentence, the position vector of the input sentence being determined based on positions of each word in the input content; fusing the semantic vector and the position vector of the input sentence to obtain the encoded vector of the input sentence.

3. The method of claim 2, wherein, The obtaining of the semantic vector and the position vector of the input sentence according to the input content of the input sentence comprises: performing word encoding processing on a word sequence of the input sentence to obtain the semantic vector of the input sentence; performing position encoding processing on positions of each word in the input sentence to obtain the position vector of the input sentence.

4. The method of claim 1, wherein, The obtaining of the semantic vector and the position vector of the video information according to the image content of the video information, and the fusing of the semantic vector and the position vector of the video information to obtain the encoded vector of the video information, comprises: performing feature extraction processing on each image frame in the video information to obtain a feature vector of each image frame, and combining the feature vector of each image frame to obtain the feature vector of the video information; performing nonlinear mapping processing on the feature vector of the video information to obtain an intermediate feature vector of the video information; Position coding processing is performed on a position of each image frame in the video information to obtain a position vector of the video information; An intermediate feature vector of the video information is added to the position vector to obtain an encoded vector of the video information.

5. The method of claim 1, wherein, The multi-head attention processing on the attention vector of the input sentence and the encoded vector of the multimedia information obtains a feature vector of the multimedia information, and the multi-head attention processing on the feature vector of the video information and the encoded vector of the audio information obtains a feature vector of the audio information. The multi-head attention processing on the attention vector of the input sentence and the encoded vector of the video information obtains a feature vector of the video information, and the multi-head attention processing on the feature vector of the video information and the encoded vector of the audio information obtains a feature vector of the audio information. The self-attention processing on the encoded vector of the input sentence obtains the attention vector of the input sentence, and the self-attention processing on the attention vector of the input sentence and the encoded vector of the multimedia information obtains the feature vector of the multimedia information.

6. The method of claim 5, wherein, The self-attention processing on the attention vector of the input sentence and the encoded vector of the multimedia information obtains a feature vector of the multimedia information, and the multi-head attention processing on the feature vector of the video information and the encoded vector of the audio information obtains a feature vector of the audio information. The multi-head attention processing on the attention vector of the input sentence and the encoded vector of the multimedia information obtains a feature vector of the multimedia information, and the multi-head attention processing on the feature vector of the video information and the encoded vector of the audio information obtains a feature vector of the audio information. The multi-head attention processing on the attention vector of the input sentence and the encoded vector of the multimedia information obtains a feature vector of the multimedia information, and the multi-head attention processing on the feature vector of the video information and the encoded vector of the audio information obtains a feature vector of the audio information.

7. The method of claim 1, further comprising, before obtaining the fusion vector: performing encoding processing on the text information to obtain an encoded vector of the text information; The fusion processing on the encoded vector of the input sentence and the feature vector of the multimedia information obtains a fusion vector, and the fusion processing on the feature vector of the video information and the encoded vector of the audio information obtains a fusion vector. The decoding processing on the encoded vector of the input sentence, the encoded vector of the text information and the feature vector of the multimedia information respectively by the decoder obtains a decoding vector corresponding to the input sentence, a decoding vector corresponding to the text information and a decoding vector corresponding to the multimedia information; The splicing processing on the decoding vector of the input sentence, the decoding vector of the text information and the decoding vector of the multimedia information obtains the fusion vector. ​ ​ ​ ​ 8. The method of claim 7, wherein, The decoding vectors corresponding to the input sentence, the decoding vectors corresponding to the text information, and the decoding vectors corresponding to the multimedia information are obtained by decoding the encoding vectors of the input sentence, the encoding vectors of the text information, and the feature vectors of the multimedia information through the decoder, and the decoding vectors corresponding to the input sentence, the decoding vectors corresponding to the text information, and the decoding vectors corresponding to the multimedia information are obtained, comprising: The decoding vectors of the text information are obtained by performing self-attention processing on the encoding vectors of the text information through the decoder; The decoding vectors of the input sentence are obtained by performing multi-head attention processing on the decoding vectors of the text information and the encoding vectors of the input sentence, and The decoding vectors of the multimedia information are obtained by performing multi-head attention processing on the decoding vectors of the input sentence and the feature vectors of the multimedia information. 9.A method for video comment based on artificial intelligence, characterized in that, Comprise: Presenting content in a video; Generating comment information in the form of a dialogue about the content according to video information and audio information of the content; Presenting the comment information in the form of a dialogue.

10. The method of claim 9, wherein, The comment information in the form of a dialogue about the content is generated according to the video information and the audio information of the content, comprising: When the video is a live video, obtaining an input sentence commenting on the content uttered by a host; Calling a dialogue generation model to perform the following operation: generating one or more virtual audience reply sentences for replying to the input sentence according to the video information and the audio information of the content.

11. The method of claim 9, wherein, The comment information in the form of a dialogue about the content is generated according to the video information and the audio information of the content, comprising: When the video is a live video, obtaining an input sentence commenting on the content uttered by a first audience; Calling a dialogue generation model to perform the following operation: generating a second audience reply sentence for replying to the input sentence according to the video information and the audio information of the content; At least one of the first audience and the second audience is a virtual audience. 12.A dialogue generation device based on artificial intelligence, characterized by, Comprise: An encoding module is configured to perform encoding processing on an input sentence for multimedia information to obtain an encoding vector of the input sentence, perform encoding processing on the multimedia information to obtain an encoding vector of the multimedia information, the multimedia information comprising video information and audio information, the encoding vector of the multimedia information comprising an encoding vector of the video information and an encoding vector of the audio information; further configured to obtain a semantic vector and a position vector of the video information according to image content of the video information, and perform fusion processing on the semantic vector and the position vector of the video information to obtain the encoding vector of the video information, the position vector of the video information being determined based on the position of each image frame in the video information; obtain a semantic vector and a position vector of the audio information according to audio content of the audio information, and perform fusion processing on the semantic vector and the position vector of the audio information to obtain the encoding vector of the audio information, the position vector of the audio information being determined based on the position of each audio frame in the audio information; The self-encoding module is configured to perform self-attention processing on the encoding vector of the input sentence to obtain an attention vector of the input sentence, and perform multi-head attention processing on the attention vector of the input sentence and the encoding vector of the multimedia information to obtain a feature vector of the multimedia information. The decoding module is configured to perform fusion processing on the encoding vector of the input sentence and the feature vector of the multimedia information to obtain a fusion vector. The generating module is configured to perform decoding processing on the fusion vector to obtain a reply sentence for replying to the input sentence.

13. An electronic device, comprising: The electronic device comprises: a memory configured to store executable instructions; a processor configured to execute the executable instructions stored in the memory to implement the artificial intelligence-based dialogue generation method in any one of claims 1 to 8 or the artificial intelligence-based video comment method in any one of claims 9 to 11.

14. A computer-readable storage medium, characterized in that, The memory stores executable instructions configured to cause the processor to implement the artificial intelligence-based dialogue generation method in any one of claims 1 to 8 or the artificial intelligence-based video comment method in any one of claims 9 to 11 when executed.

Images