A method, device, equipment and storage medium for generating cover image

Abstract

The embodiments of the present application provide a method, device, equipment and storage medium for generating a cover image, which relates to the field of artificial intelligence technology. The method includes: extracting features from each text information respectively, obtaining the first text feature of each text information, and obtaining a better feature representation for each text information. The first text features are fused to realize contextual joint modeling of the relationship between each text information, so that the obtained comprehensive text features can more accurately represent the content of the target video. Therefore, based on the comprehensive text features, when predicting at least one target description information corresponding to the target video, the accuracy of the predicted target description information can be effectively improved. By combining the original cover image of the target video with the target description information, a target cover image that can better represent the highlights of the video is obtained, thereby facilitating users to quickly and intuitively discover the highlights of the video and improving the user's experience of watching the video.

Description

Technical Field

[0001] Embodiments of the present invention relate to the field of artificial intelligence technology, and in particular, to a method, device, equipment and storage medium for generating a cover image. Background Art

[0002] With the development of video technology, people's requirements for viewing experience are getting higher and higher. When users browse recommended videos, if they cannot quickly find the highlights of the video, they may skip the current video.

[0003] In order to help users quickly and intuitively find video highlights, video creators manually construct video highlights and synthesize them into cover images before uploading videos to video platforms. However, this method requires video creators to have a high level of creative skills and takes a long time to extract the highlights in the video, which is inefficient. Summary of the invention

[0004] The embodiments of the present application provide a method, apparatus, device and storage medium for generating a cover image, which are used to improve the accuracy and efficiency of constructing video highlights and synthesizing them into a cover image.

[0005] In one aspect, an embodiment of the present application provides a method for generating a cover image, the method comprising:

[0006] Performing feature extraction on at least one text information in the target video to obtain first text features corresponding to the at least one text information;

[0007] The obtained first text features are integrated to obtain comprehensive text features corresponding to the target video;

[0008] Based on the comprehensive text features, predict at least one target description information corresponding to the target video;

[0009] At least one target cover image is generated based on the original cover image of the target video and the at least one target description information, wherein each target cover image contains at least one target description information.

[0010] In one aspect, an embodiment of the present application provides a device for generating a cover image, the device comprising:

[0011] A feature extraction module, used to extract features of at least one text information in the target video, and obtain first text features corresponding to the at least one text information;

[0012] A fusion module, used for fusing the obtained first text features to obtain comprehensive text features corresponding to the target video;

[0013] A prediction module, used for predicting at least one target description information corresponding to the target video based on the comprehensive text features;

[0014] A combining module is used to generate at least one target cover image based on the original cover image of the target video and the at least one target description information, wherein each target cover image contains at least one target description information.

[0015] Optionally, the prediction module is specifically used for:

[0016] Based on the comprehensive text features, respectively predicting a first probability that each of the at least one text information is target description information;

[0017] Based on the obtained first probability, at least one target description information is selected from the at least one text information.

[0018] Optionally, the prediction module is specifically used for:

[0019] Encoding the comprehensive text feature to obtain second text features corresponding to the at least one text information respectively;

[0020] Decoding the second text features corresponding to the at least one text information respectively, generating at least one candidate description information corresponding to the target video, and a second probability that each candidate description information is the target description information;

[0021] Based on the obtained second probability, at least one target description information is selected from the at least one candidate description information.

[0022] Optionally, the at least one target description information includes at least one first-category description information and at least one second-category description information;

[0023] The prediction module is specifically used for:

[0024] Encoding the comprehensive text feature to obtain second text features corresponding to the at least one text information respectively;

[0025] Predicting, based on the second text feature corresponding to each of the at least one text information, a first probability that each of the at least one text information is the first type of descriptive information;

[0026] Based on the obtained first probability, selecting at least one first-category description information from the at least one text information;

[0027] Decoding the second text features corresponding to the at least one text information respectively, generating at least one candidate description information corresponding to the target video, and a second probability that each candidate description information is the second type of description information;

[0028] Based on the obtained second probability, at least one second type of description information is selected from the at least one candidate description information.

[0029] Optionally, the combination module is specifically used for:

[0030] Detecting target placement positions of key elements in the original cover image;

[0031] For each piece of target description information in the at least one target description information, perform the following operations respectively:

[0032] For a target description information, determining a target adding position of the target description information from other positions other than the target display position;

[0033] The target description information is integrated into the target adding position in the original cover image to obtain a target cover image including the target description information.

[0034] Optionally, a recommendation module is also included;

[0035] The recommendation module is specifically used for:

[0036] After fusing the one target description information to the target adding position in the original cover image, obtaining the target cover image containing the one target description information, determining the matching degree between the target user's interest tag and the at least one target description information respectively;

[0037] Determining recommended description information from the at least one target description information based on the obtained matching degree;

[0038] When recommending the target video to the target user, a target cover image containing the recommendation description information is selected as the cover image of the target video.

[0039] Optionally, the recommendation module is specifically used for:

[0040] Extracting features of the target user's interest tags to obtain the target user's interest features;

[0041] For each piece of target description information in the at least one target description information, perform the following operations respectively:

[0042] For a target description information, extracting features of the target description information to obtain description features of the target description information;

[0043] Merging the interest feature with the description feature to obtain a first associated feature;

[0044] Based on the first association feature, predict the matching degree between the interest tag of the target user and the one target description information.

[0045] Optionally, the prediction module is further used for:

[0046] Before predicting at least one target description information corresponding to the target video based on the comprehensive text features, it is determined that the original cover image of the target video does not contain the target description information.

[0047] Optionally, the prediction module is specifically used for:

[0048] Obtaining cover text information in the original cover image;

[0049] Performing feature extraction on the cover text information to obtain cover text features corresponding to the cover text information;

[0050] The comprehensive text feature is merged with the cover text feature to obtain a second associated feature;

[0051] Based on the second association feature, predict a third probability that the cover text information is target description information;

[0052] If the third probability is less than a preset threshold, it is determined that the original cover image does not contain target description information.

[0053] On the one hand, an embodiment of the present application provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above-mentioned method for generating a cover image when executing the program.

[0054] On the one hand, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program that can be executed by a computer device. When the program is run on the computer device, the computer device executes the steps of the above-mentioned method for generating a cover image.

[0055] In the embodiment of the present application, feature extraction is first performed on each text information respectively to obtain the first text feature of each text information, so that each text information obtains a better feature representation. The various first text features are fused to implement contextual joint modeling of the relationship between the various text information, so that the obtained comprehensive text features can more accurately represent the content of the target video. Therefore, based on the comprehensive text features, when predicting at least one target description information corresponding to the target video, the accuracy and efficiency of predicting the target description information can be effectively improved. Furthermore, by combining the original cover image of the target video with the target description information, a target cover image that can better represent the highlights of the video is obtained, thereby facilitating users to quickly and intuitively discover the highlights of the video and improving the user's experience of watching the video. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0057] Figure 1 A schematic diagram of a system architecture provided for an embodiment of the present application;

[0058] Figure 2 A flowchart of a method for generating a cover image provided in an embodiment of the present application;

[0059] Figure 3 A flowchart of a method for extracting highlight content provided in an embodiment of the present application;

[0060] Figure 4 A flowchart of a method for extracting highlight content provided in an embodiment of the present application;

[0061] Figure 5 A flowchart of a method for extracting highlight content provided in an embodiment of the present application;

[0062] Figure 6 A flowchart of a method for extracting highlight content provided in an embodiment of the present application;

[0063] Figure 7 A flowchart of a method for obtaining target description information provided in an embodiment of the present application;

[0064] Figure 8 A flowchart of a method for extracting highlight content provided in an embodiment of the present application;

[0065] Fig. 9 A flowchart of a method for determining a location to add highlight content provided in an embodiment of the present application;

[0066] Fig.10 A flowchart of a method for obtaining a target cover image provided by an embodiment of the present application;

[0067] Fig.11 A schematic diagram of the structure of a bright spot judgment model provided in an embodiment of the present application;

[0068] Fig.12 A schematic diagram of the structure of a bright spot judgment model provided in an embodiment of the present application;

[0069] Fig.13A flowchart of a video recommendation method provided in an embodiment of the present application;

[0070] Fig.14 A schematic diagram of a video application recommendation interface provided in an embodiment of the present application;

[0071] Fig.15 A schematic diagram of the structure of a bright spot matching model provided in an embodiment of the present application;

[0072] Fig.16 A schematic diagram of the structure of a bright spot matching model provided in an embodiment of the present application;

[0073] Fig.17 A flowchart of a method for generating a cover image and video recommendation provided in an embodiment of the present application;

[0074] Fig.18 A schematic diagram of the structure of a device for generating a cover image provided in an embodiment of the present application;

[0075] Fig.19 A schematic diagram of the structure of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0076] In order to make the purpose, technical solution and beneficial effects of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0077] For ease of understanding, the terms involved in the embodiments of the present invention are explained below.

[0078] Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that machines have the functions of perception, reasoning and decision-making.

[0079] Artificial intelligence technology is a comprehensive discipline that covers a wide range of fields, including both hardware-level and software-level technologies. The basic technologies of artificial intelligence generally include sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics and other technologies. Artificial intelligence software technology mainly includes computer vision technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0080] Natural language processing (NLP) is an important direction in the fields of computer science and artificial intelligence. It studies various theories and methods that can achieve effective communication between people and computers using natural language. Natural language processing is a science that integrates linguistics, computer science, and mathematics. Therefore, research in this field will involve natural language, that is, the language used by people in daily life, so it is closely related to the study of linguistics. Natural language processing technology generally includes text processing, semantic understanding, machine translation, robot question and answer, knowledge graph and other technologies. For example, in the embodiment of the present application, NLP is used to obtain at least one target description information corresponding to the target video, and then at least one target cover image is generated based on the original cover image of the target video and at least one target description information.

[0081] BERT: (Bidirectional Encoder Representations from Transformers), that is, the encoder of the bidirectional Transformer, which describes the features of character level, word level, sentence level and even inter-sentence relations. ALBERT is a lightweight BERT.

[0082] Seq2Seq model: (Sequence-to-sequence) model, consisting of an encoder and a decoder. The Seq2Seq model can be applied to scenarios such as machine translation, human-computer dialogue, and chatbots.

[0083] Video effect enhancement: Automatically extract highlights from the video and intelligently synthesize them onto the cover image, allowing users to intuitively discover video highlights and increase video playback. In addition, based on the user's personalized viewing experience, dynamically select a cover image containing the corresponding highlight content for display.

[0084] The design concept of the embodiments of the present application is introduced below.

[0085] When users browse recommended videos, if they cannot quickly find the highlights of the video, they may skip the current video. In order to facilitate users to quickly and intuitively find the highlights of the video, video creators use manual methods to construct video highlights and synthesize them into the cover image before uploading the video to the video platform. However, this method requires video creators to have a high level of creation and takes a long time to refine the highlights in the video, which is inefficient.

[0086] Through analysis, we found that in order to help users understand the content of the video, the video generally includes text information corresponding to the video content, such as titles, subtitles, bullet comments, commentary text, etc. These text information often include highlight information related to the video content. If natural language processing technology is used to conduct multi-dimensional and in-depth understanding of these text information, the highlight content in the video can be automatically obtained, thereby improving the efficiency of obtaining video highlight information. By automatically combining the highlight content with the cover image, the video effect can be enhanced, so that users can quickly and intuitively discover the highlights of the video and improve the user's video viewing experience.

[0087] In view of this, an embodiment of the present application provides a method for generating a cover image, the method comprising: extracting features of at least one text information in a target video, obtaining first text features corresponding to at least one text information, and then fusing the obtained first text features to obtain comprehensive text features corresponding to the target video. Based on the comprehensive text features, predict at least one target description information corresponding to the target video, and then generate at least one target cover image based on the original cover image of the target video and at least one target description information, wherein each target cover image contains at least one target description information.

[0088] In the embodiment of the present application, feature extraction is first performed on each text information respectively to obtain the first text feature of each text information, so that each text information obtains a better feature representation. The various first text features are fused to implement contextual joint modeling of the relationship between the various text information, so that the obtained comprehensive text features can more accurately represent the content of the target video. Therefore, based on the comprehensive text features, when predicting at least one target description information corresponding to the target video, the accuracy and efficiency of predicting the target description information can be effectively improved. Furthermore, by combining the original cover image of the target video with the target description information, a target cover image that can better represent the highlights of the video is obtained, thereby facilitating users to quickly and intuitively discover the highlights of the video and improving the user's experience of watching the video.

[0089] refer to Figure 1 , which is a system architecture diagram applicable to an embodiment of the present application, and the system architecture at least includes a terminal device 101 and a server 102.

[0090] The terminal device 101 is pre-installed with a video application, wherein the video application includes a video playback application, a short video application, a live broadcast application, etc., and the types of video applications include a client application, a web version application, a small program application, etc. The terminal device 101 can be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, a smart TV, a car device, etc., but is not limited thereto.

[0091] The server 102 is a background server of the video application. The server 102 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers. It can also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms. The terminal device 101 and the server 102 can be directly or indirectly connected via wired or wireless communication, and this application does not limit this.

[0092] The method for generating a cover image in the embodiment of the present application can be executed by the terminal device 101 or by the server 102.

[0093] In the first case, the method for generating a cover image in the embodiment of the present application can be executed by the terminal device 101.

[0094] The terminal device 101 performs feature extraction on at least one text information in the target video to obtain a first text feature corresponding to the at least one text information, and then fuses the obtained first text features to obtain a comprehensive text feature corresponding to the target video. Based on the comprehensive text feature, predict at least one target description information corresponding to the target video, and then generate at least one target cover image based on the original cover image of the target video and at least one target description information, wherein each target cover image contains at least one target description information. When the terminal device 101 recommends a target video to a target user, it determines the recommended description information that matches the interest tag of the target user from the at least one target description information, and then selects the target cover image containing the recommended description information as the cover image of the target video, and displays the cover image of the target video in the recommendation interface of the video application.

[0095] In the second case, the method for generating a cover image in the embodiment of the present application can be executed by the server 102.

[0096] The server 102 performs feature extraction on at least one text information in the target video to obtain a first text feature corresponding to the at least one text information, and then fuses the obtained first text features to obtain a comprehensive text feature corresponding to the target video. Based on the comprehensive text feature, predict at least one target description information corresponding to the target video, and then generate at least one target cover image based on the original cover image of the target video and at least one target description information, wherein each target cover image contains at least one target description information. When the server 102 recommends the target video to the target user, it determines the recommended description information that matches the interest tag of the target user from the at least one target description information, and then selects the target cover image containing the recommended description information as the cover image of the target video. The server 102 sends the cover image of the target video to the terminal device 101, and the terminal device 101 displays the cover image of the target video in the recommendation interface of the video application.

[0097] It should be noted that the method for generating a cover image in the embodiment of the present application can also be executed interactively by the terminal device 101 and the server 102, which will not be repeated here.

[0098] based on Figure 1 The system architecture diagram shown in the figure, the embodiment of the present application provides a method process for generating a cover image, such as Figure 2 As shown, the process of the method is executed by a computer device, which may be Figure 1 The terminal device 101 or the server 102 shown includes the following steps:

[0099] Step S201: extract features of at least one text information in a target video to obtain first text features corresponding to the at least one text information.

[0100] Specifically, the text information in the target video may be a title, subtitle, barrage, commentary text, etc. A text message may refer to a sentence, a word, a paragraph, etc. The text content in the target video is obtained by means of speech recognition or image recognition, and then the text content in the target video is divided into multiple text messages by means of word segmentation or sentence segmentation, and then an encoder is used to extract features from each text message to obtain the first text feature corresponding to each text message, wherein the encoder may be ALBERT, BERT, Transformer-Encoder, etc. It should be noted that the neural network structures such as encoders involved in the embodiments of the present application can be stored on the blockchain, and will not be described in detail later.

[0101] Step S202: The obtained first text features are fused to obtain comprehensive text features corresponding to the target video.

[0102] Specifically, there is a contextual connection between the various text information in the video. When observing a text information alone, sometimes it is not possible to tell whether the text information is a highlight content in the video, but combined with the context of the text information, it is possible to more accurately determine whether the text information is a highlight content in the video. Therefore, in the embodiment of the present application, the first text features obtained are fused to achieve contextual joint modeling of the relationship between the various text information, so that the comprehensive text features corresponding to the obtained target video can more accurately represent the highlight content of the target video.

[0103] Step S203: predict at least one target description information corresponding to the target video based on the comprehensive text features.

[0104] Specifically, the target description information is used to characterize the main content or highlight content of the target video. The target description information can be a sentence, a word, or multiple words. The target description information can be extracted from at least one text information, or generated based on at least one text information.

[0105] Step S204: generating at least one target cover image based on the original cover image of the target video and at least one target description information.

[0106] Specifically, each target cover image includes at least one target description information. When the original cover image is combined with the target description information, the target description information can be set in different styles, specifically in terms of font, color, font size, display angle, display position, display method, etc.

[0107] In the embodiment of the present application, feature extraction is first performed on each text information respectively to obtain the first text feature of each text information, so that each text information obtains a better feature representation. The various first text features are fused to implement contextual joint modeling of the relationship between the various text information, so that the obtained comprehensive text features can more accurately represent the content of the target video. Therefore, based on the comprehensive text features, when predicting at least one target description information corresponding to the target video, the accuracy and efficiency of predicting the target description information can be effectively improved. Furthermore, by combining the original cover image of the target video with the target description information, a target cover image that can better represent the highlights of the video is obtained, thereby facilitating users to quickly and intuitively discover the highlights of the video and improving the user's experience of watching the video.

[0108] Optionally, in the above step S203, based on the comprehensive text features, at least one target description information corresponding to the target video is predicted. The embodiment of the present application provides at least the following implementation methods:

[0109] Implementation method 1: Extraction method.

[0110] Based on the comprehensive text features, first probabilities of at least one text information being target description information are predicted respectively, and then based on the obtained first probabilities, at least one target description information is selected from the at least one text information.

[0111] Specifically, the comprehensive text features are input into an encoder to obtain a first probability that at least one text information is the target description information, wherein the encoder may be ALBERT, BERT, Transformer-Encoder, etc. The obtained first probabilities are sorted in descending order, and the text information corresponding to the first probability greater than the screening threshold is used as the target description information, or the text information corresponding to the first probabilities ranked in the top N is used as the target description information, wherein N is a preset number.

[0112] For example, Figure 3 As shown, the text content in the target video includes video title, video text-sentence 1, video text-sentence 2, ..., video text-sentence S, where S is a positive integer. ALBERT is used to extract features of the video title to obtain the title features corresponding to the video title; ALBERT is used to extract features of the video text-sentence 1 to obtain the sentence 1 features corresponding to the video text-sentence 1; ALBERT is used to extract features of the video text-sentence 2 to obtain the sentence 2 features corresponding to the video text-sentence 2, and so on, until the sentence S features corresponding to the video text-sentence S are obtained. After the title features, sentence 1 features, sentence 2 features, sentence 3 features, ..., sentence S features are fused, the comprehensive text features of the target video are obtained. The comprehensive text features are input into ALBERT to predict the probability that the video title is the highlight content, the probability that sentence 1 is the highlight content, the probability that sentence 2 is the highlight content, the probability that sentence 3 is the highlight content, ..., and the probability that sentence S is the highlight content. Then, the video title or sentence with a probability greater than the screening threshold is retained as the highlight content of the target video.

[0113] Optionally, sometimes the video titles of some videos are too long, and users cannot quickly find the highlights of the video through the video titles, so these video titles are not suitable as the highlights of the video. However, the video titles are very helpful for understanding the video content. In order to improve the accuracy of the comprehensive text features in representing the video content and avoid repeated determination of whether the video title is the highlight of the video, in the embodiment of the present application, the first text feature corresponding to the video title and the first text feature corresponding to each other text information other than the video title are fused to obtain the comprehensive text feature corresponding to the target video. Then, based on the comprehensive text features, the first probability of each other text information other than the video title being the target description information is predicted respectively.

[0114] For example, Figure 4As shown, the text content in the target video includes video title, video text-sentence 1, video text-sentence 2, ..., video text-sentence S. ALBERT is used to extract features of the video title to obtain the title features corresponding to the video title; ALBERT is used to extract features of video text-sentence 1 to obtain the sentence 1 features corresponding to video text-sentence 1; ALBERT is used to extract features of video text-sentence 2 to obtain the sentence 2 features corresponding to video text-sentence 2, and so on, until the sentence S features corresponding to video text-sentence S are obtained. After the title features, sentence 1 features, sentence 2 features, sentence 3 features, ..., sentence S features are fused, the comprehensive text features of the target video are obtained. The comprehensive text features are input into ALBERT to predict the probability that sentence 1 is the highlight content, the probability that sentence 2 is the highlight content, the probability that sentence 3 is the highlight content, ..., and the probability that sentence S is the highlight content. Then, the sentences with probabilities greater than the screening threshold are retained as the highlight content of the target video.

[0115] In the implementation of this application, feature extraction is first performed on each text information to obtain the first text feature of each text information, so that each text information obtains a better feature representation. The first text features are fused to achieve contextual joint modeling of the relationship between the text information, so that the obtained comprehensive text features corresponding to the target video can more accurately represent the content of the target video. Therefore, based on the comprehensive text features, when at least one target description information is selected from at least one text information, the accuracy of the obtained target description information is effectively improved.

[0116] Implementation method 2: Generation method.

[0117] The comprehensive text features are encoded to obtain the second text features corresponding to the at least one text information. The second text features corresponding to the at least one text information are decoded to generate at least one candidate description information corresponding to the target video and a second probability that each candidate description information is the target description information. Then, based on the obtained second probability, at least one target description information is selected from the at least one candidate description information.

[0118] Specifically, an encoder is used to encode the comprehensive text features to obtain the second text features corresponding to the at least one text information, wherein the encoder may be ALBERT, BERT, Transformer-Encoder, etc. A decoder is used to decode the second text features corresponding to the at least one text information to generate at least one candidate description information corresponding to the target video, wherein the decoder may be Transformer-Decoder, etc. The candidate description information may be extracted from the at least one text information of the target video, or may be generated based on the at least one text information of the target video, and the candidate description information may be a sentence, or may be a word or multiple words.

[0119] The obtained second probabilities are sorted in descending order, and the candidate description information corresponding to the second probabilities greater than the screening threshold is used as the target description information, or the candidate description information corresponding to the second probabilities ranked in the top M positions is used as the target description information, where M is a preset number.

[0120] For example, Figure 5 As shown, the text content in the target video includes video title, video text-sentence 1, video text-sentence 2, ..., video text-sentence S. ALBERT is used to extract features of the video title to obtain the first title feature corresponding to the video title; ALBERT is used to extract features of video text-sentence 1 to obtain the first sentence 1 feature corresponding to video text-sentence 1; ALBERT is used to extract features of video text-sentence 2 to obtain the first sentence 2 feature corresponding to video text-sentence 2, and so on, until the first sentence S feature corresponding to video text-sentence S is obtained.

[0121] After fusing the first title feature, the first sentence 1 feature, the first sentence 2 feature, the first sentence 3 feature, ..., the first sentence S feature, the comprehensive text feature of the target video is obtained. The comprehensive text feature is input into the encoder (Transformer-Encoder) to obtain the second title feature, the second sentence 1 feature, the second sentence 2 feature, the second sentence 3 feature, ..., the second sentence S feature. Then the second title feature, the second sentence 1 feature, the second sentence 2 feature, the second sentence 3 feature, ..., the second sentence S feature are input into the decoder (Transformer-Decoder), and the attention mechanism (Encoder-Decoder Attention) is used to predict the highlight word 1, highlight word 2, highlight word 3, ..., highlight word W, and the probability that each highlight word is the highlight content of the target video. Then, the highlight words with a probability greater than the screening threshold are retained as the highlight content of the target video.

[0122] Optionally, as described in Implementation Method 1, in order to improve the accuracy of the comprehensive text features in representing the video content and avoid repeated determination of whether the video title is the highlight content of the video, in an embodiment of the present application, the first text features corresponding to the video title and the first text features corresponding to other text information other than the video title are fused to obtain the comprehensive text features corresponding to the target video. The comprehensive text features are encoded to obtain the second text features corresponding to other text information other than the video title. Each of the obtained second text features is decoded to generate at least one candidate description information corresponding to the target video and a second probability that each candidate description information is the target description information. Then, based on the obtained second probability, at least one target description information is selected from at least one candidate description information.

[0123] For example, Figure 6 As shown, the text content in the target video includes video title, video text-sentence 1, video text-sentence 2, ..., video text-sentence S. ALBERT is used to extract features of the video title to obtain the first title feature corresponding to the video title; ALBERT is used to extract features of video text-sentence 1 to obtain the first sentence 1 feature corresponding to video text-sentence 1; ALBERT is used to extract features of video text-sentence 2 to obtain the first sentence 2 feature corresponding to video text-sentence 2, and so on, until the first sentence S feature corresponding to video text-sentence S is obtained.

[0124] After fusing the first title feature, the first sentence 1 feature, the first sentence 2 feature, the first sentence 3 feature, ..., the first sentence S feature, the comprehensive text feature of the target video is obtained. The comprehensive text feature is input into the encoder (Transformer-Encoder) to predict the second sentence 1 feature, the second sentence 2 feature, the second sentence 3 feature, ..., the second sentence S feature. Then the second sentence 1 feature, the second sentence 2 feature, the second sentence 3 feature, ..., the second sentence S feature are input into the decoder (Transformer-Decoder), and the attention mechanism (Encoder-Decoder Attention) is used to predict the highlight word 1, highlight word 2, highlight word 3, ..., highlight word W, and the probability that each highlight word is the highlight content of the target video. Then, the highlight words with a probability greater than the screening threshold are retained as the highlight content of the target video.

[0125] In the implementation of this application, feature extraction is first performed on each text information to obtain the first text feature of each text information, so that each text information obtains a better feature representation. The various first text features are fused to achieve contextual joint modeling of the relationship between the various text information, so that the comprehensive text features corresponding to the target video can more accurately represent the content of the target video. Furthermore, based on the comprehensive text features, the second text features of each text information are predicted and obtained, so that the second text features contain both the content features of the text information and the contextual information of the text information. Therefore, when generating the target description information based on the obtained second text features, the accuracy of the target description information is improved, and the target description information is no longer limited to the text information in the target video, thereby improving the diversity of the target description information.

[0126] Implementation method three: combining extraction method with generation method.

[0127] In an embodiment of the present application, at least one target description information includes at least one first-category description information and at least one second-category description information, wherein the first-category description information is target description information obtained by extraction, and the second description information is target description information obtained by generation.

[0128] Based on the comprehensive text features, the process of predicting at least one target description information corresponding to the target video is as follows: Figure 7 As shown, the following steps are included:

[0129] Step S701: Encode the comprehensive text features to obtain second text features corresponding to at least one text information.

[0130] An encoder is used to encode the comprehensive text features to obtain second text features corresponding to at least one text information. An encoder can also be used to encode the comprehensive text features to obtain second text features corresponding to other text information other than the video title. The encoder can be ALBERT, BERT, Transformer-Encoder, etc.

[0131] Step S702: predicting a first probability that each of the at least one text information is the first type of description information based on a second text feature corresponding to each of the at least one text information.

[0132] Step S703: selecting at least one first category description information from at least one text information based on the obtained first probability.

[0133] The obtained first probabilities are sorted in descending order, and the text information corresponding to the first probabilities greater than the screening threshold is used as the first category of description information, or the text information corresponding to the first probabilities ranked in the top N positions is used as the first category of description information, where N is a preset number.

[0134] Step S704: Decode the second text features corresponding to the at least one text information respectively to generate at least one candidate description information corresponding to the target video and a second probability that each candidate description information is the second type of description information.

[0135] A decoder is used to decode the second text features corresponding to the at least one text information respectively, and at least one candidate description information corresponding to the target video is generated, wherein the decoder can be a Transformer-Decoder, etc.

[0136] Step S705: selecting at least one second type of description information from at least one candidate description information based on the obtained second probability.

[0137] The obtained second probabilities are sorted in descending order, and the candidate description information corresponding to the second probabilities greater than the screening threshold is used as the second category description information, or the candidate description information corresponding to the second probabilities ranked in the top M positions is used as the second category description information, where M is a preset number.

[0138] For example, Figure 8 As shown, the text content in the target video includes video title, video text-sentence 1, video text-sentence 2, ..., video text-sentence S, where S is a positive integer. ALBERT is used to extract features from the video title to obtain the first title feature corresponding to the video title; ALBERT is used to extract features from video text-sentence 1 to obtain the first sentence 1 feature corresponding to video text-sentence 1; ALBERT is used to extract features from video text-sentence 2 to obtain the first sentence 2 feature corresponding to video text-sentence 2, and so on, until the first sentence S feature corresponding to video text-sentence S is obtained.

[0139] After fusing the first title feature, the first sentence 1 feature, the first sentence 2 feature, the first sentence 3 feature, ..., the first sentence S feature, the comprehensive text feature of the target video is obtained. The comprehensive text feature is input into the encoder (Transformer-Encoder) to predict the second sentence 1 feature, the second sentence 2 feature, the second sentence 3 feature, ..., the second sentence S feature. Based on the second sentence 1 feature, the first probability of sentence 1 being the highlight content is predicted; based on the second sentence 2 feature, the first probability of sentence 2 being the highlight content is predicted; based on the second sentence 3 feature, the first probability of sentence 3 being the highlight content is predicted, ..., based on the second sentence S feature, the first probability of sentence S being the highlight content is predicted. Then, the sentences whose first probability is greater than the screening threshold are retained as the first category of highlight content of the target video.

[0140] The second sentence 1 feature, the second sentence 2 feature, the second sentence 3 feature, ..., the second sentence S feature are input into the decoder (Transformer-Decoder), and the attention mechanism (Encoder-Decoder Attention) is used to predict the highlight word 1, highlight word 2, highlight word 3, ..., highlight word W, and the second probability of each highlight word being the highlight content of the target video, W is a positive integer. Then, the highlight words with a second probability greater than the screening threshold are retained as the second type of highlight content of the target video.

[0141] In an embodiment of the present application, at least one target description information is selected from at least one text information based on the comprehensive text features. At the same time, based on the comprehensive text features, the second text features of each text information in the target video are predicted, and then the target description information is generated based on the obtained second text features. This not only improves the accuracy of obtaining the target description information, but also makes the target description information no longer limited to the text information in the target video, but also includes the generated description information, thereby improving the diversity of the target description information and making the obtained target description information more comprehensive.

[0142] Optionally, in the above step S204, since key elements such as faces in the original cover image are also the content that users pay more attention to, when adding target description information to the original cover image, it is necessary to try to avoid adding the target description information to the target display position of the key elements, which will block the key elements. In view of this, in the embodiment of the present application, the target display position of the key elements in the original cover image is detected. For each target description information in at least one target description information, the following operations are performed respectively:

[0143] For a target description information, a target adding position of the target description information is determined from other positions outside the target display position, and then the target description information is merged into the target adding position in the original cover image to obtain a target cover image containing the target description information.

[0144] Specifically, the key elements may be faces, objects, etc. in the original cover image. Target detection networks such as R-CNN (Region-CNN), Fast R-CNN, and Faster R-CNN may be used to detect the target display positions of the key elements in the original cover image.

[0145] For example, Fig. 9 As shown, the R-CNN model is used to detect the original cover image, determine the target display position 901 of the face in the original cover image, and determine the target adding position 902 of the highlight content from other positions outside the target display position 901.

[0146] Optionally, when the target description information and the original cover image are merged, the target description information can be set to different styles. Specifically, it can be set from the aspects of font, color, font size, display angle, display method, etc. Among them, the font can use the platform's standard font-Kaiti, or a random font can be selected; the color can choose a color range with a large contrast with the background color difference of the target adding position, and it can be randomly selected within the range with a large contrast. The font size can use the platform's standard font size, or it can be randomly selected; the display angle is generally horizontal or vertical, or it can be a random angle; the display method can be dynamic display, adding animation effects, etc.

[0147] For example, Fig.10 As shown, the target video is set to a football match video, and the highlight content of the target video is "Team A defeats Team B". The R-CNN model is used to detect the original cover image of the target video, determine the target display position 1001 of the face in the original cover image, and determine the target adding position 1002 of the highlight content from other positions outside the target display position 1001. The style of the highlight content is set to: No. 4 Kaiti, black, horizontal static display. The highlight content "Team A defeats Team B" is then merged into the target adding position 1002 to obtain the target cover image. It should be noted that in the embodiment of the present application, the target cover image is not limited to containing only one highlight content, but may also contain multiple highlight contents, which will not be repeated here.

[0148] In the embodiment of the present application, the target display position of the key elements in the original cover image is detected, and then the target adding position for displaying the target description information is selected from other positions outside the target display position, so as to avoid the target description information blocking the key elements in the original cover image. Secondly, when the target description information and the original cover image are merged, different styles are set for the target description information, which enriches the display form of the target description information and avoids boring and single display situations, thereby improving the user's viewing experience.

[0149] Optionally, the original cover images of some videos may already have highlight content, that is, the highlight content of the video has been added to the original cover image manually or by other means. If the video is pushed directly to the user, the user can also quickly and intuitively find the highlights of the video. In order to avoid the waste of resources caused by repeatedly extracting the highlight content in the video and synthesizing the target cover image of the video. In the embodiment of the present application, based on the comprehensive text features, before predicting at least one target description information corresponding to the target video, it is determined that the original cover image of the target video does not contain the target description information.

[0150] Specifically, a highlight judgment model is used to determine whether the original cover image of the target video contains target description information, wherein when training the highlight judgment model, the training samples include pre-labeled video data with and without highlights. When the highlight judgment model is used to determine that the original cover image of the target video does not contain target description information, at least one target description information corresponding to the target video is predicted based on comprehensive text features, and then at least one target cover image is generated based on the original cover image of the target video and at least one target description information. When the highlight judgment model is used to determine that the original cover image of the target video contains target description information, the original cover image is directly used as the target cover image, thereby avoiding the waste of resources caused by repeated prediction of the target description information of the target video and synthesizing the target cover image, while improving efficiency.

[0151] Optionally, when determining whether the original cover image of the target video contains target description information, the embodiment of the present application provides at least the following two implementation methods:

[0152] Implementation method 1: Obtain cover text information in the original cover image, then perform feature extraction on the cover text information to obtain cover text features corresponding to the cover text information. Fuse the comprehensive text features with the cover text features to obtain a second associated feature. Based on the second associated feature, predict the third probability that the cover text information is the target description information. If the third probability is less than a preset threshold, it is determined that the original cover image does not contain the target description information. If the third probability is not less than the preset threshold, it is determined that the original cover image contains the target description information.

[0153] Specifically, the cover text information in the original cover image is obtained through optical character recognition (OCR), and the encoder in the bright spot judgment model is used to extract features of the cover text information to obtain cover text features corresponding to the cover text information, wherein the encoder can be ALBERT, BERT, Transformer-Encoder, etc.

[0154] For example, the structure of the bright spot judgment model is as follows: Fig.11As shown, the target description information is set as the highlight content, the video text of the target video is obtained through voice / image recognition, and the cover text information in the original cover image is obtained by OCR. The cover text information is input into the encoder (Transformer-Encoder) to obtain the cover text feature. The video text is input into the encoder (Transformer-Encoder) to obtain the comprehensive text feature, and then the comprehensive text feature is merged with the cover text feature to obtain the second associated feature, and then based on the second associated feature, the third probability of the cover text information being the highlight content is predicted. If the third probability is less than the preset threshold, it is determined that the original cover image does not contain the highlight content. If the third probability is not less than the preset threshold, it is determined that the original cover image contains the highlight content.

[0155] In an embodiment of the present application, the comprehensive text feature is integrated with the cover text feature, so that the obtained second associated feature includes both the content feature of the cover text information and the context feature of the cover text information in the video text. Therefore, based on the second associated feature, when predicting the probability that the cover text information is the target description information, the prediction accuracy is effectively improved, thereby improving the accuracy of judging whether the original cover image contains the target description information.

[0156] Implementation method 2: obtaining cover text information in the original cover image, and then extracting features of the cover text information to obtain cover text features corresponding to the cover text information. Based on the cover text features, predicting a third probability that the cover text information is target description information. If the third probability is less than a preset threshold, it is determined that the original cover image does not contain the target description information; if the third probability is not less than the preset threshold, it is determined that the original cover image contains the target description information.

[0157] For example, the structure of the bright spot judgment model is as follows: Fig.12 As shown, the target description information is set as the highlight content, and OCR is used to obtain the cover text information in the original cover image. The cover text information is input into the encoder (Transformer-Encoder) to obtain the cover text features. Then, based on the cover text features, the third probability that the cover text information is the highlight content is predicted. If the third probability is less than the preset threshold, it is determined that the original cover image does not contain the highlight content. If the third probability is not less than the preset threshold, it is determined that the original cover image contains the highlight content.

[0158] Optionally, since different users are interested in different highlight contents, when recommending target videos to different users, different target cover images may be displayed according to the interests of different users.

[0159] Thus, in the embodiment of the present application, a target description information is fused to the target adding position in the original cover image, and after obtaining a target cover image containing a target description information, the matching degree between the interest tag of the target user and at least one target description information is determined. Then, based on the obtained matching degree, the recommended description information is determined from the at least one target description information. When recommending the target video to the target user, the target cover image containing the recommended description information is selected as the cover image of the target video.

[0160] Specifically, the target user's interest tags are obtained based on the target user's historical video playback records. For example, each video in a video application includes one or more video tags. When the number of times a user clicks on a video is greater than a preset threshold, or the length of time a video is watched meets a preset condition, the video is determined to be a validly played video, and the video tag of the video is used as the user's interest tag. By collecting statistics on the user's historical video playback records, the user's interest tag set is obtained.

[0161] The matching degree between the target user's interest tag and the target description information refers to the target user's interest in the target description information. The higher the matching degree, the higher the target user's interest in the target description information, and the greater the probability that the target user clicks on the target cover image containing the target description information. The lower the matching degree, the lower the target user's interest in the target description information, and the smaller the probability that the target user clicks on the target cover image containing the target description information.

[0162] At least one target cover image corresponding to the target video is generated in advance, and the target description information in each target cover image is different. Before recommending the target video to the target user, a recommended description information with the highest matching degree with the target user's interest tag is selected from the at least one target description information, and then the target cover image containing the recommended description information is recommended to the target user.

[0163] Optionally, target description information with a matching degree greater than a matching degree threshold may be first filtered out from at least one target description information, and then a recommended description information with the highest matching degree with the target user's interest tag is selected from the filtered target description information. If the matching degree corresponding to at least one target description information is less than the matching degree threshold, the original cover image is recommended to the target user.

[0164] For example, Fig.13 As shown in FIG, the process of personalized recommendation of target videos to target users includes the following steps:

[0165] Step S1301: The terminal device pre-installs a video application.

[0166] Step S1302: In response to a start-up operation of a target user for a video application, the terminal device sends a video request carrying identity information of the target user to a server.

[0167] Step S1303: The server obtains the target user's interest tag set based on the target user's identity information.

[0168] Specifically, the interest tag set of the target user is set to {football, game, World Cup}, and the server can count the interest tag set of the target user in advance based on the historical video playback records of the target user.

[0169] Step S1304: The server determines that the videos recommended to the target user are video M and video N based on the interest tag set of the target user.

[0170] Specifically, video M is a World Cup match video, and video M corresponds to target cover image m1 and target cover image m2, wherein the highlight content contained in target cover image m1 is "football match", and the highlight content contained in target cover image m2 is "Football World Cup - Team M defeats Team N".

[0171] Video N is a collection of football match highlights, and video N corresponds to target cover image n1 and target cover image n2, wherein the highlight content contained in target cover image n1 is “football match highlights”, and the highlight content contained in target cover image n2 is “football World Cup match highlights”.

[0172] Step S1305 : The server determines the matching degree between the interest tag set of the target user and the highlight contents in video M and video N respectively.

[0173] Step S1306: The server determines "Soccer World Cup - Team M defeats Team N" and "Soccer World Cup Match Highlights" as recommended highlight contents based on the obtained matching degree.

[0174] Step S1307: The server sends the target cover image m2 and the target cover image n2 to the terminal device.

[0175] Step S1308: The terminal device displays the target cover image m2 of video M and the target cover image n2 of video N in the recommendation interface of the video application.

[0176] Specifically, the recommendation interface of the video application is as follows: Fig.14 As shown, the video recommendation interface includes a search box, a target cover image m2 of video M, and a target cover image n2 of video N, wherein the target cover image m2 includes the highlight content "Football World Cup - Team M defeats Team N", and the target cover image n2 includes the highlight content "Football World Cup Match Highlights".

[0177] In the embodiment of the present application, by deeply understanding the current video content, highlight content reflecting the main content of the video is constructed, and then the highlight content is added to the appropriate position of the original cover image in a suitable display form, so that users can intuitively and quickly find the highlights of the video. When the video recommendation is displayed to the user, the target cover image containing the highlight content that matches the user's interests is dynamically selected, thereby increasing the user's interest in the video, thereby improving the overall playback of the video platform.

[0178] Optionally, when determining the matching degree between the interest tag of the target user and at least one target description information, the embodiment of the present application provides at least the following implementation modes:

[0179] Implementation method 1: extract the features of the target user's interest tags to obtain the target user's interest features. For each target description information in at least one target description information, perform the following operations respectively:

[0180] For a target description information, feature extraction is performed on the target description information to obtain a description feature of the target description information, and then the interest feature is fused with the description feature to obtain a first correlation feature. Based on the first correlation feature, the matching degree between the target user's interest tag and the target description information is predicted.

[0181] Specifically, a highlight matching model is used to predict the matching degree between the target user's interest tag and a target description information, wherein the training samples for training the highlight matching model include highlight exposure and click data, where the highlights clicked by the user after exposure are positive samples, and the highlights not clicked are negative samples. The highlight matching model includes an encoder, which can be ALBERT, BERT, Transformer-Encoder, etc.

[0182] For example, the structure of the bright spot matching model is as follows: Fig.15 As shown, the interest tag set of the target user is input into ALBERT to obtain the interest feature of the target user. The highlight content W is input into ALBERT to obtain the description feature of the highlight content W. The interest feature of the target user is fused with the description feature of the highlight content W to obtain the first correlation feature. Based on the first correlation feature, the matching degree between the interest tag of the target user and the highlight content W is predicted, that is, the probability that the target user clicks on the target cover image containing the highlight content W.

[0183] Implementation method 2: extract the features of the target user's interest tags to obtain the target user's interest features. For each target description information in at least one target description information, perform the following operations respectively:

[0184] For a target description information, feature extraction is performed on the target description information to obtain a description feature of the target description information, and then based on the similarity between the interest feature and the description feature, the matching degree between the target user's interest tag and the target description information is predicted.

[0185] For example, the structure of the bright spot matching model is as follows: Fig.16 As shown, the interest tag set of the target user is input into ALBERT to obtain the interest features of the target user. The highlight content W is input into ALBERT to obtain the description features of the highlight content W. Based on the similarity between the interest features of the target user and the description features of the highlight content W, the matching degree between the interest tags of the target user and the highlight content W is predicted, that is, the probability that the target user clicks on the target cover image containing the highlight content W.

[0186] In an embodiment of the present application, before the target video is recommended and displayed to the user, target highlight content that matches the user's interests is selected from the various highlight contents of the target video based on the user's interest tag set, and then a target cover image containing the target highlight content is selected and recommended to the user, so that the user can more intuitively discover the highlights of interest in the video, thereby enhancing the user's experience of watching the video.

[0187] In order to better explain the embodiment of the present application, the following takes the video recommendation scenario as an example to introduce a method for generating a cover image and a video recommendation provided by the embodiment of the present application. The method is executed by the server and includes the following steps: Fig.17 As shown:

[0188] Step S1701, identifying whether the original cover image of the current video includes highlight content.

[0189] The video text of the current video is obtained through voice / image recognition, and the cover text information in the original cover image of the current video is obtained by OCR. The highlight judgment model is used to identify whether the original cover image of the current video includes highlight content. Specifically, the cover text information is input into the encoder (Transformer-Encoder) to obtain the cover text representation; the video text is input into the decoder (Transformer-Encoder) to obtain the video text representation; then the video text representation and the cover text representation are merged to obtain the interactive representation of the video text and the cover text; based on the interactive representation of the video text and the cover text, the probability of the original cover image having highlight content is predicted. If the obtained probability is less than the preset threshold, it is determined that the original cover image does not have highlight content. If the obtained probability is not less than the preset threshold, it is determined that the original cover image has highlight content.

[0190] Step S1702, when it is determined that the original cover image does not contain highlight content, extract at least one highlight content in the current video, and perform cover enhancement on the original cover image based on the highlight content to obtain at least one highlight cover image.

[0191] Specifically, the highlights of the current video can be obtained by using an extraction method or a generation method, or by using an extraction method combined with a generation method. The extraction method specifically uses a sequence annotation model to extract the highlights from the text content of the current video, and the generation method uses a Seq2Seq generation model to generate the highlights based on the text content of the current video. Both the sequence annotation model and the Seq2Seq generation model are obtained by training with videos with pre-labeled highlights as training samples.

[0192] For example, a video with pre-marked highlights is used as a training sample to train a sequence labeling model. The format of the training sample set is as follows:

[0193] Video 1 Video 1 Title Video 1 Text Video 1 Highlights Content;

[0194] Video 2 Video 2 title Video 2 text Video 2 highlights;

[0195] …

[0196] Video v Video v Title Video v Text Video v Highlights Content.

[0197] After the training is completed, the sequence annotation model is used to extract the highlights from the text content of the current video. Specifically, the text content of the current video is set to include the video title, video text-sentence 1, video text-sentence 2, ..., video text-sentence S. ALBERT is used to extract features from the video title to obtain the title representation corresponding to the video title; ALBERT is used to extract features from the video text-sentence 1 to obtain the sentence 1 representation corresponding to the video text-sentence 1; ALBERT is used to extract features from the video text-sentence 2 to obtain the sentence 2 representation corresponding to the video text-sentence 2, and so on, until the sentence S representation corresponding to the video text-sentence S is obtained. After the title representation, sentence 1 representation, sentence 2 representation, sentence 3 representation, ..., sentence S representation are fused, the comprehensive text features of the current video are obtained. The comprehensive text features are input into ALBERT to predict the probability of sentence 1 being the highlight content, the probability of sentence 2 being the highlight content, the probability of sentence 3 being the highlight content, ..., and the probability of sentence S being the highlight content. Then, the sentences with probabilities greater than the screening threshold are retained as the highlights of the current video.

[0198] The R-CNN model is used to detect the original cover image of the current video and determine the target display position of the key elements in the original cover image. For each highlight content, the target addition position of the highlight content is determined from other positions outside the target display position. Then the highlight content is integrated into the target addition position in the original cover image in a preset style to obtain a highlight cover image.

[0199] Step S1703: Select highlight cover images for video recommendation based on user interests.

[0200] The highlight matching model is used to predict the matching degree between the target user's interest tags and each highlight content. Specifically, the target user's interest tag set is input into ALBERT to obtain the target user's interest representation. The highlight content W is input into ALBERT to obtain the highlight content W representation. The target user's interest representation is merged with the highlight content W representation to obtain the interactive representation of the user's interest and the highlight content W. Based on the interactive representation of the user's interest and the highlight content W, the matching degree between the target user's interest tags and the highlight content W is predicted, that is, the probability that the target user clicks on the highlight cover image containing the highlight content W.

[0201] A recommended highlight content that has the highest matching degree with the interest tag of the target user is selected from each highlight content, and when recommending the current video to the target user, a highlight cover image containing the recommended highlight content is used as the cover image of the current video.

[0202] In the embodiment of the present application, by deeply understanding the current video content, highlight content reflecting the main content of the video is constructed, and then the highlight content is added to the appropriate position of the original cover image in a suitable display form, so that users can intuitively and quickly find the highlights of the video. When the video recommendation is displayed to the user, the highlight cover image containing the highlight content that matches the user's interests is dynamically selected, thereby increasing the user's interest in the video, thereby improving the overall playback of the video platform.

[0203] Based on the same technical concept, the embodiment of the present application provides a device for generating a cover image, such as Fig.18 As shown, the device 1800 includes:

[0204] The feature extraction module 1801 is used to extract features of at least one text information in the target video to obtain first text features corresponding to the at least one text information;

[0205] A fusion module 1802 is used to fuse the obtained first text features to obtain comprehensive text features corresponding to the target video;

[0206] A prediction module 1803, configured to predict at least one target description information corresponding to a target video based on the comprehensive text features;

[0207] The combining module 1804 is used to generate at least one target cover image based on the original cover image of the target video and at least one target description information, wherein each target cover image contains at least one target description information.

[0208] Optionally, the prediction module 1803 is specifically used for:

[0209] Based on the comprehensive text features, respectively predicting a first probability that at least one text information is each target description information;

[0210] Based on the obtained first probability, at least one target description information is selected from at least one text information.

[0211] Optionally, the prediction module 1803 is specifically used for:

[0212] Encoding the comprehensive text features to obtain second text features corresponding to at least one text information respectively;

[0213] Decoding the second text features corresponding to the at least one text information respectively, generating at least one candidate description information corresponding to the target video, and a second probability that each candidate description information is the target description information;

[0214] Based on the obtained second probability, at least one target description information is selected from at least one candidate description information.

[0215] Optionally, the at least one target description information includes at least one first-category description information and at least one second-category description information;

[0216] The prediction module 1803 is specifically used for:

[0217] Encoding the comprehensive text features to obtain second text features corresponding to at least one text information respectively;

[0218] Predicting a first probability that each of the at least one text information is the first type of descriptive information based on a second text feature corresponding to each of the at least one text information;

[0219] Based on the obtained first probability, selecting at least one first category of description information from at least one text information;

[0220] Decoding the second text features corresponding to the at least one text information respectively, generating at least one candidate description information corresponding to the target video, and a second probability that each candidate description information is the second type of description information;

[0221] Based on the obtained second probability, at least one second type of description information is selected from at least one candidate description information.

[0222] Optionally, the combining module 1804 is specifically used for:

[0223] Detecting target placements of key elements in the original cover image;

[0224] For each target description information in the at least one target description information, perform the following operations respectively:

[0225] For a target description information, determine a target adding position of the target description information from other positions other than the target display position;

[0226] A target description information is fused to the target adding position in the original cover image to obtain a target cover image containing the target description information.

[0227] Optionally, a recommendation module 1805 is also included;

[0228] The recommendation module 1805 is specifically used for:

[0229] A target description information is integrated into the target adding position in the original cover image, and after obtaining the target cover image containing the target description information, a matching degree between the target user's interest tag and at least one target description information is determined;

[0230] Determining recommended description information from at least one target description information based on the obtained matching degree;

[0231] When recommending a target video to a target user, a target cover image containing the recommendation description information is selected as the cover image of the target video.

[0232] Optionally, the recommendation module 1805 is specifically used for:

[0233] Extract the features of the target user's interest tags to obtain the target user's interest features;

[0234] For each target description information in the at least one target description information, perform the following operations respectively:

[0235] For a target description information, feature extraction is performed on the target description information to obtain a description feature of the target description information;

[0236] The interest feature is integrated with the description feature to obtain a first correlation feature;

[0237] Based on the first association feature, the matching degree between the interest tag of the target user and a target description information is predicted.

[0238] Optionally, the prediction module 1803 is further used for:

[0239] Before predicting at least one target description information corresponding to the target video based on the comprehensive text features, it is determined that the original cover image of the target video does not contain the target description information.

[0240] Optionally, the prediction module 1803 is specifically used for:

[0241] Get the cover text information in the original cover image;

[0242] Extracting features of the cover text information to obtain cover text features corresponding to the cover text information;

[0243] The comprehensive text feature is integrated with the cover text feature to obtain a second correlation feature;

[0244] Based on the second associated feature, predict a third probability that the cover text information is the target description information;

[0245] If the third probability is less than a preset threshold, it is determined that the original cover image does not contain the target description information.

[0246] In the embodiment of the present application, feature extraction is first performed on each text information in the target video to obtain the first text feature of each text information, so that each text information obtains a better feature representation. The various first text features are fused to achieve contextual joint modeling of the relationship between the various text information, so that the obtained comprehensive text features can more accurately represent the content of the target video. Therefore, based on the comprehensive text features, when predicting at least one target description information corresponding to the target video, the accuracy and efficiency of predicting the target description information can be effectively improved. Furthermore, by combining the original cover image of the target video with the target description information, a target cover image that can better represent the highlights of the video is obtained, thereby facilitating users to quickly and intuitively discover video highlights and improving the user's experience of watching videos. When the target video is recommended and displayed to the user, a target cover image containing target description information that matches the user's interests is dynamically selected, thereby increasing the user's interest in the video, thereby improving the overall playback of the video platform.

[0247] Based on the same technical concept, the embodiment of the present application provides a computer device, which may be a terminal or a server, such as Fig.19 As shown, it includes at least one processor 1901 and a memory 1902 connected to the at least one processor. The specific connection medium between the processor 1901 and the memory 1902 is not limited in the embodiment of the present application. Fig.19 For example, the processor 1901 and the memory 1902 are connected via a bus. The bus can be divided into an address bus, a data bus, a control bus, etc.

[0248] In an embodiment of the present application, the memory 1902 stores instructions that can be executed by at least one processor 1901. The at least one processor 1901 can execute the steps included in the above-mentioned method for generating a cover image by executing the instructions stored in the memory 1902.

[0249] Among them, the processor 1901 is the control center of the computer device, which can use various interfaces and lines to connect various parts of the computer device, and generate a target cover image and make video recommendations by running or executing instructions stored in the memory 1902 and calling data stored in the memory 1902. Optionally, the processor 1901 may include one or more processing units, and the processor 1901 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface, and application programs, and the modem processor mainly processes wireless communications. It is understandable that the above-mentioned modem processor may not be integrated into the processor 1901. In some embodiments, the processor 1901 and the memory 1902 may be implemented on the same chip, and in some embodiments, they may also be implemented separately on independent chips.

[0250] Processor 1901 can be a general-purpose processor, such as a central processing unit (CPU), a digital signal processor, an application-specific integrated circuit (ASIC), a field programmable gate array or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component, and can implement or execute the methods, steps and logic block diagrams disclosed in the embodiments of the present application. A general-purpose processor can be a microprocessor or any conventional processor, etc. The steps of the method disclosed in the embodiments of the present application can be directly embodied as a hardware processor for execution, or can be executed by a combination of hardware and software modules in the processor.

[0251] Memory 1902, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer executable programs and modules. Memory 1902 may include at least one type of storage medium, such as flash memory, hard disk, multimedia card, card-type memory, random access memory (Random Access Memory, RAM), static random access memory (Static Random Access Memory, SRAM), programmable read-only memory (Programmable Read Only Memory, PROM), read-only memory (Read Only Memory, ROM), electrically erasable programmable read-only memory (Electrically Erasable Programmable Read-Only Memory, EEPROM), magnetic memory, disk, optical disk, etc. Memory 1902 is any other medium that can be used to carry or store the desired program code in the form of an instruction or data structure and can be accessed by a computer, but is not limited thereto. The memory 1902 in the embodiment of the present application can also be a circuit or any other device that can realize a storage function, for storing program instructions and / or data.

[0252] Based on the same inventive concept, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program executable by a computer device. When the program runs on the computer device, the computer device executes the steps of the above-mentioned method for generating a cover image.

[0253] It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0254] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0255] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0256] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0257] Although the preferred embodiments of the present invention have been described, those skilled in the art may make other changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention.

[0258] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. A method for generating a cover image, characterized in that: include: Performing feature extraction on at least one text information in the target video to obtain first text features corresponding to the at least one text information; The obtained first text features are integrated to obtain comprehensive text features corresponding to the target video; Extracting features of cover text information in the original cover image of the target video to obtain cover text features corresponding to the cover text information; The comprehensive text feature is merged with the cover text feature to obtain a second associated feature; Based on the second associated feature, predict a third probability that the cover text information is the target description information; if the third probability is less than a preset threshold, determine that the original cover image does not contain the target description information; Based on the comprehensive text features, predict at least one target description information corresponding to the target video; At least one target cover image is generated based on the original cover image of the target video and the at least one target description information, wherein each target cover image contains at least one target description information.

2. The method according to claim 1, characterized in that The predicting at least one target description information corresponding to the target video based on the comprehensive text feature includes: Based on the comprehensive text features, respectively predicting a first probability that each of the at least one text information is target description information; Based on the obtained first probability, at least one target description information is selected from the at least one text information.

3. The method according to claim 1, characterized in that The predicting at least one target description information corresponding to the target video based on the comprehensive text feature includes: Encoding the comprehensive text feature to obtain second text features corresponding to the at least one text information respectively; Decoding the second text features corresponding to the at least one text information respectively, generating at least one candidate description information corresponding to the target video, and a second probability that each candidate description information is the target description information; Based on the obtained second probability, at least one target description information is selected from the at least one candidate description information.

4. The method according to claim 1, characterized in that The at least one target description information includes at least one first type of description information and at least one second type of description information; The predicting at least one target description information corresponding to the target video based on the comprehensive text feature includes: Encoding the comprehensive text feature to obtain second text features corresponding to the at least one text information respectively; Predicting, based on the second text feature corresponding to each of the at least one text information, a first probability that each of the at least one text information is the first type of descriptive information; Based on the obtained first probability, selecting at least one first-category description information from the at least one text information; Decoding the second text features corresponding to the at least one text information respectively, generating at least one candidate description information corresponding to the target video, and a second probability that each candidate description information is the second type of description information; Based on the obtained second probability, at least one second type of description information is selected from the at least one candidate description information.

5. The method according to any one of claims 1 to 4, characterized in that: The step of generating at least one target cover image based on the original cover image of the target video and the at least one target description information comprises: Detecting target placement positions of key elements in the original cover image; For each piece of target description information in the at least one target description information, perform the following operations respectively: For a target description information, determining a target adding position of the target description information from other positions other than the target display position; The target description information is integrated into the target adding position in the original cover image to obtain a target cover image including the target description information.

6. The method according to claim 5, characterized in that After fusing the one target description information to the target adding position in the original cover image to obtain the target cover image containing the one target description information, the method further includes: Determine the matching degree between the interest tags of the target user and the at least one target description information respectively; Determining recommended description information from the at least one target description information based on the obtained matching degree; When recommending the target video to the target user, a target cover image containing the recommendation description information is selected as the cover image of the target video.

7. The method according to claim 6, characterized in that The determining the matching degree between the interest tags of the target user and the at least one target description information respectively includes: Extracting features of the target user's interest tags to obtain the target user's interest features; For each piece of target description information in the at least one target description information, perform the following operations respectively: For a target description information, extracting features of the target description information to obtain description features of the target description information; Merging the interest feature with the description feature to obtain a first associated feature; Based on the first association feature, predict the matching degree between the interest tag of the target user and the one target description information.

8. A device for generating a cover image, characterized in that: include: A feature extraction module, used to extract features of at least one text information in the target video, and obtain first text features corresponding to the at least one text information; A fusion module is used to fuse the obtained first text features to obtain comprehensive text features corresponding to the target video; extract features of cover text information in the original cover image of the target video to obtain cover text features corresponding to the cover text information; The comprehensive text feature is merged with the cover text feature to obtain a second associated feature; Based on the second associated feature, predict a third probability that the cover text information is the target description information; if the third probability is less than a preset threshold, determine that the original cover image does not contain the target description information; A prediction module, used for predicting at least one target description information corresponding to the target video based on the comprehensive text features; A combining module is used to generate at least one target cover image based on the original cover image of the target video and the at least one target description information, wherein each target cover image contains at least one target description information.

9. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps of any one of the methods of claims 1 to 7 are implemented.

10. A computer-readable storage medium, characterized in that: It stores a computer program executable by a computer device. When the program is run on the computer device, the computer device executes the steps of any method described in claims 1 to 7.

Images