Video cover extraction method and related apparatus

Abstract

This application discloses a method and related apparatus for extracting video cover images. The method involves extracting at least one frame of an image to be evaluated from a video; then evaluating the image based on a first network model to obtain a first evaluation score; and performing a quality evaluation on the image based on a second network model to obtain a second evaluation score for the aesthetic dimension. The first and second evaluation scores are then fused to obtain a target evaluation score. Finally, a target cover image is extracted from the set of images to be evaluated based on the target evaluation score. This achieves a multimodal fusion evaluation process for cover image extraction. By employing the fusion of multimodal information, the network model's understanding of video semantics is greatly improved, thereby selecting an image that better fits the semantic meaning of the video as the cover image, thus improving the accuracy of video cover image extraction.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a method and apparatus for extracting video cover images. Background Technology

[0002] With the rapid development of internet technology, people have increasingly higher demands for media content. Video thumbnails, as a common method of video traffic generation, appear on various movie and short video platforms. Therefore, setting appropriate video thumbnails has become crucial for video traffic generation.

[0003] Generally, video cover images can be determined by dividing the video into segments based on fixed time points and extracting images from them. For example, a video can be divided into several sub-videos of equal length, or the start time of each sub-video can be used as a fixed time point. Then, images can be extracted from the video as candidate images for the video cover image for users to choose from.

[0004] However, video cover images obtained by extracting video clips often suffer from problems such as blurriness and out-of-focusness, and may also have overly simple images or lack meaningful objects, affecting the accuracy of video cover extraction. Summary of the Invention

[0005] In view of this, this application provides a method for extracting video cover images, which can effectively improve the accuracy of video cover image extraction.

[0006] The first aspect of this application provides a method for extracting video cover images, which can be applied to a system or program in a terminal device that includes a video cover image extraction function, specifically including:

[0007] Extract a set of images to be evaluated from the video to be processed, the set of images to be evaluated including at least one frame of image to be evaluated;

[0008] A first evaluation is performed on the image to be evaluated based on a first network model to obtain a first evaluation score. The first network model includes an image evaluation branch and a semantic evaluation branch. The image evaluation branch is used to extract the image feature vector of the image to be evaluated, and the semantic evaluation branch is used to extract the semantic feature vector of the image to be evaluated. The image evaluation branch and the semantic evaluation branch share an encoder. The encoder is used to fuse the image feature vector and the semantic feature vector to obtain a fused feature vector. The image evaluation branch performs an image dimension quality evaluation on the image to be evaluated based on the fused feature vector. The semantic evaluation branch performs a semantic dimension quality evaluation on the correlation between the semantic information in the video to be processed and the image to be evaluated based on the fused feature vector. The first evaluation score is obtained by combining the evaluation results of the image evaluation branch and the semantic evaluation branch.

[0009] The image to be evaluated is evaluated based on the second network model to obtain a second evaluation score. The second network model is used to evaluate the quality of the image to be evaluated in the aesthetic dimension.

[0010] The target evaluation score is obtained by fusing the first evaluation score and the second evaluation score.

[0011] Based on the target evaluation score, a target image is extracted from the image group to be evaluated. The target image is the video cover image of the video to be processed.

[0012] Optionally, in some possible implementations of this application, the step of extracting the target image from the group of images to be evaluated based on the target evaluation score includes:

[0013] Obtain the low-quality image features corresponding to the video to be processed;

[0014] Images that match the low-quality image features in the image group to be evaluated are filtered out, so that the image group to be evaluated is updated to the first evaluation image group;

[0015] Based on the target evaluation score, the images to be evaluated in the first evaluation image group are sorted to obtain an evaluation sequence;

[0016] The target image is extracted from the first evaluation image group according to the image order in the evaluation sequence.

[0017] Optionally, in some possible implementations of this application, after sorting the image group to be evaluated in the first evaluation image group based on the target evaluation score to obtain the evaluation sequence, the method further includes:

[0018] Key image elements corresponding to the first evaluation image group are extracted based on preset rules;

[0019] The order of images whose image features corresponding to the key image elements in the evaluation sequence meet the preset requirements is advanced to update the sequence order of images in the evaluation sequence.

[0020] Optionally, in some possible implementations of this application, the step of advancing the image order of the key image elements in the evaluation sequence to update the sequence order of the images in the evaluation sequence, in order to advance the image order of the key image elements in the evaluation sequence, includes:

[0021] Determine target description information based on semantic information in the video to be processed;

[0022] Determine the matching information between the target description information and the image features corresponding to the key image elements;

[0023] If the matching information meets the preset requirements, the corresponding image is moved forward to update the sequence order of the images in the evaluation sequence.

[0024] Optionally, in some possible implementations of this application, the method further includes:

[0025] Text recognition is performed on the images in the evaluation sequence to obtain text information;

[0026] The sequence order of images in the evaluation sequence is updated based on the degree of matching between the text information and the target description information.

[0027] Optionally, in some possible implementations of this application, the method further includes:

[0028] Obtain the initial cover corresponding to the video to be processed, wherein the initial cover is the cover marked in the video to be processed;

[0029] The initial cover is input into the first network model for evaluation to obtain a third evaluation score;

[0030] The initial cover is input into the second network model for evaluation to obtain a fourth evaluation score;

[0031] The third evaluation score and the fourth evaluation score are fused to obtain the initial evaluation score;

[0032] The initial evaluation score is compared with the target evaluation score to determine the video cover image of the video to be processed from the target image and the initial cover.

[0033] Optionally, in some possible implementations of this application, the method further includes:

[0034] In response to a setting operation by a target user, a setting image in the video to be processed is determined, and the setting image is used as a positive sample.

[0035] Images whose difference value from the set image reaches a difference threshold are selected from the video to be processed as negative samples;

[0036] Training samples are extracted from the positive and negative samples according to the sample ratio.

[0037] The image evaluation branch and the semantic evaluation branch are trained based on the training samples to adjust the parameters of the first network model.

[0038] A second aspect of this application provides a device for extracting video cover images, comprising:

[0039] An extraction unit is used to extract a group of images to be evaluated from the video to be processed, the group of images to be evaluated including at least one frame of image to be evaluated.

[0040] An evaluation unit is used to perform a first evaluation on the image to be evaluated based on a first network model to obtain a first evaluation score. The first network model includes an image evaluation branch and a semantic evaluation branch. The image evaluation branch is used to extract the image feature vector of the image to be evaluated, and the semantic evaluation branch is used to extract the semantic feature vector of the image to be evaluated. The image evaluation branch and the semantic evaluation branch share an encoder. The encoder is used to fuse the image feature vector and the semantic feature vector to obtain a fused feature vector. The image evaluation branch performs an image dimension quality evaluation on the image to be evaluated based on the fused feature vector. The semantic evaluation branch performs a semantic dimension quality evaluation on the correlation between the semantic information in the video to be processed and the image to be evaluated based on the fused feature vector. The first evaluation score is obtained by combining the evaluation results of the image evaluation branch and the semantic evaluation branch.

[0041] The evaluation unit is further configured to perform a second evaluation on the image to be evaluated based on a second network model to obtain a second evaluation score. The second network model is used to perform a quality evaluation of the image to be evaluated in the aesthetic dimension.

[0042] A fusion unit is used to fuse the first evaluation score and the second evaluation score to obtain a target evaluation score;

[0043] The extraction unit is further configured to extract a target image from the image group to be evaluated based on the target evaluation score, wherein the target image is the video cover image of the video to be processed.

[0044] Optionally, in some possible implementations of this application, the extraction unit is specifically used to obtain low-quality image features corresponding to the video to be processed;

[0045] The extraction unit is specifically used to filter out images that hit the low-quality image features in the image group to be evaluated, so as to update the image group to be evaluated into the first evaluation image group.

[0046] The extraction unit is specifically used to sort the image group to be evaluated in the first evaluation image group based on the target evaluation score to obtain an evaluation sequence;

[0047] The extraction unit is specifically used to extract the target image from the first evaluation image group according to the image order in the evaluation sequence.

[0048] Optionally, in some possible implementations of this application, the extraction unit is specifically used to extract key image elements corresponding to the first evaluation image group based on preset rules;

[0049] The extraction unit is specifically used to advance the image order of the key image elements in the evaluation sequence that meet the preset requirements, so as to update the sequence order of the images in the evaluation sequence.

[0050] Optionally, in some possible implementations of this application, the extraction unit is specifically used to determine target description information based on the semantic information in the video to be processed;

[0051] The extraction unit is specifically used to determine the matching information between the target description information and the image features corresponding to the key image elements;

[0052] The extraction unit is specifically used to advance the corresponding image if the matching information meets the preset requirements, so as to update the sequence order of the images in the evaluation sequence.

[0053] Optionally, in some possible implementations of this application, the extraction unit is specifically used to perform text recognition on the images in the evaluation sequence to obtain text information;

[0054] The extraction unit is specifically used to update the sequence order of images in the evaluation sequence based on the degree of matching between the text information and the target description information.

[0055] Optionally, in some possible implementations of this application, the evaluation unit is specifically used to obtain the initial cover corresponding to the video to be processed, wherein the initial cover is the cover marked in the video to be processed;

[0056] The evaluation unit is specifically used to input the initial cover into the first network model for evaluation to obtain a third evaluation score;

[0057] The evaluation unit is specifically used to input the initial cover into the second network model for evaluation to obtain a fourth evaluation score;

[0058] The evaluation unit is specifically used to fuse the third evaluation score and the fourth evaluation score to obtain an initial evaluation score;

[0059] The evaluation unit is specifically used to compare the initial evaluation score with the target evaluation score to determine the video cover image of the video to be processed from the target image and the initial cover.

[0060] Optionally, in some possible implementations of this application, the evaluation unit is specifically used to determine the setting image in the video to be processed in response to the setting operation of the target user, and to use the setting image as a positive sample;

[0061] The evaluation unit is specifically used to select images from the video to be processed whose difference value with the set image reaches a difference threshold as negative samples.

[0062] The evaluation unit is specifically used to extract samples from the positive and negative samples according to the sample ratio to obtain training samples;

[0063] The evaluation unit is specifically used to train the image evaluation branch and the semantic evaluation branch based on the training samples, so as to adjust the parameters of the first network model.

[0064] A third aspect of this application provides a computer device, comprising: a memory, a processor, and a bus system; the memory is used to store program code; the processor is used to execute the video cover extraction method described in the first aspect or any one of the first aspects according to the instructions in the program code.

[0065] A fourth aspect of this application provides a computer-readable storage medium storing instructions that, when executed on a computer, cause the computer to perform the video cover extraction method described in the first aspect or any one of the first aspects.

[0066] According to one aspect of this application, a computer program product or computer program is provided, comprising computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the video cover extraction method provided in the first aspect or various optional implementations thereof.

[0067] As can be seen from the above technical solutions, the embodiments of this application have the following advantages:

[0068] The evaluation process involves extracting a set of images to be evaluated from the video to be processed, each set including at least one frame of the image to be evaluated. A first evaluation is then performed on the images to be evaluated based on a first network model to obtain a first evaluation score. This first network model includes an image evaluation branch and a semantic evaluation branch. The image evaluation branch extracts image feature vectors from the images to be evaluated, and the semantic evaluation branch extracts semantic feature vectors from the images to be evaluated. Both branches share a common encoder, which fuses the image feature vectors and semantic feature vectors to obtain a fused feature vector. The image evaluation branch performs an image-dimensional quality evaluation on the images to be evaluated based on the fused feature vector, and the semantic evaluation branch performs a semantic-dimensional quality evaluation based on the correlation between the semantic information in the video to be processed and the images to be evaluated. The first evaluation score is obtained by combining the evaluation results from the image evaluation branch and the semantic evaluation branch. A second evaluation is then performed on the images to be evaluated based on a second network model to obtain a second evaluation score. This second network model performs an aesthetic-dimensional quality evaluation on the images to be evaluated. The first and second evaluation scores are then fused to obtain a target evaluation score. Finally, a target image, which is the video cover image of the video to be processed, is extracted from the set of images to be evaluated based on the target evaluation score. This enables the cover extraction process to achieve multimodal fusion evaluation. By combining the correlation between semantic modality and image modality during the multimodal information fusion process and using the same encoder to fuse semantic features and image features, the network model's ability to understand video semantics can be improved. The image can be evaluated from the dimensions of image, semantics, and aesthetics, thereby selecting an image that better fits the semantic meaning of the video as the cover image, thus improving the accuracy of video cover extraction. Attached Figure Description

[0069] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of this application. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0070] Figure 1 Network architecture diagram of the system for extracting video cover images;

[0071] Figure 2 A flowchart illustrating the process architecture for extracting video covers, provided as an embodiment of this application;

[0072] Figure 3 A flowchart illustrating a method for extracting video covers, provided as an embodiment of this application;

[0073] Figure 4A schematic diagram of a video cover extraction method provided in an embodiment of this application;

[0074] Figure 5 A schematic diagram of a model for another method of extracting video covers provided in an embodiment of this application;

[0075] Figure 6 A schematic diagram illustrating the steps of another video cover extraction method provided in this application embodiment;

[0076] Figure 7 A flowchart illustrating another method for extracting video covers provided in this application embodiment;

[0077] Figure 8 A schematic diagram illustrating a method for extracting video covers according to an embodiment of this application;

[0078] Figure 9 A schematic diagram of a video cover extraction device provided in an embodiment of this application;

[0079] Figure 10 This is a schematic diagram of the structure of a terminal device provided in an embodiment of this application;

[0080] Figure 11 This is a schematic diagram of the structure of a server provided in an embodiment of this application. Detailed Implementation

[0081] This application provides a method and related apparatus for extracting video cover images, which can be applied to systems or programs in terminal devices that include video cover image extraction functionality. The method involves extracting a group of images to be evaluated from a video to be processed, the group including at least one frame of the image to be evaluated; then performing a first evaluation on the images to be evaluated based on a first network model to obtain a first evaluation score. The first network model includes an image evaluation branch and a semantic evaluation branch. The image evaluation branch extracts image feature vectors from the images to be evaluated, and the semantic evaluation branch extracts semantic feature vectors from the images to be evaluated. The image evaluation branch and the semantic evaluation branch share an encoder, which fuses the image feature vectors and semantic feature vectors to obtain a first evaluation score. The process involves fusing feature vectors. The image evaluation branch performs an image-dimensional quality assessment of the image to be evaluated based on the fused feature vectors, while the semantic evaluation branch performs a semantic-dimensional quality assessment based on the relevance between the semantic information in the video and the image to be evaluated. The first evaluation score is obtained by combining the evaluation results of the image and semantic evaluation branches. Then, a second evaluation is performed on the image to be evaluated using a second network model to obtain a second evaluation score. This second network model is used to perform an aesthetic-dimensional quality assessment of the image. The first and second evaluation scores are then fused to obtain a target evaluation score. Finally, a target image, which serves as the video cover image, is extracted from the image group to be evaluated based on the target evaluation score. This achieves a multimodal fusion evaluation cover image extraction process. By combining the correlation between semantic and image modalities during multimodal information fusion and using the same encoder to fuse semantic and image features, the network model's ability to understand video semantics is improved. The image is evaluated from image, semantic, and aesthetic dimensions, thus selecting an image that better fits the semantic meaning of the video as the cover image, improving the accuracy of video cover image extraction.

[0082] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a particular order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented, for example, in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “corresponding to,” and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0083] First, some terms that may appear in the embodiments of this application will be explained.

[0084] Deep Behavior Recognition (Temporal Segment Networks, TSN): Combines sparse temporal sampling strategies with video-level monitoring to enable effective learning of the entire action video.

[0085] Multimodal encoder (BimodaL EncoDer, BLENDer): An encoder that fuses multimodal information.

[0086] BERT: The BERT network architecture uses the multi-layer Transformer structure proposed in "Attention is all you need". Its biggest feature is that it abandons the traditional RNN and CNN and uses the Attention mechanism to convert the distance between two words at any position into 1, which effectively solves the tricky long-term dependency problem in natural language recognition (NLP).

[0087] It should be understood that the video cover extraction method provided in this application can be applied to systems or programs in terminal devices that include video cover extraction functionality, such as video playback platforms. Specifically, the video cover extraction system can run on systems such as... Figure 1 In the network architecture shown, such as Figure 1 The diagram shown illustrates the network architecture of the video cover extraction system. As can be seen, the system can extract video covers from multiple information sources. Specifically, it extracts covers from multiple videos sent from the server via upload operations on the terminal side for display. This means that... Figure 1 The document shows various terminal devices, which can be computer devices. In real-world scenarios, more or fewer types of terminal devices may be involved in the video cover extraction process. The specific number and types depend on the actual scenario and are not limited here. Figure 1 The example shows one server, but in real-world scenarios, multiple servers can be involved, especially in scenarios involving multi-model training and interaction. The specific number of servers depends on the actual scenario.

[0088] In this embodiment, the server can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The terminal can be a smartphone, tablet, laptop, desktop computer, smart speaker, smartwatch, smart voice interaction device, smart home appliance, in-vehicle terminal, etc., but is not limited to these. The terminal and server can be directly or indirectly connected via wired or wireless communication, and the terminal and server can be connected to form a blockchain network; this application does not impose any restrictions.

[0089] It is understood that the aforementioned video cover extraction system can run on personal mobile terminals, such as video playback platforms, or on servers, or on third-party devices to provide video cover extraction and obtain the extraction and processing results of the video cover from the information source. Specifically, the video cover extraction system can run as a program on the aforementioned devices, or as a system component within the aforementioned devices, or as a cloud service program. This embodiment can be applied to various scenarios such as cloud technology, artificial intelligence, smart transportation, and assisted driving. The specific operating mode depends on the actual scenario and is not limited here.

[0090] With the rapid development of internet technology, people have increasingly higher demands for media content. Video thumbnails, as a common method of video traffic generation, appear on various movie and short video platforms. Therefore, setting appropriate video thumbnails has become crucial for video traffic generation.

[0091] Generally, video cover images can be determined by dividing the video into segments based on fixed time points and extracting images from them. For example, a video can be divided into several sub-videos of equal length, or the start time of each sub-video can be used as a fixed time point. Then, images can be extracted from the video as candidate images for the video cover image for users to choose from.

[0092] However, video cover images obtained by extracting video clips often suffer from problems such as blurriness and out-of-focusness, and may also have overly simple images or lack meaningful objects, affecting the accuracy of video cover extraction.

[0093] Furthermore, with the rapid development of deep machine learning technology and its significant advancements in image and speech recognition, a solution for automatically generating video thumbnails based on deep machine learning technology has been developed to address the aforementioned problems in selecting video cover images. This solution utilizes a deep neural network (DNN). User-uploaded images used as video cover images serve as a "high-quality" training set, while randomly selected images from the video file serve as a "low-quality" training set. The DNN-based machine learning model is then pre-trained using both the "high-quality" and "low-quality" training sets to obtain a pre-trained DNN model. During the video thumbnail generation process, images are randomly selected from the video file (e.g., one frame per second). The pre-trained DNN model then scores the selected images, and the best image (or several images) with the highest score is selected as the video cover. In this scenario, directly using user-uploaded images as the "high-quality" training set and images captured from videos at fixed time points as the "low-quality" training set introduces a large amount of "dirty data." That is, user-uploaded images may contain many low-quality images, and images captured from videos at fixed time points may contain many high-quality images. Therefore, this training set containing "dirty data" will directly lead to the trained machine learning model failing to achieve good classification results.

[0094] To address the aforementioned issues, this application proposes a method for extracting video cover images. This application employs a multimodal recognition process combining computer vision and natural language processing. Specifically, Computer Vision (CV) is a science that studies how machines "see," or more specifically, it refers to machine vision that uses cameras and computers to replace human eyes for target recognition, tracking, and measurement, and further performs image processing to create images more suitable for human observation or transmission to instruments for detection. As a scientific discipline, computer vision researches related theories and technologies, attempting to establish artificial intelligence systems capable of extracting information from images or multidimensional data. Computer vision technologies typically include image processing, image recognition, image semantic understanding, image retrieval, OCR, video processing, video semantic understanding, video content / behavior recognition, 3D object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous localization and mapping (SLAM), and other technologies, as well as common biometric recognition technologies such as facial recognition and fingerprint recognition.

[0095] Furthermore, Natural Language Processing (NLP) is an important field within computer science and artificial intelligence. It studies the theories and methods for enabling effective communication between humans and computers using natural language. NLP is a science that integrates linguistics, computer science, and mathematics. Therefore, research in this field involves natural language—the language people use in daily life—and thus it has a close connection with linguistic research. NLP techniques typically include text processing, semantic understanding, machine translation, question answering, and knowledge graphs.

[0096] In this application, the method for extracting video cover images is applied to... Figure 2 In the workflow framework shown for extracting video cover images, such as... Figure 2 The diagram shown illustrates a flowchart for extracting a video cover according to an embodiment of this application. First, multimodal fusion of computer vision and natural language recognition is used to score each video frame, thereby extracting at least one image frame to be evaluated from the video. Then, the extracted images are filtered out from one or more perspectives, such as clarity, brightness, and meaninglessness in monochrome images. Next, the remaining video frames are evaluated according to the video description. Based on the evaluation results, an image frame that meets preset cover conditions is selected from the at least one image frame to be evaluated as the cover of the video to be processed. This targeted selection of the cover is beneficial for the promotion of the video.

[0097] It is understood that the method provided in this application can be a program written as processing logic in a hardware system, or a video cover extraction device, implemented in an integrated or external manner to achieve the aforementioned processing logic. As one implementation, the video cover extraction device extracts at least one frame of an image to be evaluated from the video to be processed; then evaluates the image to be evaluated based on a first network model to obtain a first evaluation score. The first network model includes an image evaluation branch and a semantic evaluation branch, and the first evaluation score is based on the combined evaluation results of the image evaluation branch and the semantic evaluation branch; and performs a quality evaluation on the image to be evaluated based on a second network model to obtain a second evaluation score, which is the quality score corresponding to the aesthetic evaluation rules; then, the first evaluation score and the second evaluation score are fused to obtain a target evaluation score; further, a target image is extracted from the image group to be evaluated based on the target evaluation score. The target image is the video cover image of the video to be processed. This achieves a multimodal fusion evaluation cover extraction process. Due to the fusion of multimodal information, the network model's understanding of video semantics can be greatly improved, thereby selecting an image that better fits the semantic meaning of the video as the cover image, improving the accuracy of video cover extraction.

[0098] The solutions provided in this application relate to computer vision technology and natural language recognition technology in artificial intelligence, and are specifically illustrated through the following embodiments:

[0099] Based on the above process architecture, the method for extracting the video cover in this application will be described below. Please refer to [link / reference]. Figure 3 , Figure 3 The flowchart illustrates a method for extracting video cover images according to an embodiment of this application. This management method can be executed by a terminal or a server. The embodiment of this application includes at least the following steps:

[0100] 301. Extract the image group to be evaluated from the video to be processed. The image group to be evaluated includes at least one frame of the image to be evaluated.

[0101] In this embodiment, the video to be processed can be a video uploaded by the user during the use of the video platform. The specific video type can be a movie, an edited movie clip, a self-portrait, etc.; correspondingly, the image group to be evaluated is the video frame obtained by parsing the video to be processed.

[0102] Specifically, the video frame parsing process can be frame-by-frame output, that is, processing each frame in the video to be processed; or it can be an interval-based frame extraction method, such as extracting a video frame every 1 second as a group of images to be evaluated. Therefore, the group of images to be evaluated includes at least one image (video frame) to be evaluated, and the specific number depends on the actual scenario.

[0103] In one possible scenario, considering the varying richness of different video content, longer videos (e.g., less than 1 hour) can have their frames extracted at intervals, while shorter videos (e.g., less than 5 minutes) can have their frames extracted one by one, thereby improving the efficiency of video processing.

[0104] 302. Perform a first evaluation on the image to be evaluated based on the first network model to obtain a first evaluation score.

[0105] In this embodiment, the first network model includes an image evaluation branch and a semantic evaluation branch. The image evaluation branch is used to extract the image feature vector of the image to be evaluated, and the semantic evaluation branch is used to extract the semantic feature vector of the image to be evaluated. The image evaluation branch and the semantic evaluation branch share an encoder. The encoder is used to fuse the image feature vector and the semantic feature vector to obtain a fused feature vector. The image evaluation branch performs image dimension quality evaluation on the image to be evaluated based on the fused feature vector, and the semantic evaluation branch performs semantic dimension quality evaluation on the correlation between the semantic information in the video to be processed and the image to be evaluated based on the fused feature vector. The first evaluation score is obtained by combining the evaluation results of the image evaluation branch and the semantic evaluation branch. That is, the first network model can perform quality evaluation from both the image dimension and the semantic dimension of the image to be evaluated. Since semantics often has a correlation with video content, this application uses different molecules of the same model to extract multimodal information. Furthermore, by using the image dimension and the semantic dimension features to share an encoder, the generation efficiency of the fused feature vector is improved, and the correlation between the features of the image evaluation branch and the semantic evaluation branch is enhanced.

[0106] Specifically, the structure of the first network model is as follows: Figure 4 As shown, Figure 4 This diagram illustrates a model for a video cover extraction method provided in this application embodiment. The diagram shows a network model with two embedders: an ImageEmbedder (image feature vector embedding) that extracts the CV embedding (visual vector) of each frame of the video using a pre-trained TSN model, and a TextEmbedder (semantic feature vector embedding) that extracts the text embedding (semantic feature vector) of the video description by referencing the input of BERT. Both are then directly fed into a Transformer for modeling, fusing the two modalities. In this embodiment, the first network model is a single-stream Transformer structure. Unlike dual-stream models, the two modalities (image and semantic) in this embodiment share a single encoder, which reduces model complexity while improving the correlation between multimodal models.

[0107] Specifically, for Figure 4 The computer vision branch shown is used to extract the feature vector of the image to be evaluated and to evaluate the quality based on the feature vector; while the natural language recognition branch is used to evaluate the relevance between the semantic information in the video to be processed and the image to be evaluated; further, the first evaluation score is obtained by fusing the evaluation results of the image evaluation branch and the semantic evaluation branch using a multi-modal coding method.

[0108] Understandably, the use of the BERT structure for vector embedding in this embodiment stems from the significant breakthroughs in natural language generation and understanding algorithms, from the Transformer model to BERT and GPT. Inspired by this, the Visual-Linguistic BERT (VL BERT) series of algorithms emerged in the multimodal field. Therefore, a multimodal Transformer structure can be used to replace the traditional NEXVLAD for multimodal fusion. Specifically, in terms of the overall algorithm, TSN is first used to train on the target task to obtain CV embeddings, and then Blender is used to jointly train CV and NLP to obtain the final result. In addition, BERT's network architecture uses a multi-layer Transformer structure, the most significant feature of which is that it abandons the traditional RNN and CNN, and uses the Attention mechanism to convert the distance between two words at any position into 1, effectively solving the thorny long-term dependency problem in NLP.

[0109] The training process of the first network model is described below. Specifically, in response to the target user's setting operation, a setting image in the video to be processed is determined and used as a positive sample. Then, images in the video to be processed whose difference value reaches a difference threshold with the setting image are selected as negative samples. For example, the cover frame selected by the user in a video is used as a positive sample, and other frames that are not similar to the cover are used as negative samples. Samples are extracted from the positive and negative samples according to the sample ratio to obtain training samples. For example, during training, the cover task is to randomly sample a positive or negative image with a 50% probability each time for training. Then, the image evaluation branch and semantic evaluation branch are trained based on the training samples to adjust the parameters of the first network model.

[0110] Specifically, the training process involves simultaneously training the cover task (image evaluation branch) and the classification task (semantic evaluation branch) using TSN; then, the trained model extracts the CV embedding for each frame, and uses Blender to jointly train TSN and BERT on the extracted CV embedding, with the training objectives being the cover task and the classification task.

[0111] Understandably, the overall network model has two embedders: the Image Embedder extracts the CV embedding for each frame of the video using the previously trained TSN model, and the Text Embedder extracts the text embedding describing the video by referring to the input of BERT. Both are then directly fed into the Transformer for modeling, fusing the two modalities. In summary, the first network model is a single-stream Transformer structure, unlike the two-stream model where both modalities share a single encoder.

[0112] 303. Perform a second evaluation on the image to be evaluated based on the second network model to obtain a second evaluation score.

[0113] In this embodiment, the second network model is set based on aesthetic evaluation rules. That is, the second network model is used to evaluate the aesthetic quality of the image to be evaluated. Therefore, the second evaluation score indicates the quality score corresponding to the aesthetic evaluation rules. This is because, from a human perspective, images with different content should be assigned different aesthetic evaluation concepts. For example, the aesthetic evaluation of faces and scenery should not be the same (e.g., scenery scores higher than faces). Therefore, the parameters of the image content model are adaptively adjusted. Based on this, we adopt an adaptive image evaluation network structure, designing the network structure parameters to change with semantic changes. That is, different evaluation mechanisms are used for different types of images to be evaluated (landscapes, people, etc.).

[0114] Specifically, the structure of the second network model is as follows: Figure 5 As shown, Figure 5 This diagram illustrates another method for extracting video cover images provided in this application. The second network model is divided into three parts: semantic feature extraction, perceptual rule establishment, and quality prediction. The semantic feature extraction part uses ResNet50 as the network's backbone. The quality prediction part performs global average pooling on the image features at different scales generated in the previous step, then concatenates them with an embedding, and inputs them into a four-layer fully connected network to obtain the final result. For the perceptual rule establishment part, the output of the first part is passed through three Conv layers to obtain the weights and bases of the fully connected (FC) layer in the second layer, thus enabling different evaluation mechanisms to be applied to images of different types (landscapes, people, etc.).

[0115] For the training process of the second network model, the annotation cost is too high. High-quality online data can be used as high-quality videos, and the corresponding cover images can also be high-quality, while low-quality videos are low-quality. Training data can be obtained in this way to train the second network model.

[0116] 304. The target assessment score is obtained by merging the first assessment score and the second assessment score;

[0117] In this embodiment, the process of fusing the first evaluation score and the second evaluation score can be summation or weighted summation, etc.

[0118] Specifically, xgboost can be used to fuse the scores from steps 302 and 303 to obtain a comprehensive score (target evaluation score). Since xgboost has the characteristics of high training efficiency, good prediction effect, many controllable parameters, and ease of use, it can improve the accuracy of the target evaluation score.

[0119] 305. Extract the target image from the image group to be evaluated based on the target evaluation score. The target image is the video cover image of the video to be processed.

[0120] In this embodiment, the target image is the video cover image of the video to be processed. Specifically, the cover image can be the target image directly, or it can be a dynamic image obtained from the target image. That is, the adjacent frames of the video frame corresponding to the target image are collected and merged into a dynamic image, so as to perform the process of identifying exciting content and obtaining the GIF representing the climax of the video, thereby improving the richness of the cover display.

[0121] In this embodiment, to avoid extraction errors caused by abnormal individual image scores, the process of extracting target images from the image group to be evaluated based on the target evaluation score can be performed by sorting and filtering. That is, firstly, low-quality image features (such as clarity, vulgarity, horror, meaningless monochrome images, etc.) corresponding to the video to be processed are obtained; then, images that hit the low-quality image features in the image group to be evaluated are filtered out to update the image group to be evaluated into the first evaluation image group; then, the image group to be evaluated in the first evaluation image group is sorted based on the target evaluation score (e.g., according to the evaluation score) to obtain the evaluation sequence; then, the target image is extracted from the first evaluation image group according to the image order in the evaluation sequence (e.g., the image with the highest evaluation score is selected).

[0122] Optionally, since different types of images may correspond to different low-quality image features, a targeted identification process can be performed. First, determine the video description type of the video to be processed (landscape, people); then, obtain the corresponding low-quality image features based on the video description type (e.g., clarity, vulgarity, horror, meaningless monochrome images, etc.); further, filter out images in the image group to be evaluated that match the low-quality image features to update the image group to be evaluated as the first evaluation image group; then, sort the image groups to be evaluated in the first evaluation image group based on the target evaluation score (e.g., according to the evaluation score from highest to lowest) to obtain the evaluation sequence; finally, extract the target image from the first evaluation image group according to the image order in the evaluation sequence (e.g., select the image with the highest evaluation score), thereby improving the targeting of low-quality image judgment and the accuracy of image screening.

[0123] In one possible scenario, the order of the evaluation sequence can also be updated based on the display of key image elements. That is, firstly, key image elements corresponding to the first evaluation image group are extracted based on preset rules (for example, key image elements in a landscape video are animals or other objects; key image elements in a people video are human eyes or specific people, etc.); then, the image order of the images whose image features corresponding to the key image elements in the evaluation sequence meet the preset requirements is advanced to update the sequence order of the images in the evaluation sequence.

[0124] Specifically, regarding the setting of preset rules, when the first evaluation image group indicates the type of person, the image region (key image element) corresponding to the eye features is identified, and the eye-opening judgment is further performed on the image region corresponding to the identified eye features; wherein, the eye features can be learned by using the input of eye sample images, so that the image region corresponding to the eye features can be identified.

[0125] It is understandable that the key image elements set in the preset requirements can be one or more. For example, the preset requirements are that the key image elements include a specific person and that person is in an open-eyed state. The specific number depends on the actual scene.

[0126] In another possible scenario, the order of the evaluation sequence can be updated based on the matching of semantic information and image features in the video to be processed. First, target description information is determined based on the semantic information in the video to be processed. The target description information is the information used to summarize the video content, which can be represented by a title, theme, keywords, etc. Then, the matching information between the target description information and the image features corresponding to the key image elements is determined. If the matching information meets the preset requirements, the corresponding image is moved forward to update the sequence order of the images in the evaluation sequence. For example, the star corresponding to the feature in the figure matches the target description information determined in the semantic information.

[0127] Furthermore, since images can also contain text, text recognition matching can be performed. That is, firstly, text recognition is performed on the images in the evaluation sequence to obtain text information; then, the sequence order of the images in the evaluation sequence is updated based on the degree of matching between the text information and the target description information, thereby improving the accuracy of the order.

[0128] In another possible scenario, the original video cover (the first frame or initial cover) can be compared for evaluation scores. First, the initial cover corresponding to the video to be processed is obtained; then, the initial cover is input into a first network model for evaluation to obtain a third evaluation score; further, the initial cover is input into a second network model for evaluation to obtain a fourth evaluation score, and the third and fourth evaluation scores are fused to obtain the initial evaluation score. The specific processes of evaluation by the first and second network models and fusion are described in the above embodiments and will not be repeated here. The initial evaluation score is then compared with the target evaluation score to determine the video cover image of the video to be processed from the target image and the initial cover. Matching the initial cover facilitates the evaluation of the network model performance in this embodiment by relevant personnel. Alternatively, the initial cover and a randomly selected cover can be used for the same video to increase user clicks.

[0129] Below, we integrate the possible extraction processes in the above embodiments using a specific workflow step, such as... Figure 6 As shown, Figure 6 A schematic diagram illustrating the steps of another video cover extraction method provided in this application embodiment; the diagram shows:

[0130] Step 1: Extract frames per second from the user-uploaded video (the video to be evaluated).

[0131] Step 2: Extract CV embeddings for each second of video frame using TSN, and then use a pre-trained blender network (first network model) to score each video frame (image to be evaluated).

[0132] Step 3: Evaluate and score the image quality (second network model) of each frame.

[0133] Step 4: Use XGBoost to merge the scores from Step 2 and Step 3 to obtain a comprehensive score, and select the 5 video frames with the highest scores that are above the threshold.

[0134] Step 5: Judge the selected images based on clarity, vulgarity, horror, and the meaninglessness of monochrome images. If any of these criteria are met, discard the image.

[0135] Step 6: Determine whether the remaining images from Step 3 are open or closed, and move the open-eye images to an earlier position.

[0136] Step 7: Perform celebrity face recognition on the remaining images from Step 4. If the person identified in the image also appears in the text recognition (OCR) or video description, then rank them higher.

[0137] Step 8: Extract OCR from the remaining images in Step 4, and compare the OCR of each image with the video description. If the image OCR is close to the video description, move the image to an earlier position.

[0138] Step 9: Finally, output the images according to the final image ranking order and compare them with the quality scores of the original cover images. The first one selected is used as the machine-selected smart cover.

[0139] As can be seen from the above embodiments, by extracting a group of images to be evaluated from the video to be processed, the group of images to be evaluated includes at least one frame of image to be evaluated; then, a first evaluation is performed on the image to be evaluated based on a first network model to obtain a first evaluation score, wherein the first network model includes an image evaluation branch and a semantic evaluation branch. The image evaluation branch is used to extract the image feature vector of the image to be evaluated, and the semantic evaluation branch is used to extract the semantic feature vector in the image to be evaluated. The image evaluation branch and the semantic evaluation branch share an encoder, which is used to fuse the image feature vector and the semantic feature vector to obtain a fused feature vector. The image evaluation branch then evaluates the semantic feature vector based on the fused feature vector. The image to be evaluated undergoes quality assessment at the image dimension. The semantic assessment branch assesses the semantic quality based on the relevance between the semantic information in the video and the image to be evaluated, using fused feature vectors. The first evaluation score is obtained by combining the evaluation results of the image and semantic assessment branches. Then, a second evaluation is performed on the image to be evaluated using a second network model to obtain a second evaluation score. This second network model is used for aesthetic quality assessment of the image. The first and second evaluation scores are then fused to obtain a target evaluation score. Finally, a target image is extracted from the image group to be evaluated based on the target evaluation score; the target image is the video cover image of the video to be processed. This achieves the cover image extraction process for multimodal fusion evaluation. Because the correlation between semantic and image modalities is combined during multimodal information fusion, and the same encoder is used to fuse semantic and image features, the network model's ability to understand video semantics is improved. The image is evaluated from image, semantic, and aesthetic dimensions, thus selecting an image that better fits the semantic meaning of the video as the cover image, improving the accuracy of video cover extraction.

[0140] The above embodiments describe the extraction process for a single cover. In real-world scenarios, multiple covers or media composed of multiple cover images can also be displayed. The following describes this scenario. Please refer to... Figure 7 , Figure 7 A flowchart illustrating another method for extracting video covers provided in this application embodiment, which includes at least the following steps:

[0141] 701. Determine the target video uploaded by the user.

[0142] In this embodiment, the target video uploaded by the user can be a short video or a video of different lengths, such as a movie.

[0143] 702. Determine the display interface based on the popularity information corresponding to the target video.

[0144] In this embodiment, since different videos have different audiences, the number of users following different videos is different (i.e., the popularity is different). For videos with high popularity, multiple interface modules can be used to display them, thereby increasing user click behavior.

[0145] 703. Input the target video into the first network model and the second network model for evaluation to obtain the cover sequence.

[0146] In this embodiment, the process of evaluating the first network model and the second network model for the target video input is described in [reference needed]. Figure 3 Steps 302-305 of the illustrated embodiment will not be described in detail here.

[0147] 704. Based on the interface module corresponding to the display interface, extract the corresponding cover image from the cover sequence for interface display.

[0148] In this embodiment, the cover image can be displayed as one or multiple images. For example, the top three cover images can be displayed sequentially in a scrolling manner. Alternatively, it can be a dynamic image derived from a single image, such as a GIF representing the climax of a video obtained by identifying highlights from adjacent frames of the cover image.

[0149] Additionally, the interface module can be referenced. Figure 8 The scene shown, Figure 8 This is a schematic diagram of a video cover extraction method provided in an embodiment of this application. The diagram shows that module A1 can display the GIF representing the climax of the video by recognizing the highlights obtained from the adjacent frames of the cover image, or it can scroll the cover image sequentially and play it in response to the user's playback operation, thereby increasing the video's attractiveness to the user and increasing the video's click-through rate.

[0150] To better implement the above-described solutions of the embodiments of this application, related apparatus for implementing the above solutions is also provided below. Please refer to... Figure 9 , Figure 9 This is a schematic diagram of a video cover extraction device provided in an embodiment of this application. The video cover extraction device 900 includes:

[0151] Extraction unit 901 is used to extract a group of images to be evaluated from the video to be processed, the group of images to be evaluated including at least one frame of image to be evaluated.

[0152] Evaluation unit 902 is used to perform a first evaluation on the image to be evaluated based on a first network model to obtain a first evaluation score. The first network model includes an image evaluation branch and a semantic evaluation branch. The image evaluation branch is used to extract the image feature vector of the image to be evaluated, and the semantic evaluation branch is used to extract the semantic feature vector in the image to be evaluated. The image evaluation branch and the semantic evaluation branch share an encoder. The encoder is used to fuse the image feature vector and the semantic feature vector to obtain a fused feature vector. The image evaluation branch performs an image dimension quality evaluation on the image to be evaluated based on the fused feature vector. The semantic evaluation branch performs a semantic dimension quality evaluation on the correlation between the semantic information in the video to be processed and the image to be evaluated based on the fused feature vector. The first evaluation score is obtained by combining the evaluation results of the image evaluation branch and the semantic evaluation branch.

[0153] The evaluation unit 902 is further configured to perform a second evaluation on the image to be evaluated based on a second network model to obtain a second evaluation score. The second network model is used to perform a quality evaluation of the image to be evaluated in the aesthetic dimension.

[0154] The fusion unit 903 is used to fuse the first evaluation score and the second evaluation score to obtain the target evaluation score;

[0155] The extraction unit 901 is further configured to extract a target image from the image group to be evaluated based on the target evaluation score, wherein the target image is the video cover image of the video to be processed.

[0156] Optionally, in some possible implementations of this application, the extraction unit 901 is specifically used to obtain low-quality image features corresponding to the video to be processed;

[0157] The extraction unit 901 is specifically used to filter out images that hit the low-quality image features in the image group to be evaluated, so as to update the image group to be evaluated into the first evaluation image group.

[0158] The extraction unit 901 is specifically used to sort the image group to be evaluated in the first evaluation image group based on the target evaluation score to obtain an evaluation sequence;

[0159] The extraction unit 901 is specifically used to extract the target image from the first evaluation image group according to the image order in the evaluation sequence.

[0160] Optionally, in some possible implementations of this application, the extraction unit 901 is specifically used to extract key image elements corresponding to the first evaluation image group based on preset rules;

[0161] The extraction unit 901 is specifically used to advance the image order of the key image elements in the evaluation sequence that meet the preset requirements, so as to update the sequence order of the images in the evaluation sequence.

[0162] Optionally, in some possible implementations of this application, the extraction unit 901 is specifically used to determine target description information based on the semantic information in the video to be processed;

[0163] The extraction unit 901 is specifically used to determine the matching information between the target description information and the image features corresponding to the key image elements;

[0164] The extraction unit 901 is specifically used to advance the corresponding image if the matching information meets the preset requirements, so as to update the sequence order of the images in the evaluation sequence.

[0165] Optionally, in some possible implementations of this application, the extraction unit 901 is specifically used to perform text recognition on the images in the evaluation sequence to obtain text information;

[0166] The extraction unit 901 is specifically used to update the sequence order of images in the evaluation sequence based on the degree of matching between the text information and the target description information.

[0167] Optionally, in some possible implementations of this application, the evaluation unit 902 is specifically used to obtain the initial cover corresponding to the video to be processed, wherein the initial cover is the cover marked in the video to be processed;

[0168] The evaluation unit 902 is specifically used to input the initial cover into the first network model for evaluation to obtain a third evaluation score;

[0169] The evaluation unit 902 is specifically used to input the initial cover into the second network model for evaluation to obtain a fourth evaluation score;

[0170] The evaluation unit 902 is specifically used to fuse the third evaluation score and the fourth evaluation score to obtain an initial evaluation score;

[0171] The evaluation unit 902 is specifically used to compare the initial evaluation score with the target evaluation score to determine the video cover image of the video to be processed from the target image and the initial cover.

[0172] Optionally, in some possible implementations of this application, the evaluation unit 902 is specifically used to determine the setting image in the video to be processed in response to the setting operation of the target user, and to use the setting image as a positive sample;

[0173] The evaluation unit 902 is specifically used to select images from the video to be processed whose difference value with the set image reaches a difference threshold as negative samples.

[0174] The evaluation unit 902 is specifically used to extract samples from the positive samples and the negative samples according to the sample ratio to obtain training samples;

[0175] The evaluation unit 902 is specifically used to train the image evaluation branch and the semantic evaluation branch based on the training samples, so as to adjust the parameters of the first network model.

[0176] The evaluation process involves extracting a set of images to be evaluated from the video to be processed, each set including at least one frame of the image to be evaluated. A first evaluation is then performed on the images to be evaluated based on a first network model to obtain a first evaluation score. This first network model includes an image evaluation branch and a semantic evaluation branch. The image evaluation branch extracts image feature vectors from the images to be evaluated, and the semantic evaluation branch extracts semantic feature vectors from the images to be evaluated. Both branches share a common encoder, which fuses the image feature vectors and semantic feature vectors to obtain a fused feature vector. The image evaluation branch performs an image-dimensional quality evaluation on the images to be evaluated based on the fused feature vector, and the semantic evaluation branch performs a semantic-dimensional quality evaluation based on the correlation between the semantic information in the video to be processed and the images to be evaluated. The first evaluation score is obtained by combining the evaluation results from the image evaluation branch and the semantic evaluation branch. A second evaluation is then performed on the images to be evaluated based on a second network model to obtain a second evaluation score. This second network model performs an aesthetic-dimensional quality evaluation on the images to be evaluated. The first and second evaluation scores are then fused to obtain a target evaluation score. Finally, a target image, which is the video cover image of the video to be processed, is extracted from the set of images to be evaluated based on the target evaluation score. This enables the cover extraction process to achieve multimodal fusion evaluation. By combining the correlation between semantic modality and image modality during the multimodal information fusion process and using the same encoder to fuse semantic features and image features, the network model's ability to understand video semantics can be improved. The image can be evaluated from the dimensions of image, semantics, and aesthetics, thereby selecting an image that better fits the semantic meaning of the video as the cover image, thus improving the accuracy of video cover extraction.

[0177] This application also provides a terminal device, such as... Figure 10 The diagram shown is a structural schematic of another terminal device provided in an embodiment of this application. For ease of explanation, only the parts related to the embodiment of this application are shown. For specific technical details not disclosed, please refer to the method section of the embodiment of this application. The terminal can be any terminal device including mobile phones, tablets, personal digital assistants (PDAs), point-of-sale (POS) terminals, in-vehicle computers, etc. Taking a mobile phone as an example:

[0178] Figure 10 This is a block diagram illustrating a portion of the structure of a mobile phone related to the terminal provided in the embodiments of this application. (Reference) Figure 10The mobile phone includes components such as a radio frequency (RF) circuit 1010, a memory 1020, an input unit 1030, a display unit 1040, a sensor 1050, an audio circuit 1060, a wireless fidelity (WiFi) module 1070, a processor 1080, and a power supply 1090. Those skilled in the art will understand that... Figure 10 The mobile phone structure shown does not constitute a limitation on the mobile phone and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0179] The following is combined with Figure 10 A detailed introduction to each component of a mobile phone:

[0180] The RF circuit 1010 can be used for receiving and transmitting signals during information transmission or calls. Specifically, it receives downlink information from the base station and processes it with the processor 1080; additionally, it transmits uplink data to the base station. Typically, the RF circuit 1010 includes, but is not limited to, an antenna, at least one amplifier, a transceiver, a coupler, a low-noise amplifier (LNA), and a duplexer. Furthermore, the RF circuit 1010 can also communicate wirelessly with networks and other devices. The aforementioned wireless communication can use any communication standard or protocol, including but not limited to Global System for Mobile Communication (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Long Term Evolution (LTE), email, and Short Message Service (SMS).

[0181] The memory 1020 can be used to store software programs and modules. The processor 1080 executes various mobile phone functions and data processing by running the software programs and modules stored in the memory 1020. The memory 1020 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, applications required for at least one function (such as sound playback function, image playback function, etc.), etc.; the data storage area may store data created according to the use of the mobile phone (such as audio data, phonebook, etc.). In addition, the memory 1020 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device.

[0182] The input unit 1030 can be used to receive input numerical or character information, and generate key signal inputs related to user settings and function control of the mobile phone. Specifically, the input unit 1030 may include a touch panel 1031 and other input devices 1032. The touch panel 1031, also known as a touch screen, can collect touch operations performed by the user on or near it (such as operations performed by the user using a finger, stylus, or any suitable object or accessory on or near the touch panel 1031, as well as air touch operations within a certain range on the touch panel 1031), and drive the corresponding connection devices according to a pre-set program. Optionally, the touch panel 1031 may include two parts: a touch detection device and a touch controller. The touch detection device detects the user's touch position and the signal generated by the touch operation, and transmits the signal to the touch controller; the touch controller receives touch information from the touch detection device, converts it into touch point coordinates, sends it to the processor 1080, and can receive and execute commands sent by the processor 1080. Furthermore, the touch panel 1031 can be implemented using various types of sensors, including resistive, capacitive, infrared, and surface acoustic wave sensors. In addition to the touch panel 1031, the input unit 1030 may also include other input devices 1032. Specifically, these other input devices 1032 may include, but are not limited to, one or more of the following: a physical keyboard, function keys (such as volume control buttons, power buttons, etc.), a trackball, a mouse, and a joystick.

[0183] The display unit 1040 can be used to display information input by the user or information provided to the user, as well as various menus of the mobile phone. The display unit 1040 may include a display panel 1041, which may optionally be configured as a liquid crystal display (LCD), organic light-emitting diode (OLED), or similar form. Further, a touch panel 1031 may cover the display panel 1041. When the touch panel 1031 detects a touch operation on or near it, it transmits the information to the processor 1080 to determine the type of touch event. Subsequently, the processor 1080 provides corresponding visual output on the display panel 1041 according to the type of touch event. Although in Figure 10 In this embodiment, the touch panel 1031 and the display panel 1041 are two separate components to realize the input and output functions of the mobile phone. However, in some embodiments, the touch panel 1031 and the display panel 1041 can be integrated to realize the input and output functions of the mobile phone.

[0184] The mobile phone may also include at least one sensor 1050, such as a light sensor, a motion sensor, and other sensors. Specifically, the light sensor may include an ambient light sensor and a proximity sensor. The ambient light sensor can adjust the brightness of the display panel 1041 according to the ambient light level, and the proximity sensor can turn off the display panel 1041 and / or backlight when the phone is moved to the ear. As a type of motion sensor, an accelerometer sensor can detect the magnitude of acceleration in various directions (generally three axes). When stationary, it can detect the magnitude and direction of gravity and can be used for applications that recognize the phone's posture (such as landscape / portrait switching, related games, magnetometer posture calibration), vibration recognition-related functions (such as pedometer, taps), etc. Other sensors that may be configured in the mobile phone, such as gyroscopes, barometers, hygrometers, thermometers, and infrared sensors, will not be described in detail here.

[0185] The audio circuit 1060, speaker 1061, and microphone 1062 provide an audio interface between the user and the mobile phone. The audio circuit 1060 converts the received audio data into electrical signals and transmits them to the speaker 1061, where the speaker 1061 converts them into sound signals for output. On the other hand, the microphone 1062 converts the collected sound signals into electrical signals, which are then received by the audio circuit 1060, converted into audio data, and then processed by the processor 1080 before being transmitted via the RF circuit 1010 to, for example, another mobile phone, or the audio data can be output to the memory 1020 for further processing.

[0186] WiFi is a short-range wireless transmission technology. Through the WiFi module 1070, mobile phones can help users send and receive emails, browse web pages, and access streaming media, providing users with wireless broadband internet access. Although Figure 10 The WiFi module 1070 is shown, but it is understood that it is not an essential component of a mobile phone and can be omitted as needed without changing the essence of the invention.

[0187] The processor 1080 is the control center of the mobile phone, connecting various parts of the phone through various interfaces and lines. It executes various functions and processes data by running or executing software programs and / or modules stored in the memory 1020 and calling data stored in the memory 1020. Optionally, the processor 1080 may include one or more processing units; optionally, the processor 1080 may integrate an application processor and a modem processor, wherein the application processor mainly handles the operating system, user interface, and applications, and the modem processor mainly handles wireless communication. It is understood that the aforementioned modem processor may also not be integrated into the processor 1080.

[0188] The mobile phone also includes a power supply 1090 (such as a battery) that supplies power to various components. Optionally, the power supply can be logically connected to the processor 1080 through a power management system, thereby enabling functions such as charging, discharging, and power consumption management through the power management system.

[0189] Although not shown, mobile phones may also include a camera, Bluetooth module, etc., which will not be described in detail here.

[0190] In this embodiment of the application, the processor 1080 included in the terminal also has the function of performing the various steps of the page processing method described above.

[0191] This application also provides a server; please refer to [link / reference]. Figure 11 , Figure 11This is a schematic diagram of a server structure provided in an embodiment of this application. The server 1100 can vary significantly due to different configurations or performance. It may include one or more central processing units (CPUs) 1122 (e.g., one or more processors) and memory 1132, and one or more storage media 1130 (e.g., one or more mass storage devices) for storing application programs 1142 or data 1144. The memory 1132 and storage media 1130 can be temporary or persistent storage. The program stored in the storage media 1130 may include one or more modules (not shown in the diagram), each module including a series of instruction operations on the server. Furthermore, the CPU 1122 may be configured to communicate with the storage media 1130 and execute the series of instruction operations in the storage media 1130 on the server 1100.

[0192] Server 1100 may also include one or more power supplies 1126, one or more wired or wireless network interfaces 1150, one or more input / output interfaces 1158, and / or one or more operating systems 1141, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, etc.

[0193] The steps performed by the management device in the above embodiments can be based on this Figure 11 The server structure shown.

[0194] This application also provides a computer-readable storage medium storing instructions for extracting a video cover image. When executed on a computer, these instructions cause the computer to perform the aforementioned actions. Figures 3 to 8 The steps performed by the video cover extraction device in the method described in the illustrated embodiment.

[0195] This application also provides a computer program product that includes instructions for extracting video covers. When run on a computer, it causes the computer to perform the aforementioned operations. Figures 3 to 8 The steps performed by the video cover extraction device in the method described in the illustrated embodiment.

[0196] This application embodiment also provides a video cover extraction system, which may include... Figure 9 The video cover extraction device described in the embodiments, or Figure 10 The terminal device in the described embodiments, or Figure 11 The server described.

[0197] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0198] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection between apparatuses or units through some interfaces, and may be electrical, mechanical, or other forms.

[0199] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0200] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0201] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, a video cover extraction device, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0202] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A method for extracting video cover images, characterized in that, include: Extract a set of images to be evaluated from the video to be processed, the set of images to be evaluated including at least one frame of image to be evaluated; A first evaluation is performed on the image to be evaluated based on a first network model to obtain a first evaluation score. The first network model includes an image evaluation branch and a semantic evaluation branch. The image evaluation branch is used to extract the image feature vector of the image to be evaluated, and the semantic evaluation branch is used to extract the semantic feature vector of the image to be evaluated. The image evaluation branch and the semantic evaluation branch share an encoder. The encoder is used to fuse the image feature vector and the semantic feature vector to obtain a fused feature vector. The image evaluation branch performs an image dimension quality evaluation on the image to be evaluated based on the fused feature vector. The semantic evaluation branch performs a semantic dimension quality evaluation on the correlation between the semantic information in the video to be processed and the image to be evaluated based on the fused feature vector. The first evaluation score is obtained by combining the evaluation results of the image evaluation branch and the semantic evaluation branch. The image to be evaluated is evaluated based on the second network model to obtain a second evaluation score. The second network model is used to evaluate the quality of the image to be evaluated in the aesthetic dimension. The target evaluation score is obtained by fusing the first evaluation score and the second evaluation score. Based on the target evaluation score, a target image is extracted from the group of images to be evaluated as the video cover image of the video to be processed. The video cover image is the target image or a dynamic image containing the target image. Obtain the initial cover corresponding to the video to be processed, wherein the initial cover is the cover marked in the video to be processed; The initial cover is input into the first network model for evaluation to obtain a third evaluation score; The initial cover is input into the second network model for evaluation to obtain a fourth evaluation score; The third evaluation score and the fourth evaluation score are fused to obtain the initial evaluation score; The initial evaluation score is compared with the target evaluation score to determine the video cover image of the video to be processed from the target image and the initial cover.

2. The method according to claim 1, characterized in that, Extracting the target image from the image group to be evaluated based on the target evaluation score includes: Obtain the low-quality image features corresponding to the video to be processed; Images that match the low-quality image features in the image group to be evaluated are filtered out, so that the image group to be evaluated is updated to the first evaluation image group; Based on the target evaluation score, the images to be evaluated in the first evaluation image group are sorted to obtain an evaluation sequence; The target image is extracted from the first evaluation image group according to the image order in the evaluation sequence.

3. The method according to claim 2, characterized in that, After sorting the image group to be evaluated in the first evaluation image group based on the target evaluation score to obtain the evaluation sequence, the method further includes: Key image elements corresponding to the first evaluation image group are extracted based on preset rules; The order of images whose image features corresponding to the key image elements in the evaluation sequence meet the preset requirements is advanced to update the sequence order of images in the evaluation sequence.

4. The method according to claim 3, characterized in that, The step of advancing the order of images whose image features corresponding to the key image elements in the evaluation sequence meet preset requirements, in order to update the sequence order of images in the evaluation sequence, includes: Determine target description information based on semantic information in the video to be processed; Determine the matching information between the target description information and the image features corresponding to the key image elements; If the matching information meets the preset requirements, the corresponding image is moved forward to update the sequence order of the images in the evaluation sequence.

5. The method according to claim 4, characterized in that, The method further includes: Text recognition is performed on the images in the evaluation sequence to obtain text information; The sequence order of images in the evaluation sequence is updated based on the degree of matching between the text information and the target description information.

6. The method according to any one of claims 1-4, characterized in that, The method further includes: In response to a setting operation by a target user, a setting image in the video to be processed is determined, and the setting image is used as a positive sample. Images whose difference value from the set image reaches a difference threshold are selected from the video to be processed as negative samples; Training samples are extracted from the positive and negative samples according to the sample ratio. The image evaluation branch and the semantic evaluation branch are trained based on the training samples to adjust the parameters of the first network model.

7. A device for extracting video cover images, characterized in that, include: An extraction unit is used to extract a group of images to be evaluated from the video to be processed, the group of images to be evaluated including at least one frame of image to be evaluated. An evaluation unit is used to perform a first evaluation on the image to be evaluated based on a first network model to obtain a first evaluation score. The first network model includes an image evaluation branch and a semantic evaluation branch. The image evaluation branch is used to extract the image feature vector of the image to be evaluated, and the semantic evaluation branch is used to extract the semantic feature vector of the image to be evaluated. The image evaluation branch and the semantic evaluation branch share an encoder. The encoder is used to fuse the image feature vector and the semantic feature vector to obtain a fused feature vector. The image evaluation branch performs an image dimension quality evaluation on the image to be evaluated based on the fused feature vector. The semantic evaluation branch performs a semantic dimension quality evaluation on the correlation between the semantic information in the video to be processed and the image to be evaluated based on the fused feature vector. The first evaluation score is obtained by combining the evaluation results of the image evaluation branch and the semantic evaluation branch. The evaluation unit is further configured to perform a second evaluation on the image to be evaluated based on a second network model to obtain a second evaluation score. The second network model is used to perform a quality evaluation of the image to be evaluated in the aesthetic dimension. A fusion unit is used to fuse the first evaluation score and the second evaluation score to obtain a target evaluation score; The extraction unit is further configured to extract a target image from the image group to be evaluated based on the target evaluation score as the video cover image of the video to be processed, wherein the video cover image is the target image or a dynamic image containing the target image; The evaluation unit is specifically used to obtain the initial cover corresponding to the video to be processed, wherein the initial cover is the cover marked in the video to be processed; The evaluation unit is specifically used to input the initial cover into the first network model for evaluation to obtain a third evaluation score; The evaluation unit is specifically used to input the initial cover into the second network model for evaluation to obtain a fourth evaluation score; The evaluation unit is specifically used to fuse the third evaluation score and the fourth evaluation score to obtain an initial evaluation score; The evaluation unit is specifically used to compare the initial evaluation score with the target evaluation score to determine the video cover image of the video to be processed from the target image and the initial cover.

8. The apparatus according to claim 7, characterized in that, The extraction unit is specifically used to obtain the low-quality image features corresponding to the video to be processed. The extraction unit is specifically used to filter out images that hit the low-quality image features in the image group to be evaluated, so as to update the image group to be evaluated into the first evaluation image group. The extraction unit is specifically used to sort the image group to be evaluated in the first evaluation image group based on the target evaluation score to obtain an evaluation sequence; The extraction unit is specifically used to extract the target image from the first evaluation image group according to the image order in the evaluation sequence.

9. The apparatus according to claim 8, characterized in that, The extraction unit is specifically used to extract key image elements corresponding to the first evaluation image group based on preset rules. The extraction unit is specifically used to advance the image order of the key image elements in the evaluation sequence that meet the preset requirements, so as to update the sequence order of the images in the evaluation sequence.

10. The apparatus according to claim 9, characterized in that, The extraction unit is specifically used to determine target description information based on the semantic information in the video to be processed; The extraction unit is specifically used to determine the matching information between the target description information and the image features corresponding to the key image elements; The extraction unit is specifically used to advance the corresponding image if the matching information meets the preset requirements, so as to update the sequence order of the images in the evaluation sequence.

11. The apparatus according to claim 10, characterized in that, The extraction unit is specifically used to perform text recognition on the images in the evaluation sequence to obtain text information; The extraction unit is specifically used to update the sequence order of images in the evaluation sequence based on the degree of matching between the text information and the target description information.

12. The apparatus according to any one of claims 7-11, characterized in that, The evaluation unit is specifically used to determine the setting image in the video to be processed in response to the setting operation of the target user, and to use the setting image as a positive sample. The evaluation unit is specifically used to select images from the video to be processed whose difference value with the set image reaches a difference threshold as negative samples. The evaluation unit is specifically used to extract samples from the positive and negative samples according to the sample ratio to obtain training samples; The evaluation unit is specifically used to train the image evaluation branch and the semantic evaluation branch based on the training samples, so as to adjust the parameters of the first network model.

13. A computer device, characterized in that, The computer device includes a processor and memory: The memory is used to store program code; the processor is used to execute the video cover extraction method according to any one of claims 1 to 6 according to the instructions in the program code.

14. A computer program product comprising a computer program / instructions, characterized in that, When the computer program / instructions are executed by the processor, they implement the steps of the video cover extraction method according to any one of claims 1 to 6.

Images