Video cover extraction method, device and equipment and computer readable storage medium

Abstract

The application provides a video cover extraction method, device and equipment and a computer readable storage medium, and relates to the field of artificial intelligence. The method comprises the following steps: performing image feature extraction on a video frame sequence of a to-be-processed video to obtain a visual feature vector corresponding to the video frame sequence; performing word vector feature extraction on video text information of the to-be-processed video to obtain a text feature vector corresponding to the video text information; performing feature fusion on the visual feature vector and the text feature vector to obtain a multi-modal fusion feature corresponding to the to-be-processed video; and selecting a video cover frame of the to-be-processed video from the video frame sequence based on the multi-modal fusion feature. Through the application, the accuracy of video cover extraction can be improved.

Description

Technical Field

[0001] This application relates to artificial intelligence technology, and more particularly to a method, apparatus, device, and computer-readable storage medium for extracting video cover images. Background Technology

[0002] Currently, most techniques for selecting video cover images involve extracting image modal information, such as visual features, from video frames and then selecting the cover image based on this extracted visual feature information. This method of selecting video cover images using image modal information utilizes a relatively limited range of information dimensions, resulting in cover images that do not accurately represent the information within the video content and thus have low accuracy. Summary of the Invention

[0003] This application provides a video cover extraction method, apparatus, and computer-readable storage medium, which can improve the accuracy of video cover extraction.

[0004] The technical solution of this application embodiment is implemented as follows: This application provides a method for extracting video cover images, including: Image feature extraction is performed on the video frame sequence of the video to be processed to obtain the visual feature vector corresponding to the video frame sequence; Word vector features are extracted from the video text information of the video to be processed to obtain the text feature vector corresponding to the video text information; The visual feature vector and the text feature vector are fused to obtain the multimodal fusion feature corresponding to the video to be processed; Based on the multimodal fusion features, the video cover frame of the video to be processed is selected from the video frame sequence.

[0005] This application provides a video cover extraction device, including: The visual modality feature extraction module is used to extract image features from the video frame sequence of the video to be processed, and obtain the visual feature vector corresponding to the video frame sequence. The text modality feature extraction module is used to extract word vector features from the video text information of the video to be processed, and obtain the text feature vector corresponding to the video text information. The multimodal feature fusion module is used to fuse the visual feature vector with the text feature vector to obtain the multimodal fusion features corresponding to the video to be processed. The cover selection module is used to select the video cover frame of the video to be processed from the video frame sequence based on the multimodal fusion features.

[0006] In the above-described device, the visual modality feature extraction module is further configured to extract image features from each video frame in the video frame sequence to obtain image features corresponding to each video frame; perform feature fusion on the image features corresponding to each video frame to obtain initial visual features corresponding to the video frame sequence; and map the initial visual features to a preset feature dimension to obtain the visual feature vector.

[0007] In the above-described apparatus, the text modality feature extraction module is further configured to perform word segmentation and text vector conversion on the video text information to obtain a word vector sequence corresponding to the video text information; extract text features from each word vector in the word vector sequence to obtain text features corresponding to each word vector; perform feature fusion on the text features corresponding to each word vector to obtain initial text features corresponding to the video text information; and map the initial text features to a preset feature dimension to obtain the text feature vector.

[0008] In the above device, the cover selection module is further configured to use a fully connected layer to perform scoring prediction on the multimodal fusion features to obtain a prediction score corresponding to each video frame in the video frame sequence; normalize the prediction score corresponding to each video frame to obtain the prediction probability that each video frame is the video cover frame; the prediction probability represents the ratio of the prediction score corresponding to each video frame to the total prediction score corresponding to the video frame sequence; and select the video cover frame from the video frame sequence according to the prediction probability.

[0009] In the above device, the cover selection module is further configured to perform a nonlinear transformation on the multimodal fusion features using a preset activation function of the fully connected layer, and by utilizing a preset weight matrix and a preset bias constant in the preset activation function, to obtain the prediction score corresponding to each video frame; the preset weight matrix and the preset bias constant are determined through the multimodal training process of the fully connected layer.

[0010] In the aforementioned apparatus, the video cover extraction device further includes a training module. This training module is used to obtain a video sample set and an initial fully connected layer before performing a nonlinear transformation on the multimodal fusion features using a preset activation function of the fully connected layer and a preset weight matrix and a preset bias constant in the preset activation function to obtain the prediction score corresponding to each video frame. The network parameters of the initial fully connected layer include an initial weight matrix and an initial bias constant. The video sample set includes multiple sample videos. Each of the multiple sample videos contains an labeled video cover frame, sample video text information, and a sample video sequence. During each training process, the initial fully connected layer performs multimodal video cover prediction on each sample video based on the sample video text information and the sample video sequence to obtain the predicted video cover frame for each training session. The training loss for each training session is obtained based on the predicted video cover frame and the labeled video cover frame. The initial weight matrix and the initial bias constant are iteratively updated based on the training loss. When the training loss reaches a preset convergence condition, the training ends, resulting in the fully connected layer containing the preset weight matrix and the preset bias constant.

[0011] In the above-described device, the text modality feature extraction module is further configured to perform word segmentation on the video text information to obtain a word segmentation sequence; and to perform text vector transformation on the word segmentation sequence using a pre-trained word vector model to obtain the word vector sequence; the pre-trained word vector model is used to transform text information into vector representation.

[0012] In the aforementioned device, the video cover extraction device further includes a data acquisition module. This acquisition module is used to acquire the original video frame sequence corresponding to the video to be processed, as the video frame sequence, before performing image feature extraction on the video frame sequence of the video to be processed to obtain the visual feature vector corresponding to the video frame sequence; or, to sample the original video frame sequence according to a preset sampling rate to obtain the video frame sequence; or, to perform keyframe prediction on the original video frame sequence using a preset statistical model, and select multiple keyframes from the original video frame sequence based on the prediction results to obtain the video frame sequence; the statistical model is used to predict whether the original video frame is a keyframe based on the scene images contained in the original video frame.

[0013] In the above-described device, the cover selection module is further configured to use the video frames whose predicted probabilities rank in the top N positions as the video cover frames; where N is a positive integer greater than or equal to 1.

[0014] In the above-mentioned device, the video cover extraction device further includes a display module. The display module is used to select the video cover frame of the video to be processed from the video frame sequence based on the multimodal fusion features, and then display the video cover frame at a preset display position corresponding to the video to be processed on a preset video platform, thereby displaying and recommending the video to be processed.

[0015] In the aforementioned device, the display module is further configured to display and recommend the video to be processed in a carousel manner at the preset display position when the number of video cover frames is greater than 1.

[0016] In the aforementioned device, the video text information includes at least one of the following: video title, video description, and video tags.

[0017] This application provides an electronic device, including: Memory, used to store executable instructions; The processor, when executing executable instructions stored in the memory, implements the video cover extraction method provided in the embodiments of this application.

[0018] This application provides a computer-readable storage medium storing executable instructions, which, when executed by a processor, implement the video cover extraction method provided in this application.

[0019] The embodiments of this application have the following beneficial effects: This application introduces the concept of multimodal fusion into the video cover selection task. It simultaneously extracts visual feature vectors from the video frame modality and text feature vectors from the video text information modality, and then fuses these two feature vectors to obtain multimodal fusion features. These multimodal fusion features contain both image and text information from the video, encompassing a richer amount of video content information, which is more conducive to selecting video covers and thus improves the accuracy of video cover extraction. Attached Figure Description

[0020] Figure 1 This is an optional structural diagram of the network video platform architecture provided in the embodiments of this application; Figure 2 This is an optional structural diagram of the video cover extraction device provided in the embodiments of this application; Figure 3 This is an optional flowchart illustrating the video cover extraction method provided in this application embodiment; Figure 4 This is a schematic diagram of a data format of the video to be processed provided in an embodiment of this application; Figure 5This is an optional flowchart illustrating the video cover extraction method provided in this application embodiment; Figure 6 This is an optional flowchart illustrating the video cover extraction method provided in this application embodiment; Figure 7 This is an optional flowchart illustrating the video cover extraction method provided in this application embodiment; Figure 8 This is an optional flowchart illustrating the fully connected layer network structure provided in an embodiment of this application; Figure 9 This is an optional flowchart illustrating how a video cover extraction model is used to extract video cover frames in a real-world application scenario, as provided in this application embodiment. Detailed Implementation

[0021] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0022] In the following description, references are made to “some embodiments,” which describe a subset of all possible embodiments. However, it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0023] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0024] In the implementation of this application, the collection and processing of relevant data should strictly comply with the requirements of relevant laws and regulations, obtain the informed consent or separate consent of the personal information subject, and carry out subsequent data use and processing within the scope of laws and regulations and the authorization of the personal information subject.

[0025] Before providing a further detailed description of the embodiments of this application, the nouns and terms involved in the embodiments of this application will be explained, and the nouns and terms involved in the embodiments of this application shall be interpreted as follows.

[0026] 1) Artificial Intelligence (AI) is the theory, methods, technology, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that attempts to understand the essence of intelligence and produce a new kind of intelligent machine that can react in a way similar to human intelligence. AI studies the design principles and implementation methods of various intelligent machines, enabling them to possess the functions of perception, reasoning, and decision-making.

[0027] Artificial intelligence (AI) is a comprehensive discipline encompassing a wide range of fields, including both hardware and software technologies. Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies primarily include computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0028] 2) Machine Learning (ML) is a multidisciplinary field involving probability theory, statistics, approximation theory, convex analysis, and algorithm complexity theory. It specifically studies how computers can simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to endow computers with intelligence; its applications span all areas of artificial intelligence. Machine learning and deep learning typically include techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and instructional learning.

[0029] 3) Modality: Every source or form of information can be called a modality. For example, humans have touch, hearing, vision, and smell; information media include speech, video, and text; and there are various types of sensors, such as radar, infrared, and accelerometers. Each of these can be called a modality. Furthermore, modality can have a very broad definition; for example, two different languages ​​can be considered two modalities, and even datasets collected under two different circumstances can be considered two modalities.

[0030] 4) Multimodal fusion is responsible for combining information from multiple modalities to predict the target (classification or regression).

[0031] 5) Multimodal Machine Learning (MMML) aims to enable the processing and understanding of multimodal information through machine learning methods.

[0032] 6) VF: Abbreviation for VideoFrames.

[0033] With the research and advancement of artificial intelligence (AI) technology, AI is being studied and applied in various fields, such as smart homes, smart wearable devices, virtual assistants, smart speakers, smart marketing, autonomous driving, drones, robots, smart healthcare, and smart customer service. It is believed that with the development of technology, AI will be applied in more fields and play an increasingly important role.

[0034] The solutions provided in this application relate to video processing technology using artificial intelligence, and are specifically illustrated through the following embodiments: This application provides a video cover extraction method, apparatus, device, and computer-readable storage medium, which can improve the accuracy of video cover extraction. The following describes exemplary applications of the electronic devices provided in this application. These electronic devices can be implemented as various types of user terminals such as laptops, tablets, desktop computers, set-top boxes, and mobile devices (e.g., mobile phones, portable music players, personal digital assistants, dedicated messaging devices, portable gaming devices), or as servers. The following will describe exemplary applications when the electronic device is implemented as a server.

[0035] See Figure 1 , Figure 1 This is an optional architecture diagram of the network video platform 100 provided in the embodiments of this application. The terminal 400 (terminal 400-1 and terminal 400-2 are shown as examples) connects to the server 200 through the network 300. The network 300 can be a wide area network or a local area network, or a combination of the two.

[0036] Terminal 400-1 belongs to the publisher of online video and is used to upload the video to be processed, which contains video text information, to the backend server of the online video platform, namely server 200, so that the video to be processed can be published to other terminals on network 300, such as terminal 400-2, through server 200.

[0037] Server 200 is the backend server of the online video platform, belonging to the platform's administrator. It receives videos uploaded by terminal 400-1 and stores them in database 500. It performs image feature extraction on the video frame sequence of the video to be processed, obtaining visual feature vectors corresponding to the video frame sequence; it also performs word vector feature extraction on the video text information of the video to be processed, obtaining text feature vectors corresponding to the video text information; it fuses the visual feature vectors and text feature vectors to obtain multimodal fusion features corresponding to the video to be processed; based on the multimodal fusion features, it selects the video cover frame of the video to be processed from the video frame sequence. Then, server 200 can use the video cover frame to display and recommend the video to be processed on the online video platform.

[0038] Terminal 400-2 belongs to the user of the online video platform and is used to access the online video platform through the client application 410 of the online video platform. The client application 410 displays the video cover frame of the video to be processed and performs operations such as clicking and favorite on the video cover frame through the client application 410. Accordingly, the client application 410 plays or favorites the video to be processed.

[0039] In some embodiments, server 200 may be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. Terminal 400 may be a smartphone, tablet, laptop, desktop computer, smart speaker, smartwatch, etc., but is not limited to these. The terminal and server can be directly or indirectly connected via wired or wireless communication, which is not limited in this embodiment of the invention.

[0040] See Figure 2 , Figure 2 This is a schematic diagram of the structure of the server 200 provided in the embodiments of this application. Figure 2 The server 200 shown includes at least one processor 210, memory 250, at least one network interface 220, and a user interface 230. The various components in server 200 are coupled together via a bus system 240. It is understood that the bus system 240 is used to implement communication between these components. In addition to a data bus, the bus system 240 also includes a power bus, a control bus, and a status signal bus. However, for clarity, ... Figure 2 The general labeled all buses as Bus System 240.

[0041] Processor 210 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor can be a microprocessor or any conventional processor, etc.

[0042] User interface 230 includes one or more output devices 231 that enable the presentation of media content, including one or more speakers and / or one or more visual displays. User interface 230 also includes one or more input devices 232, including user interface components that facilitate user input, such as a keyboard, mouse, microphone, touch screen display, camera, other input buttons and controls.

[0043] The memory 250 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state storage, hard disk drives, optical disk drives, etc. The memory 250 may optionally include one or more storage devices physically located away from the processor 210.

[0044] The memory 250 may include volatile memory or non-volatile memory, or both. The non-volatile memory may be read-only memory (ROM), and the volatile memory may be random access memory (RAM). The memory 250 described in this application embodiment is intended to include any suitable type of memory.

[0045] In some embodiments, memory 250 is capable of storing data to support various operations, examples of which include programs, modules, and data structures or subsets or supersets thereof, as illustrated below.

[0046] Operating system 251 includes system programs for handling various basic system services and performing hardware-related tasks, such as the framework layer, core library layer, driver layer, etc., for implementing various basic business functions and handling hardware-based tasks; The network communication module 252 is used to reach other computing devices via one or more (wired or wireless) network interfaces 220, such as Bluetooth, WiFi, and Universal Serial Bus (USB). Presentation module 253 is configured to enable the presentation of information (e.g., a user interface for operating peripheral devices and displaying content and information) via one or more output devices 231 associated with user interface 230 (e.g., a display screen, a speaker, etc.). The input processing module 254 is used to detect and translate one or more user inputs or interactions from one or more input devices 232.

[0047] In some embodiments, the apparatus provided in this application can be implemented in software. Figure 2 A video cover extraction device 255 stored in memory 250 is shown. It can be software in the form of programs and plug-ins, including the following software modules: visual modality feature extraction module 2551, text modality feature extraction module 2552, multimodal feature fusion module 2553, and cover selection module 2554. These modules are logically related and can therefore be arbitrarily combined or further split according to the functions they implement.

[0048] The functions of each module will be explained below.

[0049] In other embodiments, the apparatus provided in this application can be implemented in hardware. As an example, the apparatus provided in this application can be a processor in the form of a hardware decoding processor, which is programmed to execute the video cover extraction method provided in this application. For example, the processor in the form of a hardware decoding processor can be one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), or other electronic components.

[0050] The video cover extraction method provided in this application will be described in conjunction with exemplary applications and implementations of the server provided in the embodiments of this application.

[0051] See Figure 3 , Figure 3 This is an optional flowchart illustrating the video cover extraction method provided in this application embodiment, which will be combined with... Figure 3 The steps shown are explained.

[0052] S101. Extract image features from the video frame sequence of the video to be processed to obtain the visual feature vector corresponding to the video frame sequence.

[0053] In this embodiment, for a video to be processed from which a video cover needs to be extracted, the video cover extraction device acquires a video frame sequence of the video to be processed, extracts image features from each video frame in the video frame sequence, and obtains a visual feature vector corresponding to the video frame sequence. The video frame sequence contains multiple video frames.

[0054] In this embodiment, the video cover extraction device can obtain the original video frame sequence corresponding to the video to be processed as the video frame sequence; it can also sample the original video frame sequence according to a preset sampling rate, and use the sampled original video frames as the video frame sequence; or it can use a preset statistical model, such as an object detection statistical model, to predict keyframes in the original video frame sequence, and select multiple keyframes from the original video frame sequence based on the prediction results to obtain the video frame sequence; here, the preset statistical model is used to perform network inference based on the scene images contained in the original video frames to predict whether the original video frames are keyframes. Alternatively, the video frame sequence can also be obtained manually. The specific selection depends on the actual situation, and this embodiment does not limit it.

[0055] In this embodiment, the video cover extraction device extracts image features of the visual modality for each video frame to obtain a visual feature vector corresponding to the video frame sequence. Here, the visual feature vector can characterize information such as texture, color, structure, shape, grayscale, brightness, and image correlation between each video frame contained in the video frame sequence.

[0056] S102. Extract word vector features from the video text information of the video to be processed to obtain the text feature vector corresponding to the video text information.

[0057] In this embodiment of the application, the video cover extraction device can extract word vector features from the video text information of the video to be processed, and obtain the text feature vector corresponding to the video text information.

[0058] In this embodiment, the video text information may include at least one of the following: video title, video description, and video tags. For example, when uploading a video to a video platform, the video publisher can fill in the video title, description, and keywords / tags on the upload page, and then upload the video. This text information is associated with the video title, video description, and video tags of the video being uploaded, i.e., the video text information of the video being processed.

[0059] In this embodiment of the application, the video text information contains descriptive information related to the video content. The video cover extraction device extracts word vector features of the text modality from the video text information of the video to be processed, and obtains the text feature vector corresponding to the video text information.

[0060] In some embodiments, the video text information is the video title, and the data format of the video to be processed, which includes the video frame sequence and the video title, can be as follows: Figure 4 As shown. Figure 4 This diagram illustrates the video title and video frame sequence data format for each of the multiple videos to be processed. As can be seen, the video title is text information related to the video content, and the video frame sequence contains multiple video frames in image format.

[0061] It should be noted that the execution order of S101 and S102 is not limited in the embodiments of this application. Image feature extraction and word vector feature extraction can be performed simultaneously on the video to be processed, or the image feature extraction and word vector feature extraction processes can be performed separately as needed, and the order of execution is not limited when they are executed separately.

[0062] S103. Perform feature fusion between the visual feature vector and the text feature vector to obtain the multimodal fusion features corresponding to the video to be processed.

[0063] In this embodiment, the video cover extraction device fuses the visual feature vector of the visual modality with the text feature vector of the text modality to obtain the multimodal fusion feature corresponding to the video to be processed.

[0064] In this embodiment, the feature fusion method can be feature splicing, similarity fusion, or linear fusion, etc. The specific method can be selected according to the actual situation, and this embodiment does not limit it.

[0065] Here, the multimodal fusion features simultaneously include feature information from both the image modality and the text modality of the video to be processed, thus providing a more comprehensive representation of the video's content.

[0066] S104. Based on multimodal fusion features, select the video cover frame of the video to be processed from the video frame sequence.

[0067] In this embodiment, the video cover extraction device can select the most suitable cover image as the video to be processed from multiple video frames in the video frame sequence based on the multimodal fusion features obtained above, combining the feature information of the image modality and the feature information of the text modality, and displaying and recommending the video cover frame of the video to be processed.

[0068] In this embodiment, the video cover extraction device can utilize a trained neural network model, based on multimodal fusion features, to perform feature analysis and network inference on each video frame in a video frame sequence, for each video frame, to obtain the probability that each video frame can be used as a video cover frame, and then select the video cover frame from them.

[0069] In some embodiments, the neural network model described above may be a classification neural network containing a fully-connected (FC) layer, or other types of network models. The specific model may be selected according to the actual situation, and this application embodiment does not limit it.

[0070] Understandably, this application introduces the concept of multimodal fusion into the video cover selection task. It simultaneously extracts the visual feature vectors from the video frame modality and the text feature vectors from the video text information modality, and then fuses these two feature vectors to obtain multimodal fusion features. These multimodal fusion features contain both image and text information from the video, encompassing a richer amount of video content information, which is more conducive to selecting video covers and thus improves the accuracy of video cover extraction.

[0071] In some embodiments, see Figure 5 , Figure 5 This is an optional flowchart illustrating the video cover extraction method provided in this application embodiment. Figure 3 The shown S101 can be implemented through S1011 to S1013, which will be explained in conjunction with each step.

[0072] S1011. Extract image features from each video frame in the video frame sequence to obtain the image features corresponding to each video frame.

[0073] In this embodiment of the application, the video cover extraction device can extract image features from each video frame in the video frame sequence using convolutional neural networks (CNNs) to obtain the image features corresponding to each video frame.

[0074] S1012. Perform feature fusion on the image features corresponding to each video frame to obtain the initial visual features corresponding to the video frame sequence.

[0075] In this embodiment of the application, the video cover extraction device performs feature fusion on the image features corresponding to each video frame. For example, feature fusion is performed by feature splicing to obtain the initial visual features corresponding to the video frame sequence.

[0076] S1013. Map the initial visual features to the preset feature dimensions to obtain the visual feature vector.

[0077] In this embodiment, since the features of the visual modality and the features of the text modality differ in dimensionality, in order to subsequently fuse them with the features of the text modality, the video cover extraction device can first map both to a feature space of the same dimension before fusion. The video cover extraction device maps the initial visual features to a feature space corresponding to a preset feature dimension to obtain a visual feature vector.

[0078] In some embodiments, the video cover extraction device can perform a non-linear dimensional transformation on the initial visual features through a fully connected layer in a neural network model, mapping them to a preset feature dimension.

[0079] It is understandable that by extracting the features of the image modalities of the video to be processed and fusing them with the mapping of preset feature dimensions, the video cover extraction device can obtain a visual feature vector that can be fused with the features of the text modalities, thereby improving the accuracy of video cover extraction through multimodal fusion.

[0080] In some embodiments, see Figure 6 , Figure 6 This is an optional flowchart illustrating the video cover extraction method provided in this application embodiment. Figure 3 The shown S102 can be implemented through S1021 to S1024, which will be explained in conjunction with each step.

[0081] S1021. Perform word segmentation and text vector conversion on the video text information to obtain the word vector sequence corresponding to the video text information.

[0082] In this embodiment, the video cover extraction device can segment the text content of the video text information to obtain a segmented sequence; here, the segmented sequence contains at least one word group from the video text information. The video cover extraction device can utilize a pre-trained word vector model, exemplarily Google's word2vec model, to convert the segmented sequence into text vectors, obtaining a word vector sequence. Here, the pre-trained word vector model is used to convert text information into vector representations.

[0083] In some embodiments, the video cover extraction device can also convert the word segmentation sequence into text vectors using other data conversion methods such as custom encoding to obtain a word vector sequence. The specific method chosen depends on the actual situation, and this application embodiment does not impose limitations.

[0084] In some embodiments, the video cover extraction device may use unary segmentation, that is, each character in the video text information is treated as a single word, or it may choose binary segmentation or multi-segmentation, etc., depending on the actual situation. This application embodiment does not limit the specific segmentation method.

[0085] S1022. Extract text features from each word vector in the word vector sequence to obtain the text features corresponding to each word vector.

[0086] In this embodiment, the word vector sequence contains a digitized vector representation of each word segment in the video text information, i.e., each word vector. The video cover extraction device can extract text features from each word vector to obtain the text features corresponding to each word vector.

[0087] In some embodiments, the video cover extraction device may use a bi-gated recurrent unit (Bi-GRU) to extract text features from each word vector to obtain the text features corresponding to each word vector; it may also use a long short-term memory (LSTM) network and a recurrent neural network (RNN) to extract text features. The specific choice is made according to the actual situation, and the embodiments of this application are not limited thereto.

[0088] S1023. Perform feature fusion on the text features corresponding to each word vector to obtain the initial text features corresponding to the video text information.

[0089] In this embodiment, the video cover extraction device performs feature fusion on the text features corresponding to each word vector to obtain the initial text features corresponding to the overall video text information.

[0090] S1024. Map the initial text features to a preset feature dimension to obtain the text feature vector.

[0091] In this embodiment of the application, the video cover extraction device maps the initial text features to the same preset feature dimension as the visual feature vector to obtain the text feature vector.

[0092] In some embodiments, the video cover extraction device can perform a non-linear dimensional transformation on the initial text features through a fully connected layer in a neural network model, mapping them to a preset feature dimension.

[0093] Understandably, video text information, such as video titles, contains a description of the text modality of the video to be processed. The extracted text feature vectors contain important information about the video to be processed. Combining text feature vectors with visual feature vectors can comprehensively express the content of the video to be processed, thereby improving the accuracy of video cover extraction.

[0094] In some embodiments, see Figure 7 , Figure 7 This is an optional flowchart illustrating the video cover extraction method provided in this application embodiment. Figure 3 , Figure 5 or Figure 6The shown S104 can be implemented through S1041 to S1043, which will be explained in conjunction with each step.

[0095] S1041. Use a fully connected layer to perform scoring prediction on the multimodal fusion features to obtain the predicted score for each video frame in the video frame sequence.

[0096] In this embodiment, the video cover extraction device can utilize the classification capability of fully connected layers to perform scoring prediction on multimodal features, thereby obtaining a predicted score for each video frame as a video cover frame.

[0097] Here, when the fully connected layer performs score prediction on the multimodal fusion features, it can utilize the visual modality features in the multimodal features to score based on visual evaluation indicators such as the clarity of the image or scene contained in the video frame, the richness of information, and the expressiveness of the main features of the image. At the same time, it can utilize the text modality features in the multimodal features to score the correlation between the text description and the image content. In this way, the scores from multiple modalities can be combined to obtain a comprehensive score, which serves as the prediction score for each video frame.

[0098] In some embodiments, the network structure of the fully connected layer can be as follows: Figure 8 As shown. The fully connected layer takes the feature components {D1,D2,…Dm} of the multimodal fusion features in each dimension as the input feature vector, where m can be the sum of the dimensions of the visual feature vector and the text feature vector. The video cover extraction device can use the preset activation function f of the fully connected layer, and the preset weight matrix and preset bias constant in the preset activation function to perform nonlinear transformation on the multimodal fusion features, and calculate the prediction score {S1,S2,…Sn} of each video frame as the video cover frame; where n is the number of video frames. The process is shown in formula (1) as follows.

[0099] (1) In formula (1), For the feature components in each dimension, For the preset weight matrix, As a preset bias constant, The calculated video frame is the predicted score for the video cover frame. and This can be determined through the multimodal training process of fully connected layers.

[0100] from Figure 8 As can be seen, the predicted score of each video frame is calculated based on the feature components of the multimodal fusion features across all dimensions. In other words, the predicted score of each video frame is based on the overall multimodal fusion features, thus ensuring the accuracy of the predicted score.

[0101] S1042. Normalize the prediction score corresponding to each video frame to obtain the prediction probability that each video frame is a video cover frame; the prediction probability represents the ratio of the prediction score corresponding to each video frame to the total prediction score corresponding to the video frame sequence.

[0102] In this embodiment of the application, the predicted score obtained in S1041 is usually a real number. The video cover extraction device can normalize the predicted score corresponding to each video frame to obtain a percentage value, which is used as the predicted probability that each video frame is a video cover frame. As shown in formula (2), it is as follows.

[0103] (2) In formula (2), For a video frame The corresponding predicted score, It is the sum of the prediction scores corresponding to all video frames in the video frame sequence. For video frames The corresponding prediction probability, i.e., video frame Corresponding predicted score Total prediction score corresponding to the video frame sequence The ratio of n to n. n is the number of video frames contained in the video frame sequence.

[0104] S1043. Select the video cover frame from the video frame sequence based on the predicted probability.

[0105] In this embodiment of the application, the video cover extraction device can select a video cover frame from multiple video frames in a video frame sequence based on the predicted probability that each video frame is a video cover frame.

[0106] In some embodiments, the video cover extraction device may use the video frames whose predicted probabilities rank in the top N as the video cover frames; where N is a positive integer greater than or equal to 1. Here, the video frames whose predicted probabilities rank in the top N score higher than other video frames during the prediction process in the fully connected layer, thus better reflecting the video content of the video to be processed.

[0107] In some embodiments, the video cover extraction device may also select video cover frames using other strategies based on predicted probabilities, depending on the actual selection strategy required. The specific selection is made according to the actual situation, and this application embodiment does not limit the choice.

[0108] It is understood that in the embodiments of this application, when the video cover extraction device predicts whether each video frame can become a video cover frame, it performs prediction processing based on multimodal fusion features that simultaneously include visual and textual modalities. In the prediction process, richer feature information is utilized, thus improving the accuracy of prediction and thereby improving the accuracy of video cover frame extraction.

[0109] In some embodiments of this application, based on the process of predicting multimodal fusion features using a fully connected layer in S1041, before performing a nonlinear transformation on the multimodal fusion features using a preset activation function of the fully connected layer, based on a preset weight matrix and a preset bias constant, and calculating the prediction score for each video frame as a video cover frame, S001-S003 can also be executed, as follows: S001. Obtain the video sample set and the initial fully connected layer; the network parameters of the initial fully connected layer include the initial weight matrix and the initial bias constant; the video sample set includes multiple sample videos; each of the multiple sample videos contains a labeled video cover frame, sample video text information and sample video sequence.

[0110] In this embodiment, before using the fully connected layer to predict multimodal fusion features, the weight matrix and bias constant of the fully connected layer can be trained first to improve the accuracy of the fully connected layer's prediction. Here, the video cover extraction device obtains an initial fully connected layer containing the initial weight matrix and initial bias constant in the network parameters, and obtains a video sample set for training the initial fully connected layer; wherein, the video sample set includes multiple sample videos; each of the multiple sample videos contains an annotated video cover frame, sample video text information, and a sample video sequence. The annotated video cover frame is a pre-annotated video cover frame in the sample video sequence of each sample video.

[0111] S002. In each training process, the initial fully connected layer is used to perform multimodal video cover prediction on each sample video based on the sample video text information and sample video sequence to obtain the predicted video cover frame for each training session.

[0112] S003. Based on the predicted video cover frame and the labeled video cover frame, obtain the training loss for each training session; and iteratively update the initial weight matrix and initial bias constant based on the training loss. When the training loss reaches the preset convergence condition, end the training and obtain a fully connected layer containing the preset weight matrix and preset bias constant.

[0113] In this embodiment, during each training iteration, based on the sample video text information and sample video sequence, an initial fully connected layer is used to predict the multimodal video cover for each sample video, obtaining the predicted video cover frame predicted by the initial fully connected layer in each training iteration. Then, the training loss for each training iteration is obtained based on the error between the predicted video cover frame and the labeled video cover frame. The initial weight matrix and initial bias constant are iteratively updated based on the training loss. When the training loss reaches a preset convergence condition, it indicates that the network prediction accuracy has reached the expected level, and training ends, resulting in a fully connected layer. At this point, the network parameters in the fully connected layer, namely the preset weight matrix and preset bias constant, have been adjusted to values ​​that meet the preset prediction accuracy during the machine learning process. The trained fully connected layer can then be used to predict multimodal fusion features.

[0114] It is understandable that by training and iteratively adjusting the parameters of the fully connected layer using a video sample set, the prediction accuracy of the fully connected layer can be improved, thereby further improving the accuracy of video cover frame extraction using the fully connected layer.

[0115] In some embodiments of this application, based on Figure 3 , Figure 5 , Figure 6 , Figure 7 as well as Figure 9 After S104, S105 can be executed, as follows: S105. On the preset video platform, display the video cover frame at the preset display position corresponding to the video to be processed, and display and recommend the video to be processed.

[0116] In this embodiment, after the video cover extraction device extracts the video cover frame from the video to be processed, it can display the video cover frame at a preset display position on a preset video platform to showcase and recommend the video to be processed.

[0117] In some embodiments, when the number of video cover frames is greater than 1, the video processing device can display and recommend the video to be processed in a carousel manner at a preset display position.

[0118] It is understood that, in the embodiments of this application, the video cover frame extracted based on the multimodal fusion features can more accurately represent the video content of the video to be processed. Therefore, when the video cover frame extracted based on the multimodal fusion features is used to display the video to be processed, a better display and recommendation effect can be achieved.

[0119] The following will describe an exemplary application of the embodiments of this application in a real-world application scenario.

[0120] This application provides a method for extracting video cover frames from a video to be processed using a video cover extraction model, such as... Figure 9 As shown, the video cover extraction model consists of a video frame feature extraction layer 91, a title text feature extraction layer 92, a multimodal fusion layer 93, and a video cover prediction layer 94. The video frame feature extraction layer 91 extracts visual modal feature information. A CNN network 910 is used to extract features from the video frame sequence vf={VF_1, VF_2, ..., VF_n} corresponding to the video to be processed, resulting in a visual feature sequence vf_v={vf_v_1, vf_v_2, ..., vf_v_n}, where VF_i represents a video frame in the video frame sequence, and vf_v_i represents the visual feature extracted from the i-th video frame after passing through the CNN network 910. The obtained visual feature sequence vf_v is then concatenated to obtain the visual feature information vf_vec. Here, the visual feature information is equivalent to the initial visual features. Furthermore, to facilitate fusion with the subsequently extracted text feature vectors, the video cover extraction model feeds the visual feature information vf_vec into the fully connected layer 912, mapping it to a K-dimensional visual feature vector vf. This visual feature vector vf contains visually relevant feature information from the video frames. Here, K is a preset feature dimension.

[0121] The title text feature extraction layer 92 is used to extract feature information of the text modality. The title text feature extraction layer 92 segments the video title of the video to be processed into words, and obtains the word segmentation sequence vt={w_1, w_2, ..., w_n}. The title text feature extraction layer 92 uses the pre-trained word vector model Word2Vec network 920 to vectorize the word segmentation sequence, and obtains the word vector sequence vt_w={vt_w_1, vt_w_2, ..., vt_w_n} corresponding to the video title. Then, the word vector sequence is input into the Bi-GRU network 921 to extract the video title feature sequence vt_wv={vt_wv_1, vt_wv_2, ...,vt_wv_n} of the video title, that is, the word vector sequence. Finally, similar to processing visual feature vectors, the extracted video title feature sequence is concatenated to obtain the video title feature vt_vec, i.e., the initial text feature, and then fed into the fully connected layer 922 to map it into a video title feature vector vt with dimension K. This video title feature vector vt contains the text feature information of the video title in the video.

[0122] The multimodal fusion layer 93 concatenates the video frame feature vector vf and the video title feature vector vt to obtain the multimodal fusion feature v_vec corresponding to the video to be processed. Then, the multimodal fusion feature v_vec is passed into the video cover prediction layer 94 for video cover selection.

[0123] Here, the K-dimensional video frame feature vector vf and video title feature vector vt are concatenated to obtain a multimodal fusion feature v_vec with a feature dimension of 2K. The video cover prediction layer 94 performs a nonlinear transformation based on each feature dimension of the multimodal fusion feature v_vec through the preset activation function shown in formula (1) in the fully connected layer 940, and outputs the score of each video frame as the video cover frame, i.e., the prediction score. The activation layer 941 converts the output score of the fully connected layer 940 into the probability of each video frame being the video cover frame according to formula (2); in this way, the video cover frame of the video to be processed can be determined according to the probability of each video frame being the video cover frame.

[0124] It is understood that, in the embodiments of this application, the multimodal approach can introduce both video frame and video title modal information into the video cover extraction model, thereby enabling the model to extract a cover that is more consistent with the video type and improving the accuracy of video cover extraction.

[0125] The following description continues to illustrate the exemplary structure of the video cover extraction device 255 provided in this application embodiment as a software module. In some embodiments, such as... Figure 2 As shown, the software module stored in the video cover extraction device 255 in the memory 250 may include: The visual modality feature extraction module 2551 is used to extract image features from the video frame sequence of the video to be processed, and obtain the visual feature vector corresponding to the video frame sequence. The text modality feature extraction module 2552 is used to extract word vector features from the video text information of the video to be processed, and obtain the text feature vector corresponding to the video text information. The multimodal feature fusion module 2553 is used to fuse the visual feature vector with the text feature vector to obtain the multimodal fusion features corresponding to the video to be processed; The cover selection module 2554 is used to select the video cover frame of the video to be processed from the video frame sequence based on the multimodal fusion features.

[0126] In some embodiments, the visual modality feature extraction module 2551 is further configured to extract image features from each video frame in the video frame sequence to obtain image features corresponding to each video frame; perform feature fusion on the image features corresponding to each video frame to obtain initial visual features corresponding to the video frame sequence; and map the initial visual features to a preset feature dimension to obtain the visual feature vector.

[0127] In some embodiments, the text modality feature extraction module 2552 is further configured to perform word segmentation and text vector conversion on the video text information to obtain a word vector sequence corresponding to the video text information; extract text features from each word vector in the word vector sequence to obtain text features corresponding to each word vector; perform feature fusion on the text features corresponding to each word vector to obtain initial text features corresponding to the video text information; and map the initial text features to a preset feature dimension to obtain the text feature vector.

[0128] In some embodiments, the cover selection module 2554 is further configured to use a fully connected layer to perform score prediction on the multimodal fusion features to obtain a prediction score corresponding to each video frame in the video frame sequence; normalize the prediction score corresponding to each video frame to obtain a prediction probability that each video frame is the video cover frame; the prediction probability represents the ratio of the prediction score corresponding to each video frame to the total prediction score corresponding to the video frame sequence; and select the video cover frame from the video frame sequence according to the prediction probability.

[0129] In some embodiments, the cover selection module 2554 is further configured to perform a nonlinear transformation on the multimodal fusion features using a preset activation function of the fully connected layer, and a preset weight matrix and a preset bias constant in the preset activation function, to obtain the prediction score corresponding to each video frame; the preset weight matrix and the preset bias constant are determined through the multimodal training process of the fully connected layer.

[0130] In some embodiments, the video cover extraction device further includes a training module. The training module is used to obtain a video sample set and an initial fully connected layer before performing a nonlinear transformation on the multimodal fusion features using a preset activation function of the fully connected layer and a preset weight matrix and a preset bias constant in the preset activation function to obtain the prediction score corresponding to each video frame. The network parameters of the initial fully connected layer include an initial weight matrix and an initial bias constant. The video sample set includes multiple sample videos. Each of the multiple sample videos contains an labeled video cover frame, sample video text information, and a sample video sequence. In each training process, multimodal video cover prediction is performed on each sample video based on the sample video text information and the sample video sequence through the initial fully connected layer to obtain the predicted video cover frame for each training session. The training loss for each training session is obtained based on the predicted video cover frame and the labeled video cover frame. The initial weight matrix and the initial bias constant are iteratively updated based on the training loss. When the training loss reaches a preset convergence condition, the training ends, resulting in the fully connected layer containing the preset weight matrix and the preset bias constant.

[0131] In some embodiments, the text modality feature extraction module 2552 is further configured to perform word segmentation on the video text information to obtain a word segmentation sequence; and to perform text vector transformation on the word segmentation sequence using a pre-trained word vector model to obtain the word vector sequence; the pre-trained word vector model is used to transform text information into vector representation.

[0132] In some embodiments, the video cover extraction device further includes a acquisition module, which is used to acquire the original video frame sequence corresponding to the video to be processed as the video frame sequence before performing image feature extraction on the video frame sequence of the video to be processed to obtain the visual feature vector corresponding to the video frame sequence; or, sample the original video frame sequence according to a preset sampling rate to obtain the video frame sequence; or, perform keyframe prediction on the original video frame sequence through a preset statistical model, and select multiple keyframes from the original video frame sequence according to the prediction result to obtain the video frame sequence; the statistical model is used to predict whether the original video frame is a keyframe based on the scene picture contained in the original video frame.

[0133] In some embodiments, the cover selection module 2554 is further configured to use the video frames whose predicted probabilities rank in the top N positions as the video cover frames; where N is a positive integer greater than or equal to 1.

[0134] In some embodiments, the video cover extraction device further includes a display module, which is used to display the video cover frame of the video to be processed at a preset display position corresponding to the video to be processed on a preset video platform after selecting the video cover frame of the video to be processed from the video frame sequence based on the multimodal fusion features, thereby displaying and recommending the video to be processed.

[0135] In some embodiments, the display module is further configured to display and recommend the video to be processed in a carousel manner at the preset display position when the number of video cover frames is greater than 1.

[0136] In some embodiments, the video text information includes at least one of the following: video title, video description, and video tags.

[0137] It should be noted that the description of the above device embodiments is similar to the description of the above method embodiments, and has similar beneficial effects. For technical details not disclosed in the device embodiments of the present invention, please refer to the description of the method embodiments of the present invention for understanding.

[0138] This application provides a computer-readable storage medium storing executable instructions. When these executable instructions are executed by a processor, they cause the processor to execute the video cover extraction method provided in this application. For example, ... Figure 3 , 5 The methods shown in 6 and 7.

[0139] In some embodiments, the computer-readable storage medium may be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or it may be a variety of devices including one or any combination of the above-mentioned memories.

[0140] In some embodiments, executable instructions may take the form of a program, software, software module, script, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a standalone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.

[0141] As an example, executable instructions may, but do not necessarily, correspond to files in a file system. They may be stored as part of a file that holds other programs or data, for example, in one or more scripts in a Hyper Text Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple collaborating files (e.g., a file that stores one or more modules, subroutines, or code sections).

[0142] As an example, executable instructions can be deployed to execute on a single computing device, or on multiple computing devices located in one location, or on multiple computing devices distributed across multiple locations and interconnected via a communication network.

[0143] In summary, the embodiments of this application introduce the concept of multimodal fusion into the video cover selection task. It simultaneously extracts the visual feature vectors of the video frame modality and the text feature vectors of the video text information modality, and fuses the feature vectors of the two modalities to obtain multimodal fusion features. These multimodal fusion features contain both image and text information from the video, encompassing a richer amount of video content information, which is more conducive to selecting video covers and thus improves the accuracy of video cover extraction.

[0144] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, and improvements made within the spirit and scope of this application are included within the scope of protection of this application.

Claims

1. A method for extracting video cover images, characterized in that, include: Image features are extracted from each video frame in the video frame sequence of the video to be processed to obtain the image features corresponding to each video frame; The image features corresponding to each video frame are concatenated to obtain the initial visual features corresponding to the video frame sequence. The initial visual features are then mapped to a preset feature dimension to obtain a visual feature vector. The video text information of the video to be processed is concatenated with text features to obtain the initial text features corresponding to the video text information. The initial text features are then mapped to the preset feature dimension to obtain a text feature vector. The visual feature vector and the text feature vector are concatenated to obtain the multimodal fusion feature corresponding to the video to be processed; Using a preset activation function in the fully connected layer, and utilizing a preset weight matrix and a preset bias constant within that preset activation function, a nonlinear transformation is performed on the multimodal fusion features to obtain the prediction score corresponding to each video frame. The preset weight matrix and the preset bias constant are determined through the multimodal training process of the fully connected layer, and the prediction score corresponding to each video frame is determined in the following way: Using the visual feature vector in the multimodal fusion features, scores are given based on visual evaluation indicators such as the clarity of the image or scene contained in each video frame, the richness of information, and the expressiveness of the main features of the image. Using the text feature vector in the multimodal fusion features, scores are given on the correlation between the text description and the image content in each video frame. By combining the scores from multiple modalities, a predicted score is obtained for each video frame. Based on the prediction score corresponding to each video frame, the video cover frame of the video to be processed is selected from the video frame sequence.

2. The method according to claim 1, characterized in that, The step of concatenating text features of the video text information of the video to be processed to obtain initial text features corresponding to the video text information, and mapping the initial text features to the preset feature dimension to obtain a text feature vector, includes: The video text information of the video to be processed is segmented into words and converted into text vectors to obtain the word vector sequence corresponding to the video text information; Text features are extracted from each word vector in the word vector sequence to obtain the text features corresponding to each word vector. The text features corresponding to each word vector are concatenated to obtain the initial text features corresponding to the video text information; The initial text features are mapped to a preset feature dimension to obtain the text feature vector.

3. The method according to any one of claims 1 or 2, characterized in that, The step of selecting the video cover frame of the video to be processed from the video frame sequence based on the prediction score corresponding to each video frame includes: The prediction score corresponding to each video frame is normalized to obtain the prediction probability that each video frame is the video cover frame; the prediction probability represents the ratio of the prediction score corresponding to each video frame to the total prediction score corresponding to the video frame sequence. The video cover frame is selected from the video frame sequence based on the predicted probability.

4. The method according to claim 1, characterized in that, Before obtaining the prediction score for each video frame by performing a nonlinear transformation on the multimodal fusion features using a preset activation function of the fully connected layer and a preset weight matrix and a preset bias constant in the preset activation function, the method further includes: Obtain a video sample set and an initial fully connected layer; the network parameters of the initial fully connected layer include an initial weight matrix and an initial bias constant; the video sample set includes multiple sample videos; each of the multiple sample videos contains a labeled video cover frame, sample video text information, and a sample video sequence. During each training process, the initial fully connected layer is used to perform multimodal video cover prediction on each sample video based on the sample video text information and the sample video sequence, so as to obtain the predicted video cover frame for each training session. The training loss for each training iteration is obtained based on the predicted video cover frame and the labeled video cover frame; the initial weight matrix and the initial bias constant are iteratively updated based on the training loss; when the training loss reaches the preset convergence condition, the training ends, and the fully connected layer containing the preset weight matrix and the preset bias constant is obtained.

5. The method according to claim 2, characterized in that, The step of segmenting and converting the video text information of the video to be processed into word vectors to obtain the word vector sequence corresponding to the video text information includes: The video text information is segmented into words to obtain a segmented sequence; The word segmentation sequence is transformed into a text vector using a pre-trained word vector model to obtain the word vector sequence; the pre-trained word vector model is used to convert text information into vector representation.

6. The method according to claim 1, characterized in that, Before extracting image features from each video frame in the video frame sequence of the video to be processed to obtain the image features corresponding to each video frame, the method further includes: Obtain the original video frame sequence; The original video frame sequence is used as the video frame sequence; or, The original video frame sequence is sampled according to a preset sampling rate to obtain the video frame sequence; or, The original video frame sequence is subjected to keyframe prediction using a preset statistical model. Based on the prediction results, multiple keyframes are selected from the original video frame sequence to obtain the video frame sequence. The statistical model is used to predict whether the original video frame is a keyframe based on the scene images contained in the original video frame.

7. The method according to claim 3, characterized in that, The step of selecting the video cover frame of the video to be processed from the video frame sequence based on the predicted probability includes: The video frames whose predicted probabilities are ranked in the top N positions are used as the video cover frames; where N is a positive integer greater than or equal to 1.

8. The method according to claim 7, after selecting the video cover frame of the video to be processed from the video frame sequence, the method further includes: On a preset video platform, the video cover frame is displayed at a preset display position corresponding to the video to be processed, thereby showcasing and recommending the video to be processed.

9. The method according to claim 8, characterized in that, The step of displaying the video cover frame on a preset video platform at a preset display position corresponding to the video to be processed, and showcasing and recommending the video to be processed, includes: When the number of video cover frames is greater than 1, the video to be processed is displayed and recommended in a carousel manner at the preset display position.

10. The method according to claim 1, characterized in that, The video text information includes at least one of the following: video title, video description, and video tags.

11. A video cover extraction device, characterized in that, include: The visual modality feature extraction module is used to extract image features from each video frame in the video frame sequence of the video to be processed, and obtain the image features corresponding to each video frame; to perform feature concatenation on the image features corresponding to each video frame, and obtain the initial visual features corresponding to the video frame sequence; and to map the initial visual features to a preset feature dimension to obtain a visual feature vector. The text modality feature extraction module is used to concatenate text features on the video text information of the video to be processed to obtain the initial text features corresponding to the video text information, and to map the initial text features to the preset feature dimension to obtain the text feature vector. The multimodal feature fusion module is used to concatenate the visual feature vector with the text feature vector to obtain the multimodal fusion features corresponding to the video to be processed. The cover selection module is used to perform a nonlinear transformation on the multimodal fusion features through a preset activation function of the fully connected layer, using a preset weight matrix and a preset bias constant in the preset activation function, to obtain the prediction score corresponding to each video frame. The preset weight matrix and the preset bias constant are determined through the multimodal training process of the fully connected layer. The prediction score corresponding to each video frame is determined in the following way: using the visual feature vector in the multimodal fusion features, a score is given based on visual evaluation indicators such as the clarity of the picture or scene contained in each video frame, the richness of information, and the expressiveness of the main features of the picture. The text feature vector in the multimodal fusion features is used to score the correlation between the text description and the picture content in each video frame. The scores under multiple modalities are combined to obtain the prediction score corresponding to each video frame. Based on the prediction score corresponding to each video frame, the video cover frame of the video to be processed is selected from the video frame sequence.

12. The apparatus as claimed in claim 11, characterized in that, The text modality feature extraction module is also used for: The video text information of the video to be processed is segmented into words and converted into text vectors to obtain a word vector sequence corresponding to the video text information; text features are extracted from each word vector in the word vector sequence to obtain the text features corresponding to each word vector; the text features corresponding to each word vector are concatenated to obtain the initial text features corresponding to the video text information; the initial text features are mapped to a preset feature dimension to obtain the text feature vector.

13. The apparatus as claimed in claim 11 or 12, characterized in that, The cover selection module is also used for: The prediction score corresponding to each video frame is normalized to obtain the prediction probability that each video frame is the video cover frame; the prediction probability represents the ratio of the prediction score corresponding to each video frame to the total prediction score corresponding to the video frame sequence; the video cover frame is selected from the video frame sequence according to the prediction probability.

14. The apparatus as claimed in claim 11, characterized in that, It also includes a training module, which is used to obtain a video sample set and an initial fully connected layer before obtaining the prediction score corresponding to each video frame by performing a nonlinear transformation on the multimodal fusion features through the preset activation function of the fully connected layer using the preset weight matrix and preset bias constant in the preset activation function. The network parameters of the initial fully connected layer include an initial weight matrix and an initial bias constant; the video sample set includes multiple sample videos; each of the multiple sample videos contains a labeled video cover frame, sample video text information, and a sample video sequence; During each training process, the initial fully connected layer is used to perform multimodal video cover prediction on each sample video based on the sample video text information and the sample video sequence, so as to obtain the predicted video cover frame for each training session. The training loss for each training iteration is obtained based on the predicted video cover frame and the labeled video cover frame; the initial weight matrix and the initial bias constant are iteratively updated based on the training loss; when the training loss reaches the preset convergence condition, the training ends, and the fully connected layer containing the preset weight matrix and the preset bias constant is obtained.

15. The apparatus as claimed in claim 12, characterized in that, The text modality feature extraction module is also used for: The video text information is segmented into words to obtain a segmented sequence; the segmented sequence is then transformed into a text vector using a pre-trained word vector model to obtain a word vector sequence; the pre-trained word vector model is used to convert text information into vector representation.

16. The apparatus as claimed in claim 11, characterized in that, It also includes a data acquisition module, used to extract image features from each video frame in the video frame sequence of the video to be processed, before obtaining the image features corresponding to each video frame. Obtain the original video frame sequence; use the original video frame sequence as the video frame sequence; or, sample the original video frame sequence according to a preset sampling rate to obtain the video frame sequence; or, perform keyframe prediction on the original video frame sequence using a preset statistical model, and select multiple keyframes from the original video frame sequence based on the prediction results to obtain the video frame sequence; the statistical model is used to predict whether the original video frame is a keyframe based on the scene scenes contained in the original video frame.

17. The apparatus as claimed in claim 13, characterized in that, The cover selection module is also used for: The video frames whose predicted probabilities are ranked in the top N positions are used as the video cover frames; where N is a positive integer greater than or equal to 1.

18. The apparatus as claimed in claim 17, characterized in that, It also includes a display module, which, after selecting the video cover frame of the video to be processed from the video frame sequence, displays the video cover frame at a preset display position corresponding to the video to be processed on a preset video platform, thereby displaying and recommending the video to be processed.

19. The apparatus as claimed in claim 18, characterized in that, The display module is also used for: When the number of video cover frames is greater than 1, the video to be processed is displayed and recommended in a carousel manner at the preset display position.

20. An electronic device, characterized in that, include: Memory, used to store executable instructions; A processor, when executing executable instructions stored in the memory, implements the method according to any one of claims 1 to 10.

21. A computer-readable storage medium, characterized in that, It stores executable instructions for implementing the method of any one of claims 1 to 10 when executed by a processor.

Images