Video processing method, device, equipment and storage medium

By performing frame extraction processing and quality detection on the video, using the aesthetic scoring model and target object attribute information, the video frame image is selected as the cover image, which solves the problems of low efficiency and insufficient accuracy of cover image determination in the prior art, and achieves high efficiency , accurate selection of video cover images.

CN116567369BActive Publication Date: 2024-11-19TENCENT TECHNOLOGY (SHENZHEN) CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202210104029.9
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-01-27
Publication Date
2024-11-19
Estimated Expiration
2042-01-27

AI Technical Summary

Technical Problem

The prior art is inefficient and has low accuracy when determining video cover images, and the cost of artificially specifying cover images is high, and the accuracy of specifying fixed frames as cover images is insufficient.

Method used

By extracting frames for video to be processed, quality detection of video frame images is performed based on aesthetic dimensions and target object attributes, multiple video frame images are selected as cover images, and weighted summation is used using aesthetic scoring model and target object attribute information to determine the cover image.

Benefits of technology

It realizes efficient and accurate determination of video cover images, reduces labor costs, and improves the thematic and aesthetic value of video cover images.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN116567369B_ABST
    Figure CN116567369B_ABST
Patent Text Reader

Abstract

The embodiments of the present application disclose a video processing method, device, equipment and storage medium. The method includes: performing frame extraction processing on the video to be processed to obtain a candidate image set, performing quality detection on N video frame images in the candidate image set based on at least aesthetic dimensions and target object attributes, and obtaining a quality detection result for each video frame image; selecting M video frame images from the candidate image set according to the quality detection result, and determining the M video frame images as the cover image of the video. It can be seen that by performing quality detection on the video frame images in the video, the quality detection results of each video frame image are obtained, and the cover image of the video can be determined more efficiently and accurately based on the quality detection results of each video frame image.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a video processing method, a video processing device, a computer equipment and a computer-readable storage medium. Background Art

[0002] With the continuous development of computer technology, a large number of videos have emerged on the Internet. In the process of video promotion and display, the cover image of the video is often displayed instead of directly displaying the video itself; the cover image of the video is usually selected from the video. Common methods for determining the cover image include: manually specifying the cover image, and specifying a fixed frame in the video (such as the first frame in the video) as the cover image. Practice has found that the method of manually specifying the cover image is inefficient and has high labor costs; the method of specifying a fixed frame in the video as the cover image has a low accuracy rate. Summary of the invention

[0003] The embodiments of the present application provide a video processing method, apparatus, device and storage medium, which can more efficiently and accurately determine the cover image of a video.

[0004] On the one hand, an embodiment of the present application provides a video processing method, the method comprising:

[0005] Perform frame extraction on the video to be processed to obtain a candidate image set, where the candidate image set includes N video frame images of the video, where N is a positive integer;

[0006] Based on at least the aesthetic dimension and the target object attribute, respectively perform quality detection on the N video frame images to obtain a quality detection result of each video frame image;

[0007] Selecting M video frame images from the candidate image set according to the quality detection result, where M is an integer greater than 1;

[0008] The M video frame images are determined as cover images of the video.

[0009] On the one hand, an embodiment of the present application provides a video processing device, the video processing device comprising:

[0010] An acquisition unit is used to perform frame extraction processing on the video to be processed to obtain a candidate image set, where the candidate image set includes N video frame images of the video, where N is a positive integer;

[0011] A processing unit performs quality detection on the N video frame images respectively based on at least the aesthetic dimension and the target object attribute to obtain a quality detection result for each video frame image;

[0012] and for selecting M video frame images from the candidate image set according to the quality detection result, where M is an integer greater than 1;

[0013] and used to determine the M video frame images as cover images of the video.

[0014] In one implementation, any one of the N video frame images is represented as the i-th video frame image, where i is a positive integer and i is less than or equal to N; the processing unit is used to perform quality detection on the N video frame images respectively to obtain a quality detection result of each video frame image, specifically for:

[0015] Obtain the corresponding aesthetic scoring model under the aesthetic dimension;

[0016] Calling the aesthetic scoring model to perform quality detection on the i-th video frame image, and obtaining the first quality score of the i-th video frame image under the aesthetic dimension;

[0017] The first quality score is determined as the quality detection result of the i-th video frame image.

[0018] In one embodiment, the aesthetic scoring model includes a convolution layer, a fully connected layer and an activation layer; a plurality of aesthetic score levels are provided in the aesthetic scoring model; the processing unit is used to call the aesthetic scoring model to perform quality detection on the i-th video frame image to obtain a first quality score of the i-th video frame image under the aesthetic dimension, specifically for:

[0019] Extract the features of the i-th video frame image through the convolution layer;

[0020] Call the fully connected layer to predict the probability of the i-th video frame image at each aesthetic score level based on the features of the i-th video frame image;

[0021] The activation layer is called to determine the aesthetic score level to which the i-th video frame image belongs according to the probability, and a first quality score of the i-th video frame image is determined according to the aesthetic score level.

[0022] In one implementation, any one of the N video frame images is represented as the i-th video frame image, where i is a positive integer and i is less than or equal to N;

[0023] The processing unit is used to perform quality detection on N video frame images respectively to obtain a quality detection result of each video frame image, specifically for:

[0024] Detect whether there is a target object in the i-th video frame image;

[0025] If the target object does not exist, the second quality score of the i-th video frame image under the target object attribute is set to a preset score;

[0026] If there is a target object, then according to the attribute information of the target object, the second quality score of the i-th video frame image under the target object attribute is calculated;

[0027] The second quality score is determined as the quality detection result of the i-th video frame image.

[0028] In one implementation, the attribute information of the target object includes the size and position of the target object; the processing unit is used to calculate the second quality score of the i-th video frame image under the target object attribute according to the attribute information of the target object, specifically for:

[0029] Determine a size score of the i-th video frame image according to a ratio between a size of the target object and a size of the i-th video frame image;

[0030] Determine the position score of the i-th video frame image according to the distance between the position of the target object and the target point of the i-th video frame image;

[0031] Based on the size score and the position score, a second quality score of the i-th video frame image under the target object attribute is calculated.

[0032] In one implementation, the target object is a human face, and the attribute information of the target object further includes an expression of the target object; the processing unit is further configured to:

[0033] Determine the expression score of the i-th video frame image according to the category of the expression of the target object;

[0034] Based on the expression score, the second quality score of the i-th video frame image under the target object attribute is updated.

[0035] In one implementation, if the first target object and the second target object exist in the i-th video frame image, the processing unit is used to calculate the second quality score of the i-th video frame image under the target object attribute according to the attribute information of the target object, specifically for:

[0036] Calculate, according to the attribute information of the first target object, a first candidate score of the i-th video frame image under the target object attribute;

[0037] Calculating a second candidate score of the i-th video frame image under the target object attribute according to the attribute information of the second target object;

[0038] The larger one of the first candidate score and the second candidate score is determined as the second quality score of the i-th video frame image under the target object attribute.

[0039] In one implementation, any one of the N video frame images is represented as the i-th video frame image, where i is a positive integer and i is less than or equal to N;

[0040] The processing unit is used to perform quality detection on N video frame images respectively to obtain a quality detection result of each video frame image, specifically for:

[0041] Obtaining a first quality score of the i-th video frame image under an aesthetic dimension, and obtaining a second quality score of the i-th video frame image under a target object attribute;

[0042] Performing a weighted summation on the first quality score and the second quality score to obtain a comprehensive quality score of the i-th video frame image;

[0043] The comprehensive quality score is determined as the quality detection result of the i-th video frame image.

[0044] In one implementation, the quality detection result includes a quality score; and the processing unit is used to select M video frame images from the candidate image set according to the quality detection result, specifically to:

[0045] Sort the video frame images in the candidate image set in descending order of quality scores;

[0046] According to the sorting result, M video frame images are selected from the candidate image set.

[0047] In one implementation, the processing unit is used to select M video frame images from the candidate image set according to the sorting result, specifically to:

[0048] Select the top M video frame images from the candidate image set; or,

[0049] According to the sorting result, P video frame images with quality scores greater than the score threshold are selected from the candidate image set; M video frame images with associated relationships are selected from the P candidate cover images, where P is an integer greater than or equal to M;

[0050] The association relationship between any two video frame images includes any of the following: having the same target object and belonging to the same subject.

[0051] In one embodiment, the processing unit is further configured to:

[0052] Output the cover image of the video;

[0053] The output method includes any of the following: displaying the cover images of the videos one by one, displaying the cover images of the videos in tiles, composing the cover images of the videos into an animation, or splicing and displaying them.

[0054] Accordingly, the present application provides a computer device, the device comprising:

[0055] A processor for loading and executing computer programs;

[0056] A computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the video processing method is implemented.

[0057] Accordingly, the present application provides a computer-readable storage medium, which stores a computer program, and the computer program is suitable for being loaded by a processor and executing the above-mentioned video processing method.

[0058] Accordingly, the present application provides a computer program product or a computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device performs the above-mentioned video processing method.

[0059] In the embodiment of the present application, the video to be processed is subjected to frame extraction processing to obtain a candidate image set, and the quality of the N video frame images in the candidate image set is respectively detected based on at least the aesthetic dimension and the target object attribute to obtain the quality detection result of each video frame image; M video frame images are selected from the candidate image set according to the quality detection result, and the M video frame images are determined as the cover image of the video. It can be seen that by performing quality detection on the video frame images in the video, the quality detection results of each video frame image are obtained, and the cover image of the video can be determined more efficiently and accurately based on the quality detection results of each video frame image. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0061] Figure 1 An architecture diagram of a video processing system provided in an embodiment of the present application;

[0062] Figure 2 A flowchart of a video processing method provided in an embodiment of the present application;

[0063] Figure 3 A flowchart of another video processing method provided in an embodiment of the present application;

[0064] Figure 4a A schematic diagram of calling an aesthetic scoring model to perform quality detection on a video frame image provided in an embodiment of the present application;

[0065] Figure 4b A schematic diagram of the relationship between a size fraction and a ratio provided in an embodiment of the present application;

[0066] Figure 4c A schematic diagram of the relationship between a position score and a distance provided in an embodiment of the present application;

[0067] Figure 4d A flow chart of selecting a cover image for a video provided in an embodiment of the present application;

[0068] Figure 5 A schematic diagram of the structure of a video processing device provided in an embodiment of the present application;

[0069] Figure 6 A schematic diagram of the structure of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0070] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0071] The embodiments of the present application involve cloud technology and artificial intelligence. The following briefly introduces the relevant terms and concepts of cloud technology and artificial intelligence:

[0072] Cloud technology refers to a hosting technology that unifies hardware, software, network and other resources within a wide area network or a local area network to achieve data computing, storage, processing and sharing. In this application, cloud technology is mainly used to integrate videos on the Internet and provide the service target with the cover image of the video so that the service target can quickly understand the theme and content of the video.

[0073] Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that machines have the functions of perception, reasoning and decision-making.

[0074] AI technology is a comprehensive discipline that covers a wide range of fields, including both hardware-level and software-level technologies. The basic technologies of artificial intelligence generally include sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, large application processing technology, operation / interaction systems, mechatronics and other technologies. Artificial intelligence software technology mainly includes computer vision technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0075] Among them, computer vision technology is a science that studies how to make machines "see". To put it more specifically, it refers to using cameras and computers to replace human eyes to identify, track and measure targets, and further perform graphic processing to make computer processing into images that are more suitable for human eye observation or transmission to instruments for detection. As a scientific discipline, computer vision studies related theories and technologies, and attempts to establish an artificial intelligence system that can obtain information from images or multidimensional data; it usually includes image processing, video processing, video semantic understanding, video content / behavior recognition and other technologies. The embodiment of the present application mainly involves using an aesthetic scoring model to perform quality detection on video frame images of a video, and obtain a first quality score for each video frame image under the aesthetic dimension.

[0076] Machine learning is a multi-disciplinary subject that involves probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory and other disciplines. It specializes in studying how computers simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of AI and the fundamental way to make computers intelligent. Its applications are spread across all areas of artificial intelligence. Machine learning / deep learning usually includes artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and learning by teaching.

[0077] Deep learning: The concept of deep learning originates from the study of artificial neural networks. A multilayer perceptron with multiple hidden layers is a deep learning structure. Deep learning combines low-level features to form more abstract high-level representation attribute categories or features to discover distributed feature representations of data. The embodiments of the present application mainly involve optimizing the initial model using a training data set to obtain an aesthetic scoring model.

[0078] Based on the above introduction to the cloud technology and artificial intelligence involved in the embodiments of this application, the following briefly introduces the video processing solution proposed by the embodiments of this application based on the above cloud technology and artificial intelligence, which can more efficiently and accurately determine the cover image of the video. Figure 1 , Figure 1 This is an architecture diagram of a video processing system provided in an embodiment of the present application. Figure 1As shown, the video processing system may include: a terminal device 101, a server 102 and a terminal device 103. The video processing method provided in the embodiment of the present application may be executed by the server 102, or by a terminal device equipped with a video processing device, such as the terminal device 103. The terminal device 101 and the terminal device 103 may include but are not limited to: smart phones (such as Android phones, IOS phones, etc.), tablet computers, portable personal computers, mobile Internet devices (Mobile Internet Devices, referred to as MID), vehicle terminals, intelligent voice interaction devices, smart home appliances, aircraft, and other smart devices with display functions, which are not limited in the embodiment of the present application. The server 102 may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network, content distribution network), and big data and artificial intelligence platforms, which are not limited in the embodiment of the present application.

[0079] It should be noted that Figure 1 In the terminal device 101, the server 103 and the terminal device 103 can be directly or indirectly connected through wired communication or wireless communication, and this application does not limit this. The number of terminal devices and servers is only used for example and does not constitute an actual limitation of this application; for example, the video processing system may also include a terminal device 104 or a server 105.

[0080] In the specific implementation, the general principle of the video processing solution is as follows:

[0081] (1) The server 102 obtains the video to be processed uploaded by the terminal device 101 (such as obtaining the Uniform Resource Locator (URL) of the video), extracts frames of the video to be processed, and obtains a candidate image set, where the candidate image set includes N video frame images of the video, where N is a positive integer; for example, the server 102 can extract video frame images from the video to be processed at a preset period (such as 1 second) to obtain a candidate image set; it can also extract key frames of the video to be processed (such as a frame of a target object during a motion change process) to obtain a candidate image set.

[0082] (2) The server 102 performs quality detection on the N video frame images respectively based on at least the aesthetic dimension and the target object attribute, and obtains the quality detection result of each video frame image. On the one hand, the server 102 performs quality detection on the N video frame images respectively under the aesthetic dimension (such as scoring the N video frame images respectively through the aesthetic scoring model) to obtain the quality detection result of each video frame image. On the other hand, the server 102 performs quality detection on the N video frame images respectively under the target object attribute (such as detecting whether the video frame image contains a face) to obtain the quality detection result of each video frame image. Further, the server 102 performs quality detection on the N video frame images respectively under the target object attribute and the aesthetic dimension to obtain the quality detection result of each video frame image.

[0083] It should be noted that, under the aesthetic dimension, quality detection is performed on N video frame images respectively, which can be understood as: quality detection of video frame images is performed based on one or more factors such as color, hue, clarity, image structure, etc. of the video frame images; the aesthetic dimension and target object attributes are only used for examples. In practical applications, new dimensions can be added according to actual needs to perform quality detection on video frame images.

[0084] (3) The server 102 selects M video frame images from the candidate image set according to the quality detection result, where M is an integer greater than 1; the quality detection result may include one or more of the following: quality level, quality score. In one embodiment, the quality detection result includes a quality score, and the quality score of the M video frame images selected by the server 102 from the candidate image set is greater than a score threshold. In another embodiment, the quality detection result includes a quality level, and the M video frame images selected by the server 102 from the candidate image set belong to a target quality level.

[0085] (4) The server 102 determines the M video frame images as the cover images of the video, and sends the cover images of the video to the terminal device 103, so that the terminal device 103 displays the cover images of the video to the service target object. It can be understood that selecting multiple video frame images as the cover images of the video can more completely and comprehensively display the content of the video.

[0086] In the embodiment of the present application, the video to be processed is subjected to frame extraction processing to obtain a candidate image set, and the quality of the N video frame images in the candidate image set is respectively detected based on at least the aesthetic dimension and the target object attribute to obtain the quality detection result of each video frame image; M video frame images are selected from the candidate image set according to the quality detection result, and the M video frame images are determined as the cover image of the video. It can be seen that by performing quality detection on the video frame images in the video, the quality detection results of each video frame image are obtained, and the cover image of the video can be determined more efficiently and accurately based on the quality detection results of each video frame image.

[0087] Based on the above video processing solution, the embodiment of the present application proposes a more detailed video processing method. The video processing method proposed in the embodiment of the present application will be described in detail below in conjunction with the accompanying drawings.

[0088] See also Figure 2 , Figure 2 A flowchart of a video processing method provided in an embodiment of the present application. The video processing method can be executed by a computer device, which can be Figure 1 The server 102 is shown. Figure 2 As shown, the video processing method may include but is not limited to steps S201-S204:

[0089] S201, extract frames from the video to be processed to obtain a set of candidate images.

[0090] The candidate image set includes N video frame images of the video, where N is a positive integer. The computer device may perform frame extraction processing on the video to be processed in a manner including but not limited to: extracting video frame images from the video to be processed according to a preset period (such as 0.1s); extracting key frames of the video to be processed (such as a frame of a target object in a motion change process, the first frame of a scene switch, etc.) to obtain a candidate image set; extracting video frame images from the video to be processed according to a preset number of video frame images, etc.

[0091] Optionally, if the number of video frame images contained in the video to be processed is less than a threshold, all video frame images contained in the video may be used as a candidate image set.

[0092] S202: Perform quality detection on the N video frame images respectively based on at least the aesthetic dimension and the target object attribute to obtain a quality detection result for each video frame image.

[0093] In one implementation, the computer device performs quality detection on N video frame images respectively based on aesthetic dimensions and target object attributes to obtain a quality detection result for each video frame image.

[0094] On the one hand, the computer device obtains the corresponding aesthetic scoring model under the aesthetic dimension, and the aesthetic scoring model can be obtained by optimizing and training the initial model using the training data set; the computer device calls the aesthetic scoring model to perform quality detection on the i-th video frame image to obtain the first quality score of the i-th video frame image under the aesthetic dimension. Among them, the aesthetic scoring model includes a convolution layer, a fully connected layer and an activation layer, and the convolution layer is used to extract the features of the i-th video frame image; the fully connected layer and the activation layer are used to predict the first quality score of the i-th video frame image according to the features of the i-th video frame image; the i-th video frame image can be any one of the N video frame images.

[0095] On the other hand, the computer device detects whether there is a target object in the i-th video frame image; if there is no target object, the second quality score of the i-th video frame image under the target object attribute is set to a preset score (such as 0 points); accordingly, if there is a target object, the second quality score of the i-th video frame image under the target object attribute is calculated according to the attribute information of the target object; wherein the attribute information of the target object may include at least one of the following: the size of the target object, the position of the target object; when the target object is a face, the attribute information of the target object may also include the expression of the target object, and the i-th video frame image may be any one of the N video frame images. For example, the computer device may calculate the second quality score of the i-th video frame image under the target object attribute according to the proportion of the target object in the i-th video frame image; or, calculate the second quality score of the i-th video frame image under the target object attribute according to the position of the target object in the i-th video frame image.

[0096] After obtaining the first quality score of the i-th video frame image under the aesthetic dimension and the second quality score under the target object attribute, the computer device can perform weighted summation on the first quality score and the second quality score to obtain the comprehensive quality score of the i-th video frame image; wherein the weights corresponding to each dimension can be dynamically adjusted according to demand, and this application does not limit this, and the i-th video frame image can be any one of the N video frame images. Further, after obtaining the comprehensive quality score of the i-th video frame image, the computer device can determine the comprehensive quality score as the quality detection result of the i-th video frame image.

[0097] Optionally, the computer device can also add new dimensions based on the aesthetic dimension and the target object attribute according to actual needs to perform quality detection on the N video frame images respectively to obtain the quality detection result of each video frame image. The specific implementation method can refer to the above implementation method and will not be repeated here.

[0098] S203 . Select M video frame images from the candidate image set according to the quality detection result.

[0099] The quality inspection results may include one or more of the following: quality grade, quality score.

[0100] In one embodiment, the quality detection result includes a quality score, and the computer device selects M video frame images from the candidate image set, and the quality scores of these M video frame images are greater than a score threshold; for example, assuming that the candidate image set includes video frame images 1-video frame images 5, the quality score of video frame image 1 is 93, the quality score of video frame image 2 is 71, the quality score of video frame image 3 is 87, the quality score of video frame image 4 is 63, the quality score of video frame image 5 is 83, and the score threshold is 85, then the computer device selects video frame image 1 and video frame image 3 according to the quality scores of video frame images 1-video frame image 5.

[0101] In another embodiment, the quality detection result includes a quality level, and the computer device selects M video frame images from the candidate image set, and these M video frame images belong to the target quality level; for example, assuming that the candidate image set includes video frame images 1-video frame images 5, the quality level of video frame image 1 is level B, the quality level of video frame image 2 is level A, the quality level of video frame image 3 is level A, the quality level of video frame image 4 is level C, the quality level of video frame image 5 is level A, and the target quality level is level A, then the computer device selects video frame image 2, video frame image 3 and video frame image 5 according to the quality levels of video frame images 1-video frame image 5.

[0102] S204: Determine the M video frame images as cover images of the video.

[0103] The computer device uses the selected M video frame images as the cover images of the video.

[0104] In the embodiment of the present application, the video to be processed is subjected to frame extraction processing to obtain a candidate image set, and the quality of the N video frame images in the candidate image set is respectively detected based on at least the aesthetic dimension and the target object attribute to obtain the quality detection result of each video frame image; M video frame images are selected from the candidate image set according to the quality detection result, and the M video frame images are determined as the cover image of the video. It can be seen that by performing quality detection on the video frame images in the video, the quality detection results of each video frame image are obtained, and the cover image of the video can be determined more efficiently and accurately based on the quality detection results of each video frame image.

[0105] See also Figure 3 , Figure 3 A flowchart of another video processing method provided in an embodiment of the present application. The video processing method can be executed by a computer device, which can be Figure 1The server 102 is shown. Figure 3 As shown, the video processing method may include but is not limited to steps S301-S307:

[0106] S301 , extract frames from the video to be processed to obtain a set of candidate images.

[0107] The specific implementation of step S301 can be found in Figure 2 The implementation of step S201 will not be repeated here.

[0108] S302: Calling an aesthetic scoring model to perform quality detection on the i-th video frame image to obtain a first quality score of the i-th video frame image in an aesthetic dimension.

[0109] The aesthetic scoring model can be obtained by the computer device from the target device, or it can be obtained by the computer device optimizing and training the initial model through a training data set; specifically, the training data set includes input data and the aesthetic scores corresponding to the input data. The computer device can use the initial model to predict the input data to obtain the predicted score of the input data, and based on the difference between the predicted score of the input data and the aesthetic score corresponding to the input data, optimize and adjust the parameters in the initial model to obtain the aesthetic scoring model.

[0110] Figure 4a A schematic diagram of calling an aesthetic scoring model to perform quality detection on a video frame image is provided in an embodiment of the present application. Figure 4a As shown, the aesthetic scoring model includes a convolutional layer and a fully connected layer; in addition, the aesthetic scoring model is also provided with multiple aesthetic score levels. The computer device calls the aesthetic scoring model to perform quality detection on the i-th video frame image, and obtains the first quality score of the i-th video frame image in the aesthetic dimension. The specific implementation method is: firstly extract the features of the i-th video frame image through the convolutional layer, and then call the fully connected layer to predict the probability of the i-th video frame image at each aesthetic score level according to the features of the i-th video frame image; then call the activation layer (such as the softmax function) to determine the aesthetic score level to which the i-th video frame image belongs according to the probability, and determine the first quality score of the i-th video frame image according to the aesthetic score level to which the i-th video frame image belongs; wherein the i-th video frame image is any one of the N video frame images, i is a positive integer, and i is less than or equal to N.

[0111] S303: Obtain a second quality score of the i-th video frame image under the target object attribute.

[0112] The target object is included in at least one video frame image of the video to be processed, and the target object can be set according to actual needs; for example, the target object can specifically refer to a human face, an animal, a specific object, etc., and this application does not limit this. The target object attribute refers to the attribute of the target object that affects the second quality score of the video frame image, and the attribute of the target object may include but is not limited to: the size of the target object in the video frame image, and the position of the target object in the video frame image.

[0113] Since different target objects may have different attributes, in practical applications the attributes of the target object may also be determined based on actual needs; for example, when the target object is a face, the attributes of the target object may include not only the size of the face in the video frame image, the position of the face in the video frame image, but also the expression of the face.

[0114] In one embodiment, the computer device detects whether there is a target object in the i-th video frame image; for example, the computer device can detect whether there is a target object in the i-th video frame image through a multi-task convolutional neural network (MTCNN). The target object may include but is not limited to: a human face, an animal, a target object, etc. If there is no target object in the i-th video frame image, the second quality score of the i-th video frame image under the target object attribute is set to a preset score (such as 0 points). Correspondingly, if there is a target object in the i-th video frame image, the second quality score of the i-th video frame image under the target object attribute is calculated according to the attribute information of the target object.

[0115] In one embodiment, the attribute information of the target object includes the size of the target object and the position of the target object. The specific implementation method of the computer device calculating the second quality score of the i-th video frame image under the target object attribute according to the attribute information of the target object is:

[0116] On the one hand, the computer device determines a size score of the i-th video frame image according to a ratio between a size of the target object and a size of the i-th video frame image; Figure 4b This is a schematic diagram of the relationship between a size fraction and a ratio provided in an embodiment of the present application. Figure 4b As shown, the computer device can determine the size score of the i-th video frame image based on the linear relationship between the size score and the ratio; for example, when the ratio between the size of the target object and the size of the i-th video frame image is S (such as S=0.3), the size score of the i-th video frame image is 1.

[0117] On the other hand, the computer device determines the position score of the i-th video frame image based on the distance between the position of the target object and the target point of the i-th video frame image; the target point can be the center point, or it can be other core points in the i-th video frame image except the center point set according to needs; for example, assuming that the length of the i-th video frame image is L, the height is H, and the lower left corner is the origin, the coordinates of the core point can be (0.5L, 0.7H), that is, the height of the core point is higher than the height of the center point, and this position is more consistent with the position of the face in the actual scene. Figure 4c A schematic diagram of the relationship between a position score and a distance provided in an embodiment of the present application. Figure 4c As shown, the computer device can determine the position score of the i-th video frame image based on the linear relationship between the position score and the distance; for example, when the target object is located at the target position of the i-th video frame image (that is, the distance between the target object and the target point of the i-th video frame image is 0), the position score of the i-th video frame image is 1; for another example, when the target object is located at the edge of the i-th video frame image (that is, the distance between the target object and the target point of the i-th video frame image is D), the position score of the i-th video frame image is 0.

[0118] After obtaining the size score and position score of the i-th video frame image, the computer device can calculate the second quality score of the i-th video frame image under the target object attribute based on the size score and the position score; for example, the second quality score = size score*A+position score*B; where A is the weight of the size score and B is the weight of the position score.

[0119] Furthermore, if the target object is a face, the attributes of the target object may include not only the position and size of the face but also the expression of the face; in this case, the computer device determines the expression score of the i-th video frame image according to the category of the expression of the target object; specifically, the expression of the face can be divided into: anger, disgust, fear, sadness, happiness, surprise, calmness, etc.; wherein the scores corresponding to anger, disgust, fear, and sadness are preset scores 1, the scores corresponding to calmness are preset scores 2, the scores corresponding to surprise are preset scores 3, and the scores corresponding to happiness are preset scores 4, and preset scores 1<preset scores 2<preset scores 3<preset scores 4. After determining the expression score of the i-th video frame image according to the category of the expression of the target object, the computer device may update the second quality score of the i-th video frame image under the target object attributes based on the expression score; for example, the updated second quality score = the second quality score before updating*C+expression score*E; wherein C is the weight of the second quality score before updating, and E is the weight of the expression score.

[0120] Similarly, when the target object includes a new attribute, the computer device may update the second quality score of the i-th video frame image under the target object attribute based on the score of the new attribute to obtain the updated second quality score of the i-th video frame image under the target object attribute.

[0121] Optionally, the computer device may also determine the second quality score of the i-th video frame image under the target object attribute through a single attribute of the target object; for example, the second quality score of the i-th video frame image under the target object attribute may be determined based on the position of the target object in the i-th video frame image.

[0122] In another embodiment, if there are multiple target objects in the i-th video frame image, the computer device calculates the alternative scores of the i-th video frame image under the target object attributes based on the attribute information of each target object, and uses the highest alternative score among the alternative scores as the second quality score of the i-th video frame image under the target object attributes; for example, assuming that there are a first target object and a second target object in the i-th video frame image, the computer device calculates the first alternative score of the i-th video frame image under the target object attributes based on the attribute information of the first target object; calculates the second alternative score of the i-th video frame image under the target object attributes based on the attribute information of the second target object; and determines the larger of the first alternative score and the second alternative score as the second quality score of the i-th video frame image under the target object attributes.

[0123] S304 . Perform a weighted summation on the first quality score and the second quality score to obtain a comprehensive quality score of the i-th video frame image, and determine the comprehensive quality score as a quality detection result of the i-th video frame image.

[0124] In one implementation, the comprehensive quality score of the i-th video frame image = first weight * first quality score + second weight * second quality score; wherein the sum of the first weight and the second weight is 1. In practical applications, the values ​​of the first weight and the second weight can be dynamically adjusted according to actual needs. For example, for landscape videos, the first weight can be set to be greater than the second weight; for plot videos, the first weight can be set to be less than the second weight.

[0125] S305 . Select M video frame images from the candidate image set according to the quality detection result.

[0126] The computer device may obtain quality detection results of M video frame images according to the above steps S302 to S304, where M is an integer greater than 1, and the value of M may be preset or specified by the service target object.

[0127] In one embodiment, the quality detection result includes a quality score, and the computer device sorts the video frame images in the candidate image set in descending order of the quality score; selects the top M video frame images from the candidate image set; for example, assuming that the candidate image set includes video frame images 1-video frame images 5, and the result of sorting video frame images 1-video frame images 5 in descending order of the quality score is: video frame image 2, video frame image 5, video frame image 1, video frame image 4, video frame image 3. If M=3, the computer device selects video frame image 2, video frame image 5 and video frame image 1.

[0128] In another embodiment, the quality detection result includes a quality score, and the computer device sorts each video frame image in the candidate image set in order of quality score from high to low; selects P video frame images with quality scores greater than the score threshold from the candidate image set according to the sorting result; selects M video frame images with an associated relationship from the P candidate cover images, where P is an integer greater than or equal to M; wherein the association relationship between any two video frame images includes any of the following: having the same target object and belonging to the same theme. For example, suppose the candidate image set includes video frame images 1-video frame images 5, and the result of sorting video frame images 1-video frame images 5 in order of quality scores from high to low is: video frame image 2, video frame image 5, video frame image 1, video frame image 4, video frame image 3. If M=2 and P=3, the computer device selects video frame image 2, video frame image 5 and video frame image 1 from the candidate image set. If both video frame image 5 and video frame image 1 contain the target object, the computer device selects video frame image 5 and video frame image 1. It is understandable that selecting multiple video frame images with the same target object or belonging to the same theme as the video cover can make the video cover more story-telling and express the theme of the video more comprehensively and completely (i.e., have stronger expressive ability).

[0129] S306: Determine the M video frame images as cover images of the video.

[0130] The specific implementation of step S306 can be found in Figure 2 The implementation of step S204 will not be repeated here.

[0131] Figure 4d This is a flow chart of selecting a cover image for a video provided in an embodiment of the present application. Figure 4dAs shown, the computer device first extracts frames from the acquired video to be processed to obtain multiple video frame images, and performs quality detection on each video frame image from the aesthetic dimension and the target object attribute, and obtains the quality detection result of the video frame image based on the first quality score of each video frame image under the aesthetic dimension and the second quality score under the target object attribute, and finally selects the cover of the video based on the quality detection result of each video frame image.

[0132] S307: Output the cover image of the video.

[0133] The output method includes any of the following: displaying the video cover images one by one, displaying the video cover images in a tiled manner, composing an animation of the video cover images, or splicing and displaying the video cover images. In addition, the computer device can also send the video cover image to other devices other than itself (such as sending the video cover image to other devices in the form of a URL through a communication interface) so that the device displays the video cover image.

[0134] In an embodiment of the present application, the video to be processed is subjected to frame extraction processing to obtain a candidate image set, and quality detection is performed on N video frame images in the candidate image set at least based on aesthetic dimensions and target object attributes to obtain quality detection results for each video frame image; M video frame images are selected from the candidate image set according to the quality detection results, and the M video frame images are determined as the cover image of the video. It can be seen that by performing quality detection on the video frame images in the video, the quality detection results of each video frame image are obtained, and the cover image of the video can be determined more efficiently and accurately based on the quality detection results of each video frame image. In addition, the video processing method provided in the present application can make the selected video cover more thematic, story-telling and aesthetically valuable, improve the user's experience of browsing videos, and effectively reduce labor costs.

[0135] The method of the embodiment of the present application is described in detail above. In order to facilitate better implementation of the above scheme of the embodiment of the present application, the device of the embodiment of the present application is provided below accordingly.

[0136] See also Figure 5 , Figure 5 A schematic diagram of the structure of a video processing device provided in an embodiment of the present application, the device can be mounted on a computer device, and the computer device can specifically be Figure 1 Server 102 is shown. Figure 5 The video processing device shown can be used to perform the above Figure 2 and Figure 3 Some or all of the functions of the described method embodiments. Figure 5 , the detailed description of each unit is as follows:

[0137] An acquisition unit 501 is used to perform frame extraction processing on the video to be processed to obtain a candidate image set, where the candidate image set includes N video frame images of the video, where N is a positive integer;

[0138] The processing unit 502 is used to perform quality detection on the N video frame images respectively based on at least the aesthetic dimension and the target object attribute to obtain a quality detection result of each video frame image;

[0139] and for selecting M video frame images from the candidate image set according to the quality detection result, where M is an integer greater than 1;

[0140] and used to determine the M video frame images as cover images of the video.

[0141] In one implementation, any one of the N video frame images is represented as the i-th video frame image, where i is a positive integer and i is less than or equal to N; the processing unit 502 is used to perform quality detection on the N video frame images respectively to obtain a quality detection result of each video frame image, specifically for:

[0142] Obtain the corresponding aesthetic scoring model under the aesthetic dimension;

[0143] Calling the aesthetic scoring model to perform quality detection on the i-th video frame image, and obtaining the first quality score of the i-th video frame image under the aesthetic dimension;

[0144] The first quality score is determined as the quality detection result of the i-th video frame image.

[0145] In one embodiment, the aesthetic scoring model includes a convolutional layer, a fully connected layer and an activation layer; the aesthetic scoring model is provided with multiple aesthetic score levels; the processing unit 502 is used to call the aesthetic scoring model to perform quality detection on the i-th video frame image to obtain a first quality score of the i-th video frame image under the aesthetic dimension, specifically for:

[0146] Extract the features of the i-th video frame image through the convolution layer;

[0147] Call the fully connected layer to predict the probability of the i-th video frame image at each aesthetic score level based on the features of the i-th video frame image;

[0148] The activation layer is called to determine the aesthetic score level to which the i-th video frame image belongs according to the probability, and a first quality score of the i-th video frame image is determined according to the aesthetic score level.

[0149] In one implementation, any one of the N video frame images is represented as the i-th video frame image, where i is a positive integer and i is less than or equal to N;

[0150] The processing unit 502 is used to perform quality detection on the N video frame images respectively to obtain a quality detection result of each video frame image, specifically for:

[0151] Detect whether there is a target object in the i-th video frame image;

[0152] If the target object does not exist, the second quality score of the i-th video frame image under the target object attribute is set to a preset score;

[0153] If there is a target object, then according to the attribute information of the target object, the second quality score of the i-th video frame image under the target object attribute is calculated;

[0154] The second quality score is determined as the quality detection result of the i-th video frame image.

[0155] In one implementation, the attribute information of the target object includes the size and position of the target object; the processing unit 502 is used to calculate the second quality score of the i-th video frame image under the target object attribute according to the attribute information of the target object, specifically for:

[0156] Determine a size score of the i-th video frame image according to a ratio between a size of the target object and a size of the i-th video frame image;

[0157] Determine the position score of the i-th video frame image according to the distance between the position of the target object and the target point of the i-th video frame image;

[0158] Based on the size score and the position score, a second quality score of the i-th video frame image under the target object attribute is calculated.

[0159] In one implementation, the target object is a human face, and the attribute information of the target object further includes an expression of the target object; the processing unit 502 is further configured to:

[0160] Determine the expression score of the i-th video frame image according to the category of the expression of the target object;

[0161] Based on the expression score, the second quality score of the i-th video frame image under the target object attribute is updated.

[0162] In one implementation, if the first target object and the second target object exist in the i-th video frame image, the processing unit 502 is used to calculate the second quality score of the i-th video frame image under the target object attribute according to the attribute information of the target object, specifically for:

[0163] Calculate, according to the attribute information of the first target object, a first candidate score of the i-th video frame image under the target object attribute;

[0164] Calculating a second candidate score of the i-th video frame image under the target object attribute according to the attribute information of the second target object;

[0165] The larger one of the first candidate score and the second candidate score is determined as the second quality score of the i-th video frame image under the target object attribute.

[0166] In one implementation, any one of the N video frame images is represented as the i-th video frame image, where i is a positive integer and i is less than or equal to N;

[0167] The processing unit 502 is used to perform quality detection on the N video frame images respectively to obtain a quality detection result of each video frame image, specifically for:

[0168] Obtaining a first quality score of the i-th video frame image under an aesthetic dimension, and obtaining a second quality score of the i-th video frame image under a target object attribute;

[0169] Performing a weighted summation on the first quality score and the second quality score to obtain a comprehensive quality score of the i-th video frame image;

[0170] The comprehensive quality score is determined as the quality detection result of the i-th video frame image.

[0171] In one implementation, the quality detection result includes a quality score; the processing unit 502 is used to select M video frame images from the candidate image set according to the quality detection result, specifically to:

[0172] Sort the video frame images in the candidate image set in descending order of quality scores;

[0173] According to the sorting result, M video frame images are selected from the candidate image set.

[0174] In one implementation, the processing unit 502 is configured to select M video frame images from the candidate image set according to the sorting result, specifically to:

[0175] Select the top M video frame images from the candidate image set; or,

[0176] According to the sorting result, P video frame images with quality scores greater than the score threshold are selected from the candidate image set; M video frame images with associated relationships are selected from the P candidate cover images, where P is an integer greater than or equal to M;

[0177] The association relationship between any two video frame images includes any of the following: having the same target object and belonging to the same subject.

[0178] In one implementation, the processing unit 502 is further configured to:

[0179] Output the cover image of the video;

[0180] The output method includes any of the following: displaying the cover images of the videos one by one, displaying the cover images of the videos in tiles, composing the cover images of the videos into an animation, or splicing and displaying them.

[0181] According to one embodiment of the present application, Figure 2 and Figure 3 Some of the steps involved in the video processing method shown can be represented by Figure 5 The video processing device shown in the figure is executed by each unit. For example, Figure 2 The step S201 shown in FIG. 1 can be performed by Figure 5 The acquisition unit 501 shown in FIG. 1 is executed, and steps S202 to S204 can be performed by Figure 5 The processing unit 502 shown executes; Figure 3 Steps S301 and S303 shown in FIG. 1 can be represented by Figure 5 The acquisition unit 501 shown in FIG. 1 is executed, step S302, and steps S304 to S307 can be performed by Figure 5 Processing unit 502 is shown executing. Figure 5 The various units in the video processing device shown can be respectively or all merged into one or several other units to constitute, or some of the units (some) can also be split into multiple smaller units in function to constitute, which can achieve the same operation without affecting the realization of the technical effects of the embodiments of the present application. The above-mentioned units are divided based on logical functions. In practical applications, the functions of one unit can also be realized by multiple units, or the functions of multiple units can be realized by one unit. In other embodiments of the present application, the video processing device can also include other units. In practical applications, these functions can also be implemented with the assistance of other units, and can be implemented by the collaboration of multiple units.

[0182] According to another embodiment of the present application, the program can be executed by running on a general computing device such as a computer including a central processing unit (CPU), a random access memory medium (RAM), a read-only memory medium (ROM) and other processing elements and storage elements. Figure 2 and Figure 3 A computer program (including program code) for each step involved in the corresponding method shown in Figure 5 The video processing device shown in the embodiment of the present application is used to implement the video processing method of the embodiment of the present application. The computer program can be recorded on a computer-readable recording medium, for example, and loaded into the above-mentioned computing device through the computer-readable recording medium and run therein.

[0183] Based on the same inventive concept, the principles and beneficial effects of solving the problems by the video processing device provided in the embodiment of the present application are similar to the principles and beneficial effects of solving the problems by the video processing method in the method embodiment of the present application. Please refer to the principles and beneficial effects of the implementation of the method. For the sake of concise description, they will not be repeated here.

[0184] See also Figure 6 , Figure 6 A schematic diagram of the structure of a computer device provided in an embodiment of the present application is shown in FIG. Figure 6 As shown, the computer device at least includes a processor 601, a communication interface 602 and a memory 603. Among them, the processor 601, the communication interface 602 and the memory 603 can be connected via a bus or other means. Among them, the processor 601 (or the central processing unit (CPU)) is the computing core and control core of the terminal, which can parse various instructions in the terminal and process various data of the terminal. For example, the CPU can be used to parse the power on and off instructions sent by the user to the terminal, and control the terminal to perform power on and off operations; for example, the CPU can transmit various interactive data between the internal structures of the terminal, and so on. The communication interface 602 can optionally include a standard wired interface, a wireless interface (such as WI-FI, a mobile communication interface, etc.), which can be used to send and receive data under the control of the processor 601; the communication interface 602 can also be used for the transmission and interaction of internal data of the terminal. The memory 603 (Memory) is a memory device in the terminal for storing programs and data. It can be understood that the memory 603 here can include both the built-in memory of the terminal and the extended memory supported by the terminal. The memory 603 provides a storage space, which stores the operating system of the terminal, including but not limited to: Android system, iOS system, Windows Phone system, etc., which is not limited in this application.

[0185] The embodiment of the present application also provides a computer-readable storage medium (Memory), which is a memory device in the terminal for storing programs and data. It is understandable that the computer-readable storage medium here can include both the built-in storage medium in the terminal and the extended storage medium supported by the terminal. The computer-readable storage medium provides a storage space that stores the processing system of the terminal. In addition, one or more instructions suitable for being loaded and executed by the processor 601 are also stored in the storage space, and these instructions can be one or more computer programs (including program codes). It should be noted that the computer-readable storage medium here can be a high-speed RAM memory, or a non-volatile memory, such as at least one disk storage; optionally, it can also be at least one computer-readable storage medium located away from the aforementioned processor.

[0186] In one embodiment, the computer device may specifically be Figure 1 The processor 601 executes the following operations by running the executable program code in the memory 603:

[0187] Perform frame extraction on the video to be processed to obtain a candidate image set, where the candidate image set includes N video frame images of the video, where N is a positive integer;

[0188] Based on at least the aesthetic dimension and the target object attribute, respectively perform quality detection on the N video frame images to obtain a quality detection result of each video frame image;

[0189] Selecting M video frame images from the candidate image set according to the quality detection result, where M is an integer greater than 1;

[0190] The M video frame images are determined as cover images of the video.

[0191] As an optional embodiment, any one of the N video frame images is represented as the i-th video frame image, i is a positive integer, and i is less than or equal to N; the processor 601 performs quality detection on the N video frame images respectively, and a specific embodiment of obtaining the quality detection result of each video frame image is:

[0192] Obtaining the corresponding aesthetic scoring model under the aesthetic dimension through the communication interface 602;

[0193] Calling the aesthetic scoring model to perform quality detection on the i-th video frame image, and obtaining the first quality score of the i-th video frame image under the aesthetic dimension;

[0194] The first quality score is determined as the quality detection result of the i-th video frame image.

[0195] As an optional embodiment, the aesthetic scoring model includes a convolution layer, a fully connected layer and an activation layer; a plurality of aesthetic score levels are provided in the aesthetic scoring model; the processor 601 calls the aesthetic scoring model to perform quality detection on the i-th video frame image, and a specific embodiment of obtaining a first quality score of the i-th video frame image in the aesthetic dimension is:

[0196] Extract the features of the i-th video frame image through the convolution layer;

[0197] Call the fully connected layer to predict the probability of the i-th video frame image at each aesthetic score level based on the features of the i-th video frame image;

[0198] The activation layer is called to determine the aesthetic score level to which the i-th video frame image belongs according to the probability, and a first quality score of the i-th video frame image is determined according to the aesthetic score level.

[0199] As an optional embodiment, any one of the N video frame images is represented as the i-th video frame image, i is a positive integer, and i is less than or equal to N; the processor 601 performs quality detection on the N video frame images respectively, and a specific embodiment of obtaining the quality detection result of each video frame image is:

[0200] Detect whether there is a target object in the i-th video frame image;

[0201] If the target object does not exist, the second quality score of the i-th video frame image under the target object attribute is set to a preset score;

[0202] If there is a target object, then according to the attribute information of the target object, the second quality score of the i-th video frame image under the target object attribute is calculated;

[0203] The second quality score is determined as the quality detection result of the i-th video frame image.

[0204] As an optional embodiment, the attribute information of the target object includes the size and position of the target object; the processor 601 calculates the second quality score of the i-th video frame image under the target object attribute according to the attribute information of the target object as follows:

[0205] Determine a size score of the i-th video frame image according to a ratio between a size of the target object and a size of the i-th video frame image;

[0206] Determine the position score of the i-th video frame image according to the distance between the position of the target object and the target point of the i-th video frame image;

[0207] Based on the size score and the position score, a second quality score of the i-th video frame image under the target object attribute is calculated.

[0208] As an optional embodiment, the target object is a human face, and the attribute information of the target object further includes the expression of the target object; the processor 601 further performs the following operations by running the executable program code in the memory 603:

[0209] Determine the expression score of the i-th video frame image according to the category of the expression of the target object;

[0210] Based on the expression score, the second quality score of the i-th video frame image under the target object attribute is updated.

[0211] As an optional embodiment, if the first target object and the second target object exist in the i-th video frame image, the processor 601 calculates the second quality score of the i-th video frame image under the target object attribute according to the attribute information of the target object. The specific embodiment is:

[0212] Calculate, according to the attribute information of the first target object, a first candidate score of the i-th video frame image under the target object attribute;

[0213] Calculating a second candidate score of the i-th video frame image under the target object attribute according to the attribute information of the second target object;

[0214] The larger one of the first candidate score and the second candidate score is determined as the second quality score of the i-th video frame image under the target object attribute.

[0215] As an optional embodiment, any one of the N video frame images is represented as the i-th video frame image, where i is a positive integer and i is less than or equal to N;

[0216] The processor 601 performs quality detection on N video frame images respectively, and obtains a specific embodiment of the quality detection result of each video frame image as follows:

[0217] Obtaining a first quality score of the i-th video frame image under an aesthetic dimension, and obtaining a second quality score of the i-th video frame image under a target object attribute;

[0218] Performing a weighted summation on the first quality score and the second quality score to obtain a comprehensive quality score of the i-th video frame image;

[0219] The comprehensive quality score is determined as the quality detection result of the i-th video frame image.

[0220] As an optional embodiment, the quality detection result includes a quality score; and the processor 601 selects M video frame images from the candidate image set according to the quality detection result. A specific embodiment is as follows:

[0221] Sort the video frame images in the candidate image set in descending order of quality scores;

[0222] According to the sorting result, M video frame images are selected from the candidate image set.

[0223] As an optional embodiment, the specific embodiment of the processor 601 selecting M video frame images from the candidate image set according to the sorting result is:

[0224] Select the top M video frame images from the candidate image set; or,

[0225] According to the sorting result, P video frame images with quality scores greater than the score threshold are selected from the candidate image set; M video frame images with associated relationships are selected from the P candidate cover images, where P is an integer greater than or equal to M;

[0226] The association relationship between any two video frame images includes any of the following: having the same target object and belonging to the same subject.

[0227] As an optional embodiment, the processor 601 further performs the following operations by running the executable program code in the memory 603:

[0228] Output the cover image of the video;

[0229] The output method includes any of the following: displaying the cover images of the videos one by one, displaying the cover images of the videos in tiles, composing the cover images of the videos into an animation, or splicing and displaying them.

[0230] Based on the same inventive concept, the principles and beneficial effects of solving the problems by the computer device provided in the embodiment of the present application are similar to the principles and beneficial effects of solving the problems by the video processing method in the method embodiment of the present application. Please refer to the principles and beneficial effects of the implementation of the method. For the sake of concise description, they will not be repeated here.

[0231] An embodiment of the present application also provides a computer-readable storage medium, in which one or more instructions are stored, and the one or more instructions are suitable for being loaded by a processor and executing the video processing method of the above method embodiment.

[0232] The embodiment of the present application also provides a computer program product including instructions, which, when executed on a computer, enables the computer to execute the video processing method of the above method embodiment.

[0233] The embodiment of the present application also provides a computer program product or a computer program, which includes a computer instruction stored in a computer-readable storage medium. The processor of the computer device reads the computer instruction from the computer-readable storage medium, and the processor executes the computer instruction, so that the computer device performs the above-mentioned video processing method.

[0234] The steps in the method of the embodiment of the present application can be adjusted in order, combined and deleted according to actual needs.

[0235] The modules in the device of the embodiment of the present application can be merged, divided and deleted according to actual needs.

[0236] A person of ordinary skill in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing related hardware through a program, and the program can be stored in a computer-readable storage medium, and the readable storage medium can include: a flash drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0237] What is disclosed above is only a preferred embodiment of the present application, and it certainly cannot be used to limit the scope of rights of the present application. Ordinary technicians in this field can understand that all or part of the processes of implementing the above embodiment and equivalent changes made according to the claims of the present application are still within the scope covered by the application.

Claims

1. A video processing method, characterized in that: The method comprises: Performing frame extraction processing on the video to be processed to obtain a candidate image set, wherein the candidate image set includes N video frame images of the video, where N is a positive integer; wherein the frame extraction processing method includes: extracting key frames of the video; extracting any one or more video frame images from the video according to a preset number of video frame images; Obtaining a first quality score of an i-th video frame image among the N video frame images in an aesthetic dimension; Detect whether there is a target object in the i-th video frame image; i is a positive integer, and i is less than or equal to N; If the target object exists, the second quality score of the i-th video frame image under the target object attribute is calculated according to the attribute information of the target object; if the target object does not exist, the second quality score of the i-th video frame image under the target object attribute is set to a preset score; Performing a weighted summation on the first quality score and the second quality score to obtain a comprehensive quality score of the i-th video frame image; wherein the weight used in the weighted summation is set according to the type of the video; Determine the comprehensive quality score as the quality detection result of the i-th video frame image; According to the quality detection results of each video frame image, P video frame images whose comprehensive quality scores are greater than the score threshold are selected from the candidate image set, and M video frame images with associated relationships are selected from the P video frame images, where M is an integer greater than 1; the value of M is preset or specified by the service target object; the associated relationship includes the same subject; Determine the M video frame images as cover images of the video; Wherein, the attribute information of the target object includes the size and position of the target object; and calculating the second quality score of the i-th video frame image under the target object attribute according to the attribute information of the target object includes: determining the size score of the i-th video frame image according to the ratio between the size of the target object and the size of the i-th video frame image; determining the position score of the i-th video frame image according to the distance between the position of the target object and the target point of the i-th video frame image; and obtaining the second quality score of the i-th video frame image under the target object attribute by performing weighted summation based on the size score and the position score; wherein the target point includes at least one of the following: the center point in the i-th video frame image, and other core points in the i-th video frame image except the center point set according to requirements.

2. The method according to claim 1, characterized in that Any one of the N video frame images is represented as the i-th video frame image, where i is a positive integer and i is less than or equal to N; and performing quality detection on the N video frame images respectively to obtain a quality detection result of each video frame image includes: Obtain the corresponding aesthetic scoring model under the aesthetic dimension; Calling the aesthetic scoring model to perform quality detection on the i-th video frame image to obtain a first quality score of the i-th video frame image in an aesthetic dimension; The first quality score is determined as a quality detection result of the i-th video frame image.

3. The method according to claim 2, characterized in that The aesthetic scoring model includes a convolutional layer, a fully connected layer and an activation layer; the aesthetic scoring model is provided with multiple aesthetic score levels; The calling of the aesthetic scoring model to perform quality detection on the i-th video frame image to obtain a first quality score of the i-th video frame image in the aesthetic dimension includes: Extracting features of the i-th video frame image through the convolution layer; Calling the fully connected layer to predict the probability of the i-th video frame image at each aesthetic score level according to the features of the i-th video frame image; The activation layer is called to determine the aesthetic score level to which the i-th video frame image belongs according to the probability, and a first quality score of the i-th video frame image is determined according to the aesthetic score level.

4. The method according to claim 1, characterized in that The target object is a human face, and the attribute information of the target object also includes the expression of the target object; the method further includes: Determining an expression score of the i-th video frame image according to the category of the expression of the target object; Based on the expression score, a second quality score of the i-th video frame image under the target object attribute is updated.

5. The method according to claim 1, characterized in that If the first target object and the second target object exist in the i-th video frame image, then calculating the second quality score of the i-th video frame image under the target object attribute according to the attribute information of the target object includes: Calculating, according to the attribute information of the first target object, a first candidate score of the i-th video frame image under the target object attribute; Calculating, according to the attribute information of the second target object, a second candidate score of the i-th video frame image under the target object attribute; The larger one of the first candidate score and the second candidate score is determined as the second quality score of the i-th video frame image under the target object attribute.

6. The method according to any one of claims 1 to 5, characterized in that: The method further comprises: Outputting a cover image of the video; The output method includes any one of the following: displaying the cover images of the video one by one, displaying the cover images of the video in tiles, composing the cover images of the video into an animation, or splicing and displaying them.

7. A video processing device, characterized in that: The video processing device comprises: The acquisition unit is used to perform frame extraction processing on the video to be processed to obtain a candidate image set, wherein the candidate image set includes N video frame images of the video, where N is a positive integer; wherein the frame extraction processing method includes: extracting key frames of the video; extracting any one or more video frame images from the video according to a preset number of video frame images; A processing unit, configured to: obtain a first quality score of an i-th video frame image in the N video frame images under an aesthetic dimension; detect whether a target object exists in the i-th video frame image; i is a positive integer, and i is less than or equal to N; if a target object exists, calculate a second quality score of the i-th video frame image under the target object attribute according to the attribute information of the target object; if no target object exists, set the second quality score of the i-th video frame image under the target object attribute to a preset score; perform weighted summation of the first quality score and the second quality score to obtain the i-th video frame image; The comprehensive quality score of the image; wherein the weight used in the weighted summation is set according to the type of the video; the comprehensive quality score is determined as the quality detection result of the i-th video frame image; P video frame images with comprehensive quality scores greater than a score threshold are selected from the candidate image set according to the quality detection results of each video frame image, and M video frame images with an associated relationship are selected from the P video frame images, where M is an integer greater than 1; the value of M is preset or specified by the service target object; the associated relationship includes the same theme; the M video frame images are determined as the cover image of the video; Wherein, the attribute information of the target object includes the size and position of the target object; and calculating the second quality score of the i-th video frame image under the target object attribute according to the attribute information of the target object includes: determining the size score of the i-th video frame image according to the ratio between the size of the target object and the size of the i-th video frame image; determining the position score of the i-th video frame image according to the distance between the position of the target object and the target point of the i-th video frame image; and obtaining the second quality score of the i-th video frame image under the target object attribute by performing weighted summation based on the size score and the position score; wherein the target point includes at least one of the following: the center point in the i-th video frame image, and other core points in the i-th video frame image except the center point set according to requirements.

8. A computer device, characterized in that: include: storage devices and processors; a memory, wherein a computer program is stored in the memory; A processor, used to load the computer program to implement the video processing method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and the computer program is suitable for being loaded by a processor and executing the video processing method according to any one of claims 1 to 6.

10. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instruction is processed and executed, the steps of the video processing method according to any one of claims 1 to 6 are implemented.