Video clarity processing method, device and electronic equipment based on artificial intelligence

By extracting video frames and combining foreground and background clarity models, the clarity recognition of dynamic videos is solved, and the recognition accuracy and efficiency are improved.

CN111432206BActive Publication Date: 2024-11-26TENCENT TECH (BEIJING) CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202010334489.1
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2020-04-24
Publication Date
2024-11-26
Estimated Expiration
2040-04-24

AI Technical Summary

Technical Problem

The prior art has poor recognition effect on video clarity in dynamic scenarios, and the recognition speed based on the timing model is slow, resulting in low background processing efficiency.

Method used

By extracting multiple image frames in the video, the foreground and background are recognized in clarity, combining the foreground clarity model and background clarity model, the clarity of the video is determined, and equally spaced frame extraction and clustered filtering of fuzzy frames are used to improve recognition efficiency.

Benefits of technology

It realizes efficient and accurate clarity recognition of dynamic videos, improves recognition accuracy and processing efficiency, and is suitable for a variety of video categories and adapts to business scenario needs.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN111432206B_ABST
    Figure CN111432206B_ABST
Patent Text Reader

Abstract

The present invention provides a method, device and electronic device for processing video clarity based on artificial intelligence; the method comprises: extracting multiple image frames to be identified from a video; performing clarity identification on the foreground in the multiple image frames to obtain the foreground clarity of each of the image frames; determining the first video clarity of the video based on the foreground clarity of each of the image frames, and using it as the clarity identification result of the video; when the first video clarity of the video does not meet the clarity condition, performing clarity identification on the background of the multiple image frames to obtain the background clarity of each of the image frames; determining the second video clarity of the video based on the background clarity of each of the image frames, and using it as the updated clarity identification result of the video. Through the present invention, the clarity of a video can be efficiently and accurately identified.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence, and in particular to a video clarity processing method, device and electronic equipment based on artificial intelligence. Background Art

[0002] Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that machines have the functions of perception, reasoning and decision-making.

[0003] Computer vision technology, as an important technical content in artificial intelligence software technology, has developed rapidly in recent years. Image recognition technology is an important branch of computer vision technology. Image recognition technology can be used to identify image frames and give video clarity evaluation results. However, the scenes of these videos are mainly static scenes. In dynamic scenes, the recognition results of video clarity are not ideal. Summary of the invention

[0004] The embodiments of the present invention provide a video clarity processing method, device and electronic device based on artificial intelligence, which can efficiently and accurately identify the clarity of a video.

[0005] The technical solution of the embodiment of the present invention is achieved as follows:

[0006] The embodiment of the present invention provides a video definition processing method based on artificial intelligence, comprising:

[0007] Extracting multiple image frames to be recognized from the video;

[0008] Performing clarity recognition on the foreground in the plurality of image frames to obtain the foreground clarity of each of the image frames;

[0009] Determine a first video definition of the video based on the foreground definition of each of the image frames, and use the first video definition as a definition recognition result of the video;

[0010] When the first video definition of the video does not meet the definition condition, performing definition recognition on the backgrounds of the multiple image frames to obtain the background definition of each of the image frames;

[0011] Based on the background definition of each of the image frames, a second video definition of the video is determined and used as an updated definition recognition result of the video.

[0012] The embodiment of the present invention provides a video definition processing device based on artificial intelligence, comprising:

[0013] An extraction module, used for extracting a plurality of image frames to be identified from a video;

[0014] A first recognition module, configured to perform clarity recognition on the foreground in the plurality of image frames to obtain the foreground clarity of each of the image frames;

[0015] A first determination module, which determines a first video definition of the video based on the foreground definition of each of the image frames and uses the first video definition as a definition recognition result of the video;

[0016] A second recognition module is used for performing clarity recognition on the backgrounds of the multiple image frames to obtain the background clarity of each of the image frames when the first video clarity of the video does not meet the clarity condition;

[0017] The second determination module is used to determine the second video definition of the video based on the background definition of each of the image frames, and use it as an updated definition recognition result of the video.

[0018] In the above solution, the extraction module is used to:

[0019] Extracting frames from the video at equal intervals to obtain a first image frame set;

[0020] Clustering the image frames in the first image frame set to obtain a plurality of similar image frame subsets, randomly extracting an image frame from each of the similar image frame subsets, and combining with images in the first image set that are not clustered into any of the similar image frame subsets to form a second image frame set;

[0021] Image frames meeting the blur condition are filtered out from the second image frame set, and the remaining multiple image frames in the second image frame set are used as image frames to be identified.

[0022] In the above solution, the first identification module is used to:

[0023] The image features of the image frame are mapped into confidences corresponding to different foreground clarity categories, and the foreground clarity category corresponding to the maximum confidence is used as the foreground clarity of the image frame.

[0024] In the above solution, the first determining module is used to:

[0025] The foreground clarity categories include: clear foreground, general foreground and blurred foreground;

[0026] Based on the foreground definition category to which each of the image frames belongs, determining the number of the image frames to be identified included in each of the foreground definition categories;

[0027] The first video definition of the video is determined according to the proportion of the number of image frames included in each foreground definition category in the total number, wherein the total number is a count corresponding to the plurality of image frames to be identified.

[0028] In the above solution, the first determining module is used to:

[0029] The foreground clarity category of clear foreground corresponds to a first ratio threshold, the foreground clarity category of average foreground corresponds to a second ratio threshold, the foreground clarity category of blurred foreground corresponds to a third ratio threshold, and the second ratio threshold, the first ratio threshold and the third ratio threshold are arranged in descending order;

[0030] When the proportion of the number of image frames included in the foreground clarity category with clear foreground in the total number is greater than the first proportion threshold, determining that the first video clarity of the video is clear;

[0031] When the proportion of the number of image frames included in the foreground clarity category with a general foreground in the total number is greater than the second ratio threshold, and the proportion of the number of image frames included in the foreground clarity category with a blurred foreground in the total number is less than the third ratio threshold, it is determined that the first video clarity of the video is general;

[0032] When the proportion of the number of image frames included in the foreground clarity category with a clear foreground in the total number is less than the first proportion threshold, and the proportion of the number of image frames included in the foreground clarity category with a blurred foreground in the total number is zero, determining that the first video clarity of the video is fair;

[0033] When the proportion of the number of image frames included in the foreground definition category with blurred foreground in the total number is greater than a third proportion threshold, it is determined that the first video definition of the video is blurred.

[0034] In the above solution, the second identification module is used to:

[0035] The following processing is performed on each of the image frames:

[0036] Mapping the image features of the image frame into confidences of different background definition categories;

[0037] The background clarity categories include: clear background and blurred background.

[0038] In the above solution, the second determining module is used to:

[0039] Accumulating the confidence that each of the image frames belongs to the background clarity category with clear background and taking the average to obtain a mean confidence;

[0040] When the mean confidence is greater than the confidence threshold, the second video clarity of the video is determined to be blurred; when the mean confidence is less than or equal to the confidence threshold, the second video clarity of the video is determined to be average.

[0041] In the above solution, the second determining module is further used to:

[0042] Obtaining category information of the video;

[0043] In the correspondence between multiple video categories and confidence thresholds, the confidence threshold corresponding to the video category information is searched.

[0044] In the above scheme, the artificial intelligence-based video clarity processing device also includes: a recommendation module, which is used to: send the clarity recognition result of the video to the recommendation system, so that the recommendation system performs corresponding recommendation operations according to the clarity of the video.

[0045] An embodiment of the present invention further provides an electronic device, including:

[0046] A memory for storing executable instructions;

[0047] The processor is used to implement the video clarity processing method based on artificial intelligence provided by an embodiment of the present invention when executing the executable instructions stored in the memory.

[0048] An embodiment of the present invention provides a computer-readable storage medium storing executable instructions for implementing the artificial intelligence-based video clarity processing method provided by the embodiment of the present invention when executed by a processor.

[0049] The embodiments of the present invention have the following beneficial effects:

[0050] By extracting multiple image frames from a video, performing clarity recognition on the foreground of the multiple image frames to obtain a video clarity recognition result, and based on the judgment of the recognition result, performing clarity recognition on the background of the corresponding image frame to obtain an updated video clarity recognition result, the method can be applicable to dynamic videos to achieve efficient and accurate recognition, thereby improving the efficiency and accuracy of video clarity recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1It is a schematic diagram of the architecture of a video definition processing system based on artificial intelligence provided by an embodiment of the present invention;

[0052] Figure 2 is a schematic diagram of the architecture of a terminal device provided by an embodiment of the present invention;

[0053] Figure 3 is a schematic diagram of the architecture of a video definition processing device based on artificial intelligence provided by an embodiment of the present invention;

[0054] Figure 4A It is a flowchart of a video definition processing method based on artificial intelligence provided by an embodiment of the present invention;

[0055] Figure 4B It is a flow chart of a video definition processing method based on artificial intelligence provided by an embodiment of the present invention;

[0056] Figure 4C It is a flow chart of a video definition processing method based on artificial intelligence provided by an embodiment of the present invention;

[0057] Figure 5 is a structural schematic diagram of a foreground clarity model provided by an embodiment of the present invention;

[0058] Figure 6 is a schematic diagram of the architecture of a recommendation system provided by an embodiment of the present invention;

[0059] Figure 7 A schematic diagram of two image frames extracted from a short video provided by an embodiment of the present invention;

[0060] Figure 8 It is a flow chart of a video definition processing method based on artificial intelligence provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0061] In order to make the purpose, technical solutions and advantages of the present invention clearer, the present invention will be further described in detail below in conjunction with the accompanying drawings. The described embodiments should not be regarded as limiting the present invention. All other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present invention.

[0062] In the following description, reference is made to “some embodiments”, which describe a subset of all possible embodiments, but it can be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0063] In the following description, the terms "first\second\third" involved are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first\second\third" can be interchanged with a specific order or sequence where permitted, so that the embodiments of the present invention described herein can be implemented in an order other than that illustrated or described herein.

[0064] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which the present invention belongs. The terms used herein are only for the purpose of describing the embodiments of the present invention and are not intended to limit the present invention.

[0065] The relevant data collection and processing in the embodiments of this application should be strictly in accordance with the requirements of relevant laws and regulations when applied in examples, obtain the informed consent or separate consent of the personal information subject, and carry out subsequent data use and processing within the scope of authorization of laws and regulations and the personal information subject.

[0066] Before further describing the embodiments of the present invention in detail, the nouns and terms involved in the embodiments of the present invention are described. The nouns and terms involved in the embodiments of the present invention are subject to the following interpretations.

[0067] 1) Video clarity: It is an important indicator to measure the quality of a video. Clarity refers to the clarity of each detail and its boundary on the image. Therefore, the image quality is compared by looking at the clarity of the replayed image. In this application, the video clarity identified by artificial intelligence is called the clarity recognition result.

[0068] 2) Convolutional Neural Networks (CNN): A type of feedforward neural network (FNN) that includes convolutional calculations and has a deep structure. It is one of the representative algorithms of deep learning. Convolutional neural networks have representation learning capabilities and can perform shift-invariant classification on input images according to their hierarchical structure.

[0069] 3) Foreground: The clarity of the foreground of a person or object in front of or near the front of the subject in the video is called foreground clarity.

[0070] 4) Background: the scenery behind the main subject and far away from the camera in the video, which is used to enrich the spatial content of the picture and reflect the characteristics of place, environment, season, time, etc. The emotional degree of the background is called background clarity.

[0071] In the related art, there are three methods for judging the clarity of a video: judging the clarity of a video based on the clarity of a target frame, judging the clarity of a video based on a 3D convolutional neural network (deep learning method), and judging the clarity of a video based on a time series model such as a 2D convolutional neural network (deep learning method) + a long short-term memory network (LSTM, Long Short-Term Memory). These methods are introduced below.

[0072] (1) Judging the clarity of the video based on the clarity of the target frame: extracting video frames at fixed time points, or filtering some excessive frames through some traditional operators to select the target frame of the video; after obtaining the target frame, use traditional operators (canny operator, sober operator, Laplace operator, etc.) to extract the gradient features of the target frame, calculate the weighted values ​​of these features, and compare the weighted values ​​with the pre-set threshold to obtain the clarity of the video.

[0073] (2) Determine the clarity of the video based on a 3D convolutional neural network (deep learning method): By building a commonly used 3D convolutional neural network model such as 3D-resnet, the labeled video data is put into the model for training, and finally the trained model is used to predict the video clarity.

[0074] (3) Determine the clarity of the video based on 2D convolutional neural network (deep learning method) + LSTM and other time series models: By building commonly used convolutional neural network models such as 2D-resnet, after obtaining the features of each frame, the features between frames are fused, and the video clarity is predicted based on the fused features.

[0075] In the embodiments of the present invention, it is found that the above method of the related technology may cause the following technical problems during actual application:

[0076] (1) Since videos have rich scenes and fast-changing content, especially some life scenes such as square dancing, street skateboarding and other categories of videos with high frequency, it is easy to extract motion frames during the frame extraction process. If the target frames obtained are all motion frames, the recognition results of the target frames cannot accurately represent the clarity of the entire video. Therefore, the recognition results of this method are completely dependent on the target frames obtained, and it is impossible to accurately identify the clarity of various categories of videos.

[0077] (2) The above methods (2) and (3) both take into account the continuity between frames. Since the background processing capabilities are limited in actual business scenarios, the recognition speed based on the timing model by these two methods is slow, resulting in slow real-time processing efficiency in the background.

[0078] In view of the above problems, the embodiments of the present invention provide a video clarity processing method, device and electronic device based on artificial intelligence, which can efficiently and accurately identify the clarity of the video.

[0079] The following describes an exemplary application of an electronic device provided by an embodiment of the present invention. The electronic device provided by an embodiment of the present invention can be a server, such as a server deployed in the cloud, which can extract multiple image frames from the video according to the remote video upload of the terminal device, perform clarity recognition on the foreground of the multiple image frames to obtain a video clarity recognition result, and perform clarity recognition on the background of the corresponding image frame according to the judgment of the recognition result to obtain an updated video clarity recognition result; it can also be a terminal device, such as a handheld terminal device, which can perform a series of processes for clarity recognition on the video input in the terminal device to obtain a video clarity recognition result. By running the video clarity processing solution based on artificial intelligence provided by the embodiment of the present invention, the electronic device can improve the accuracy of recognizing video clarity, enhance the applicability of video clarity processing in actual business scenarios, and improve the processing efficiency of electronic device recognition clarity, which is applicable to multiple application scenarios, such as the recommendation system can give priority to recommending videos with high clarity.

[0080] The following describes an exemplary application of the video definition processing method based on artificial intelligence provided by an embodiment of the present invention. Figure 1 , Figure 1 1 is a schematic diagram of the architecture of the video definition processing system 100 based on artificial intelligence provided by an embodiment of the present invention. The video definition processing system 100 based on artificial intelligence includes: a server 200, a network 300 and a terminal device 400 (terminal device 400-1 and terminal device 400-2 are shown as examples), and the terminal device 400 is connected to the server 200 through the network 300. The network 300 can be a wide area network or a local area network, or a combination of the two.

[0081] The terminal device 400 is used to obtain video samples. For example, when a user (such as Figure 1 User A and user B) in the video input interface input video (for example, select a local video file or shoot a video).

[0082] In some embodiments, the terminal device 400 locally executes the artificial intelligence-based video clarity processing method provided in the embodiment of the present invention to complete the video input by the user to obtain a video clarity recognition result. For example, on the terminal device 400, the user opens the video input interface and inputs the video in the video input interface. The terminal device 400 performs a series of clarity recognition processes on the video to obtain a video clarity recognition result, and displays the obtained video clarity recognition result on the video input interface 410 of the terminal 400 (video input interface 410-1 and video input interface 410-2 are shown as examples).

[0083] In some embodiments, the terminal device 400 may also send the video input by the user on the terminal device 400 to the server 200 through the network 300, and call the artificial intelligence-based video clarity processing function provided by the server 200. The server 200 performs a series of clarity recognition processing on the input video through the artificial intelligence-based video clarity processing method provided in an embodiment of the present invention to obtain a video clarity recognition result. For example, the user opens the video input interface on the terminal device 400 and inputs the video in the video input interface. The terminal device sends the video to the server 200 through the network 300. After receiving the video, the server 200 identifies the video clarity recognition result, returns the obtained video clarity recognition result to the terminal device, and displays the video clarity result of the video on the display interface 410 of the terminal device 400. Alternatively, the server 200 directly gives the video clarity result of the video.

[0084] The embodiments of the present invention can be widely used in video clarity processing scenarios. For example, in the background of a video APP, when the background system reviews the basic information of the video information (whether the video content is clear, etc.), a strategy is formulated in combination with the characteristics of the video itself, and multiple image frames are extracted from the video. The clarity of the foreground of the multiple image frames is identified to obtain the recognition result of the video clarity, and according to the judgment of the recognition result, the clarity of the background of the corresponding image frame is identified to obtain the updated video clarity recognition result, and the clarity of the video is efficiently and accurately identified, and finally the purpose of simulating human senses to give the clarity of the video is achieved, and the efficiency of real-time processing is accelerated; the video clarity processing system 100 based on artificial intelligence can also be applied to the recommendation system, and the obtained video clarity result is input into the recommendation system, so that the recommendation system recommends videos with higher clarity to users, so as to increase the video click-through rate and the user's viewing time. The obtained video clarity result can also be stored in the server, and then used offline by the recommendation system. In addition, the scenarios related to video clarity processing belong to the potential application scenarios of the invention.

[0085] The following description will be made by taking the electronic device as a terminal device as an example. Figure 2 , Figure 2 The terminal device 400 provided in the embodiment of the present invention (for example, it may be Figure 1 The schematic diagram of the architecture of the terminal device 400-1 and the terminal device 400-2 shown in FIG. Figure 2 The terminal device 400 shown includes: at least one processor 410, a memory 450, at least one network interface 420 and a user interface 430. The various components in the terminal 400 are coupled together via a bus system 440. It is understood that the bus system 440 is used to achieve connection and communication between these components. In addition to the data bus, the bus system 440 also includes a power bus, a control bus and a status signal bus. However, for the sake of clarity, the bus system 440 is not shown in FIG. Figure 2 Various buses are labeled as bus system 440 .

[0086] Processor 410 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., where the general-purpose processor can be a microprocessor or any conventional processor, etc.

[0087] The user interface 430 includes one or more output devices 431 that enable presentation of media content, including one or more speakers and / or one or more visual display screens. The user interface 430 also includes one or more input devices 432, including user interface components that facilitate user input, such as a keyboard, mouse, microphone, touch screen display, camera, other input buttons and controls.

[0088] The memory 450 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state memory, hard disk drives, optical disk drives, etc. The memory 450 may optionally include one or more storage devices that are physically remote from the processor 410.

[0089] The memory 450 includes a volatile memory or a nonvolatile memory, and may also include both volatile and nonvolatile memories. The nonvolatile memory may be a read-only memory (ROM), and the volatile memory may be a random access memory (RAM). The memory 450 described in the embodiment of the present invention is intended to include any suitable type of memory.

[0090] In some embodiments, memory 450 can store data to support various operations, examples of which include programs, modules, and data structures, or a subset or superset thereof, as exemplarily described below.

[0091] Operating system 451, including system programs for processing various basic system services and performing hardware-related tasks, such as a framework layer, a core library layer, a driver layer, etc., for implementing various basic services and processing hardware-based tasks;

[0092] A network communication module 452, for reaching other computing devices via one or more (wired or wireless) network interfaces 420, exemplary network interfaces 420 include: Bluetooth, Wireless Compatibility Authentication (WiFi), and Universal Serial Bus (USB);

[0093] a presentation module 453 for enabling presentation of information via one or more output devices 431 (e.g., display screen, speaker, etc.) associated with the user interface 430 (e.g., a user interface for operating peripherals and displaying content and information);

[0094] The input processing module 454 is used to detect one or more user inputs or interactions from one of the one or more input devices 432 and translate the detected inputs or interactions.

[0095] In some embodiments, the video definition processing device based on artificial intelligence provided by the embodiments of the present invention can be implemented in software. Figure 2 The video definition processing device 455 based on artificial intelligence stored in the memory 450 is shown, which can be software in the form of programs and plug-ins, including the following software modules: extraction module 4551, first recognition module 4552, first determination module 4553, second recognition module 4554, second determination module 4555 and recommendation module 4556; wherein, the extraction module 4551, the first recognition module 4552, the first determination module 4553, the second recognition module 4554 and the second determination module 4555 are used to implement the video definition processing method based on artificial intelligence provided by the embodiment of the present invention, and the recommendation module 4556 is used to implement the recommendation of the video definition recognition result of the embodiment of the present invention, and these modules are logical, so they can be arbitrarily combined or further split based on the functions implemented. The functions of each module will be explained below.

[0096] The video definition processing method based on artificial intelligence provided by the embodiment of the present invention can be executed by the above-mentioned server, or by a terminal device (for example, it can be Figure 1 The process may be executed by the terminal device 400-1 and the terminal device 400-2 as shown, or may be executed jointly by the server and the terminal device.

[0097] The artificial intelligence-based video clarity processing method provided by the embodiment of the present invention will be described below in conjunction with the exemplary application and implementation of the terminal device provided by the embodiment of the present invention.

[0098] See also Figure 3 and Figure 4A , Figure 3 4 is a schematic diagram of the architecture of the video definition processing device 455 based on artificial intelligence provided by an embodiment of the present invention, showing the process of implementing video definition processing through a series of modules. Figure 4A is a flow chart of a video definition processing method based on artificial intelligence provided by an embodiment of the present invention, which will be combined with Figure 3 right Figure 4A The steps shown are explained.

[0099] In step S101 , the server extracts a plurality of image frames to be identified from a video.

[0100] The user can input the video on the input interface of the terminal. When the input is completed, the terminal can forward the video to the server. After receiving the video, the server can extract multiple image frames to be identified from the video, so as to obtain the clarity recognition result of the video based on the image frames.

[0101] In some embodiments, see Figure 3 The server extracts multiple image frames to be identified from the video, including: extracting frames from the video at equal intervals to obtain a first image frame set; clustering the image frames in the first image frame set to obtain multiple similar image frame subsets, randomly extracting an image frame from each similar image frame subset, and combining the images in the first image set that are not clustered into any similar image frame subset to form a second image frame set; filtering out image frames that meet the fuzzy condition from the second image frame set, and using the remaining multiple image frames in the second image frame set as image frames to be identified.

[0102] As an example, the server extracts frames from the video at equal intervals to obtain a first set of image frames, which can be implemented through a multimedia video processing tool (FFMpeg, Fast Forward Mpeg). That is, after the server receives the video, it reads the stream information in the video file, calls the corresponding decoder in the FFMpeg decoding library to open the stream information, and decodes multiple video frames from the video by setting the number of frames extracted per second to obtain the first set of image frames.

[0103] As an example, the server implements the filtering process of the image frames through clustering processing. The following specifically describes the process of clustering the image frames: projecting the first image frame set into the feature space to obtain the image feature vector corresponding to each first image frame; calculating the distance (Euclidean distance or cosine distance) between each feature vector and other feature vectors; classifying the feature vectors whose calculated distance falls within the numerical threshold range into similar image frame categories to obtain multiple similar image frame subsets, and taking the feature vectors that are not clustered into any similar image frame subset as a new similar image frame category; randomly extracting an image frame from each similar image frame subset as a similar image frame category; and combining the image frames of all similar image frame categories together to form a second image frame set.

[0104] The above-mentioned method of extracting image frames is applicable to various types of videos and can filter out blurry frames caused by the extraction method, so that the subsequent video definition processing process can be accurately performed.

[0105] In step S102, the server performs clarity recognition on the foreground in multiple image frames to obtain the foreground clarity of each image frame.

[0106] In some embodiments, see Figure 3 The server identifies the clarity of the foreground in multiple image frames to obtain the foreground clarity of each image frame, including: performing the following processing on each image frame based on the foreground clarity model: through the forward propagation process between the various layers of the foreground clarity model, the image features of the image frame are mapped to confidences corresponding to different foreground clarity categories, and the foreground clarity category corresponding to the maximum confidence is used as the foreground clarity of the image frame.

[0107] As an example, see Figure 3 The foreground clarity model includes an input layer, a hidden layer, and an output layer. The server passes the image features of each image frame through the forward propagation process among the input layer, hidden layer, and output layer of the foreground clarity model, and outputs the confidence of the foreground clarity category to which each image frame belongs. The foreground clarity category corresponding to the maximum confidence is used as the foreground clarity of the image frame. The foreground clarity categories include: foreground clear, foreground general, and foreground blurred.

[0108] For example, the foreground clarity of an image frame is divided into three categories: foreground blur, foreground general, and foreground clear. For an image frame, the output result is: foreground blur 2%, foreground general 7%, foreground clear 91%, then the foreground clarity of the image frame is clear. It is worth noting that the closer the confidence probability of belonging to the correct category is to 1, the better the prediction effect.

[0109] Here, the forward propagation process of the foreground clarity model is explained. The process of propagating sample data from low level to high level is the forward propagation process of the foreground clarity model. In the forward propagation process, the image is input into the input layer, and after the image features are extracted by the hidden layer, it enters the output layer for classification to obtain the result of the foreground clarity category. When the output result matches the expected value, the result is output.

[0110] In step S103, the server determines a first video definition of the video based on the foreground definition of each image frame, and uses the first video definition as a definition recognition result of the video.

[0111] See also Figure 4B , Figure 4B is a flow chart of a video definition processing method based on artificial intelligence provided by an embodiment of the present invention. In some embodiments, Figure 4B Show Figure 4A Step S103 can be Figure 4B The steps S1031-S1032 shown are implemented.

[0112] In step S1031, the server determines the number of image frames to be identified included in each foreground clarity category based on the foreground clarity category to which each image frame belongs; in step S1032, the server determines the first video clarity of the video according to the proportion of the number of image frames included in each foreground clarity category in the total number, wherein the total number is a count of the corresponding multiple image frames to be identified.

[0113] In some embodiments, the foreground clarity category with clear foreground corresponds to a first ratio threshold, the foreground clarity category with average foreground corresponds to a second ratio threshold, and the foreground clarity category with blurred foreground corresponds to a third ratio threshold, and the second ratio threshold, the first ratio threshold and the third ratio threshold are arranged in descending order. The first video clarity of the video is determined according to the ratio of the number of image frames included in each foreground clarity category to the total number, which can be identified by corresponding conditions, as exemplified below.

[0114] Condition 1) When the proportion of the number of image frames included in the foreground clarity category with clear foreground in the total number is greater than the first ratio threshold, the first video clarity of the video is determined to be clear; when the proportion of the number of image frames included in the foreground clarity category with general foreground in the total number is greater than the second ratio threshold, and the proportion of the number of image frames included in the foreground clarity category with blurred foreground in the total number is less than the third ratio threshold, the first video clarity of the video is determined to be general.

[0115] Condition 2) When the proportion of the number of image frames included in the foreground clarity category with clear foreground in the total number is less than the first proportion threshold, and the proportion of the number of image frames included in the foreground clarity category with blurred foreground in the total number is zero, it is determined that the first video clarity of the video is general.

[0116] Condition 3) When the proportion of the number of image frames included in the foreground clarity category with blurred foreground in the total number is greater than a third proportion threshold, it is determined that the first video clarity of the video is blurred.

[0117] For example, assuming that the total number of counts of multiple image frames to be identified is m frames, where m is a natural number greater than zero, the server calls the foreground clarity model to identify the clarity of the foreground in the multiple image frames, and obtains that the number of image frames included in the foreground clarity category with clear foreground is a, the number of image frames included in the foreground clarity category with average foreground is b, and the number of image frames included in the foreground clarity category with blurred foreground is c; where a, b, and c are all natural numbers greater than zero. It can be divided into the following cases:

[0118] Case 1) If When , the result of judging the clarity of the first video is clear;

[0119] Case 2) If and If , the first video definition of the video is judged to be average; if and When , the first video definition of the video is average;

[0120] Case 3) If When , the first video clarity of the video is blurred.

[0121] As an example of step S1032, the server determines the first video clarity of the video according to the proportion of the number of image frames included in each foreground clarity category in the total number, and may also include: when the proportion of the number of image frames included in the foreground clarity category with clear foreground in the total number is greater than the first proportion threshold, determining that the first video clarity of the video is clear; when the proportion of the number of image frames included in the foreground clarity category with clear foreground in the total number is less than the first proportion threshold, and the proportion of the number of image frames included in the foreground clarity category with general foreground in the total number is greater than the second proportion threshold, determining that the first video clarity of the video is general; when the proportion of the number of image frames included in the foreground clarity category with blurred foreground in the total number is greater than the third proportion threshold, determining that the first video clarity of the video is blurred.

[0122] In step S104, when the first video definition of the video does not meet the definition condition, the server performs definition recognition on the backgrounds of multiple image frames to obtain the background definition of each image frame.

[0123] In some embodiments, the server performs clarity recognition on the backgrounds of multiple image frames to obtain the background clarity of each image frame, including: performing the following processing on each image frame: mapping the image features of the image frame into confidence levels of different background clarity categories; wherein the background clarity categories include: clear background and blurred background.

[0124] It should be noted that the background clarity model is a binary classification model, which outputs background clarity and its corresponding confidence and background blur and its corresponding confidence for each image frame. The application only uses the background blur confidence. As an example, the background clarity model can use the resnet-50 network.

[0125] It is worth noting that the first video clarity recognition result of the video may be a qualitative clarity category, such as clear, general, or fuzzy; or a quantitative clarity score, such as any score between 0 and 10.

[0126] As an example of step S104, when the first video definition of the video is a blurred image, the definition recognition is performed on the background of multiple image frames; or, when the first video definition of the video is an image with a score lower than the definition score threshold, the definition recognition is performed on the background of multiple image frames. For example, when the score of the first video definition of the video is 0-2 points, it is determined that the first video definition of the video does not meet the definition condition.

[0127] In step S105, based on the background definition of each image frame, a second video definition of the video is determined and used as an updated definition recognition result of the video.

[0128] See also Figure 4C , Figure 4C is a flow chart of a video definition processing method based on artificial intelligence provided by an embodiment of the present invention. In some embodiments, Figure 4C Show Figure 4A Step S105 can be Figure 4C The steps S1051-S1052 shown are implemented. In step S1051, the confidences of the background definition category of each image frame belonging to background blur are accumulated and averaged to obtain a mean confidence; in step S1052, when the mean confidence is greater than the confidence threshold, the second video definition of the video is determined to be blurred, and when the mean confidence is less than or equal to the confidence threshold, the second video definition of the video is determined to be general.

[0129] In some embodiments, see Figure 4C , Figure 4C is a flow chart of a video definition processing method based on artificial intelligence provided by an embodiment of the present invention, based on Figure 3 Before step S105, the following steps may also be performed:

[0130] In step S201, the category information of the video is obtained;

[0131] In step S202, the confidence threshold corresponding to the video category information is searched in the correspondence between the plurality of video categories and the confidence thresholds.

[0132] It is worth noting that the confidence threshold used may be different depending on the category information of the video. For example, for dancing videos, since the characters may be blurry but the background is clear, a slight blur does not affect the human perception; for close-up videos, a slight blur will aggravate the human perception of the video. Compared with close-up videos, the confidence threshold of dancing videos may be set higher. Therefore, the confidence threshold set for videos of different categories may be different. As an example, the correspondence between the video category information and the confidence threshold can be stored in the database of the server or terminal device for easy call by the server or terminal device.

[0133] The above-mentioned method of video clarity processing based on artificial intelligence can automatically select different recognition methods for different categories of videos, thereby realizing efficient and accurate recognition of various categories of videos. For clear and general videos, it is only necessary to call the foreground clarity model for clarity recognition, so as to obtain the clarity recognition results efficiently and accurately, thereby improving the recall and precision of clear videos and improving processing efficiency. For blurred videos, the background clarity model will be further called for clarity recognition to update the results of video clarity recognition. In this way, videos that are judged to be blurred due to the blurred foreground can be recalled, thereby improving the precision of blurred videos. Ultimately, the purpose of simulating human senses to recognize the clarity of videos can be achieved.

[0134] In some embodiments, see Figure 4B ,based on Figure 4A After step S105, the following steps may also be performed:

[0135] In step S106, the video definition recognition result is sent to the recommendation system, so that the recommendation system performs corresponding recommendation operations according to the video definition.

[0136] In some embodiments, see Figure 6 , Figure 6 FIG. 1 is a schematic diagram of the architecture of the recommendation system provided by an embodiment of the present invention. Figure 6In the example, the recommendation system includes a clarity module, a personalization module, a recall module, a sorting module, a diversity module, and a recommendation module based on diversity + clarity. Among them, the personalization module is used to calculate the object portrait according to the object behavior, so as to obtain interest preferences under different dimensions such as object attributes, historical behaviors, and interest content; the clarity module implements the clarity processing process of the above-mentioned video to obtain candidate videos with higher clarity, and the clarity recognition results obtained by the clarity module can also be saved locally and used directly; the recall module includes recall models of multiple channels such as collaborative filtering, topic models, content recall, and social network services (SNS, Social Network Software), to ensure the diversity of candidate videos during recall; the sorting module is used to uniformly score and sort the recall results, and select the videos that users are most interested in and have high clarity from the candidate videos, that is, select the best videos from the candidate videos to obtain videos that meet the diversity and clarity conditions. The above recommendation system takes into account multiple dimensions such as diversity, clarity, and personalization of the recommendation results, and can better meet the diverse needs of users.

[0137] The above-mentioned AI-based video clarity model can realize the background processing of the clarity of various videos, saving a lot of manpower costs. At the same time, the obtained video clarity results can also be applied to the recommendation system, in which videos with higher clarity are recommended to users to increase the video click-through rate and user viewing time; the video clarity results can also be stored in the server for subsequent offline use by the recommendation system.

[0138] The following describes an exemplary application of an embodiment of the present invention in a practical application scenario.

[0139] During the implementation of the embodiments of the present invention, it was found that the relevant technology has the following problems: Taking the application scenario of judging the clarity of short videos as an example, with the continuous development of mobile Internet and the rapid rise of mobile platforms such as smart phones, short videos based on smart phones / tablets have become a new form of content dissemination in recent years. With the explosive growth of short video data, it is crucial for the background system to quickly and accurately judge the clarity of short videos. However, due to the characteristics of short videos being many categories and most video frames being motion frames, the difficulty of judging the clarity of short videos is increased. If the video frame is a motion frame, the recognition result cannot accurately represent the clarity of the entire short video; however, the recognition of the fusion features between frames through a timing model will result in a slower recognition speed. Therefore, how to efficiently and accurately identify the clarity of a video is a problem solved by the present invention.

[0140] As an example, see Figure 7 , Figure 7A schematic diagram of two image frames extracted from a short video provided by an embodiment of the present invention, Figure 7 Foreground 301, foreground 302, background 303, and background 304 are shown. It can be seen that in the prior art, since the foregrounds of the two image frames shown contain phantoms, the recognition results of the models in the prior art are all blurred, but it can be seen that the backgrounds of the two image frames are relatively clear, and the clarity of the short videos containing the two image frames should be judged to be average based on human senses. For example, videos of categories such as dancing and sports: the characters in the video frames may be phantoms, but the background is relatively clear; long-range videos: the main body in a single video frame is too small to be clearly distinguished; these categories of videos have a good overall look and feel, and the clarity of such short videos should be judged to be average based on human senses. In other words, the clarity recognition method in the prior art is not suitable for various business scenarios.

[0141] In response to the above problems, an embodiment of the present invention proposes an artificial intelligence-based video clarity processing method. The artificial intelligence-based video clarity processing method provided by the embodiment of the present invention can not only provide video clarity processing that is more suitable for business scenarios in combination with business scenarios, but also has a faster processing efficiency, effectively improving the efficiency and accuracy of video clarity processing.

[0142] The artificial intelligence-based video clarity processing and detection method provided in an embodiment of the present invention extracts multiple image frames from a video, performs clarity recognition on the foreground of the multiple image frames to obtain a video clarity recognition result, and based on the judgment of the recognition result, performs clarity recognition on the background of the corresponding image frames to obtain an updated video clarity recognition result.

[0143] See also Figure 8 , Figure 8 : is a flow chart of a video definition processing method based on artificial intelligence provided by an embodiment of the present invention. The implementation scheme of the embodiment of the present invention is as follows:

[0144] The multimedia video processing tool (FFMpeg) is used to extract k frames from the video at equal intervals, and then the video frames are clustered according to features such as color histogram and canny operator to filter out duplicate frames. At the same time, the frames are initially screened to mainly filter out frames that are too blurred. Finally, m frames are selected from the k frames, where k and m are fixed constants.

[0145] The short video uploaded by the user is extracted, and each of the m frames is normalized and sent to the foreground clarity model to determine the clarity of the foreground content of each frame. The foreground clarity model supports pictures of any size as input; the output is three categories: foreground clear, foreground general, foreground blurred, and the corresponding confidence.

[0146] In some examples, foreground clarity categories are identified by corresponding conditions:

[0147] Condition 1) The above frames are clear, and the short video is clear;

[0148] Condition 2) The following frames are clear, and the remaining frames are fair, so the short video is fair;

[0149] Condition 3) The above frames are normal. If the following frames are blurred, then the short video is average;

[0150] Condition 4) If The above frame is blurred and input to the background clarity model.

[0151] If the foreground clarity model determines that the video is blurred, the category information of the video is obtained, and the clarity of the short video is given based on the confidence of the m-frame results given by the background clarity model and the video category information.

[0152] As an example, assume that the output result of the background clarity model is (cof_1, cof_2), cof_1 represents the confidence of the output background blur, and cof_2 represents the confidence of the output background not blur, where cof_1+cof_2=1, and the cof_1 of m frames is added. Since the blur range may be different for videos of different categories, the threshold of cof_1 is given based on different categories. If cof_1_avg>thre of m frames, the short video is given a blurred label, otherwise, a general label is given, where thre is the confidence threshold corresponding to the video category.

[0153] In some examples, the background clarity model is mainly used to determine the background clarity of transition frames (such as motion frames, blurred subjects, and clear backgrounds). This model is a binary classification model that serves as an auxiliary judgment of the foreground clarity model to determine whether the background of the image is clear. The network mainly used by the background clarity model is resnet-50.

[0154] For some examples, see Figure 5 , Figure 5It is a structural diagram of the foreground clarity model provided by an embodiment of the present invention. As an example, the backbone network of the foreground clarity model is mainly composed of a convolution layer, a pooling layer, a residual module, a downsampling layer, an adaptive downsampling layer, a random inactivation layer (dropout) and a fully connected layer; the residual module mainly selects some convolution layers with 5*5, 3*3 and 1*1 convolution kernels, and then undergoes a direct connection operation; the downsampling layer mainly uses a convolution layer or a pooling layer with a step size of 2 to downsample the image; the adaptive downsampling layer can convert feature maps of any scale with the same number of channels into feature vectors of the same dimension, which can allow the convolutional neural network model to use images of any scale as the input of the model.

[0155] In some examples, a framework for foreground clarity models is described, see Figure 5 ,based on Figure 5 , Figure 5 It is a structural diagram of the foreground clarity model provided by an embodiment of the present invention. As an example, the input layer of the foreground clarity model normalizes the input image; the categories of hidden layers of the foreground clarity module may include: convolution layer, pooling layer, residual module, downsampling layer, adaptive pooling layer, random inactivation layer, and fully connected layer.

[0156] Convolution layer: Perform convolution linear mapping on the input image to extract image features. It should be noted that the input image can extract certain features of the image through mathematical operations with the convolution kernel. Different convolution kernels will extract different image features. Therefore, for the training of the foreground clarity model, the complexity of the model can be reduced by extracting the best performing image features, saving a lot of computing resources and time.

[0157] The pooling layer can be either mean pooling or maximum pooling to obtain the main image features. Mean pooling is to take the average value of the values ​​in the pooling area, while maximum pooling is to divide the feature map into several rectangular areas and take the maximum value of each area. After the pooling operation, unimportant image features in the feature map of the convolutional layer are removed, and the number of parameters is reduced to reduce overfitting.

[0158] The downsampling layer is a nonlinear downsampling method that can perform four serial downsampling processes and output the features extracted by each downsampling in parallel to extract four groups of feature maps of different sizes. A residual module is added before the downsampling process. It should be noted that the direct connection operation in the residual module plays a direct transmission role through a simple identity mapping, retaining the spatial structure of the gradient and alleviating the problem of gradient fragmentation in the model.

[0159] The adaptive pooling layer converts the 4 sets of feature maps of different sizes output by the downsampling layer into 4 sets of feature maps of the same size, and integrates the 4 sets of feature maps of the same size into 1 set of feature maps through connection processing. It should be noted that the connection processing is to add the 4 sets of feature maps in pairs through addition operations and output all the added feature maps; the adaptive downsampling layer automatically calculates the size of the convolution kernel and the step size of each movement according to the set input image and output image sizes to output the set output image size, that is, the adaptive downsampling layer can convert feature maps of any size with the same number of channels into feature vectors of the same dimension, so that the foreground clarity model supports processing images of any size.

[0160] The fully connected layer integrates all the features obtained before convolution into an N-dimensional feature vector. Between the two fully connected layers, a random dropout layer is used to discard some neuron nodes with a certain probability to weaken the joint adaptability between the neuron nodes. For example, the dropout rate can be 50%, which is used to discard half of the neuron nodes.

[0161] The output layer uses the logistic regression softmax function to classify the N-dimensional feature vector to output the clarity category of each frame image and the confidence corresponding to each clarity category, where N is a natural number greater than zero.

[0162] In some examples, before using the model, based on the nature of the video itself and the needs of the business side, quantitative standards are established for the three clarity categories of short videos: clear, general, and blurred, and the training samples are labeled.

[0163] The above AI video clarity model can directly process the clarity of short videos in the background, saving a lot of manpower costs. At the same time, the obtained results can also be applied to the recommendation system, in which short videos with higher clarity are recommended to users to increase click-through rate and user viewing time.

[0164] The following is a description of an exemplary structure of the video definition processing device 455 based on artificial intelligence provided by the embodiment of the present invention implemented as a software module. In some embodiments, Figure 2 As shown, the software modules stored in the artificial intelligence-based video definition processing device 455 of the memory 440 may include:

[0165] An extraction module 4551 is used to extract multiple image frames to be identified from a video; a first identification module 4552 is used to perform clarity identification on the foreground in the multiple image frames to obtain the foreground clarity of each image frame; a first determination module 4553 is used to determine the first video clarity of the video based on the foreground clarity of each image frame, and use it as the clarity identification result of the video; a second identification module 4554 is used to perform clarity identification on the background of the multiple image frames when the first video clarity of the video does not meet the clarity condition, and obtain the background clarity of each image frame; a second determination module 4555 is used to determine the second video clarity of the video based on the background clarity of each image frame, and use it as the updated clarity identification result of the video.

[0166] In the above scheme, the extraction module 4551 is used to: extract frames from the video at equal intervals to obtain a first image frame set; cluster the image frames in the first image frame set to obtain multiple similar image frame subsets, randomly extract an image frame from each similar image frame subset, and combine the images in the first image set that are not clustered into any similar image frame subset to form a second image frame set; filter out image frames that meet the blur condition from the second image frame set, and use the remaining multiple image frames in the second image frame set as image frames to be identified.

[0167] The first identification module 4552 is used to:

[0168] The image features of the image frame are mapped to confidences corresponding to different foreground clarity categories, and the foreground clarity category corresponding to the maximum confidence is used as the foreground clarity of the image frame.

[0169] The first determining module 4553 is used to:

[0170] Foreground clarity categories included: clear foreground, fair foreground, and blurred foreground;

[0171] Based on the foreground definition category to which each image frame belongs, determining the number of image frames to be identified included in each foreground definition category;

[0172] The first video definition of the video is determined according to the proportion of the number of image frames included in each foreground definition category in the total number, wherein the total number is a count of the corresponding multiple image frames to be identified.

[0173] The first determining module 4553 is used to:

[0174] The foreground clarity class with clear foreground corresponds to a first ratio threshold, the foreground clarity class with average foreground corresponds to a second ratio threshold, and the foreground clarity class with blurred foreground corresponds to a third ratio threshold, and the second ratio threshold, the first ratio threshold and the third ratio threshold are arranged in descending order;

[0175] When the proportion of the number of image frames included in the foreground clarity category with clear foreground in the total number is greater than a first proportion threshold, determining that the first video clarity of the video is clear;

[0176] When the proportion of the number of image frames included in the foreground clarity category with a general foreground in the total number is greater than a second proportion threshold, and the proportion of the number of image frames included in the foreground clarity category with a blurred foreground in the total number is less than a third proportion threshold, it is determined that the first video clarity of the video is general;

[0177] When the proportion of the number of image frames included in the foreground clarity category with a clear foreground in the total number is less than a first proportion threshold, and the proportion of the number of image frames included in the foreground clarity category with a blurred foreground in the total number is zero, determining that the first video clarity of the video is fair;

[0178] When the proportion of the number of image frames included in the foreground definition category with blurred foreground in the total number is greater than a third proportion threshold, it is determined that the first video definition of the video is blurred.

[0179] The second identification module 4554 is used to:

[0180] The following processing is performed on each image frame:

[0181] Mapping image features of the image frame into confidence scores for different background clarity categories;

[0182] Among them, the background clarity categories include: clear background and blurred background.

[0183] The second determining module 4555 is used to:

[0184] The confidence that each image frame belongs to the background clarity category with clear background is accumulated and averaged to obtain the mean confidence;

[0185] When the mean confidence is greater than the confidence threshold, it is determined that the second video clarity of the video is blurred; when the mean confidence is less than or equal to the confidence threshold, it is determined that the second video clarity of the video is average.

[0186] The second determining module 4555 is further used for:

[0187] Get the category information of the video;

[0188] In the correspondence between multiple video categories and confidence thresholds, the confidence threshold corresponding to the video category information is searched.

[0189] The recommendation module 4556 is used to send the video clarity recognition result to the recommendation system so that the recommendation system performs corresponding recommendation operations according to the video clarity.

[0190] An embodiment of the present invention provides a storage medium storing executable instructions, wherein the executable instructions are stored. When the executable instructions are executed by a processor, the processor will be caused to execute the method provided by the embodiment of the present invention, for example, Figure 4A , Figure 4B or Figure 4C The video clarity processing method based on artificial intelligence is shown.

[0191] In some embodiments, the storage medium may be a memory such as FRAM, ROM, PROM, EPROM, EE PROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or may be various devices including one or any combination of the above memories.

[0192] In some embodiments, executable instructions may be in the form of a program, software, software module, script or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine or other unit suitable for use in a computing environment.

[0193] As an example, executable instructions may, but need not, correspond to a file in a file system, may be stored as part of a file that stores other programs or data, such as in one or more scripts in a HyperText Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files storing one or more modules, subroutines, or code portions).

[0194] By way of example, executable instructions may be deployed to be executed on one computing device, or on multiple computing devices located at one site, or on multiple computing devices distributed across multiple sites and interconnected by a communication network.

[0195] In summary, through the embodiment of the present invention, multiple image frames are extracted from the video, and the clarity recognition is performed on the foreground of the multiple image frames to obtain the recognition result of the video clarity, and according to the judgment of the recognition result, the clarity recognition is performed on the background of the corresponding image frame to obtain the updated video clarity recognition result. For different categories of videos, different recognition methods can be automatically selected to efficiently and accurately recognize the clarity of short videos, so as to achieve the purpose of simulating human sensory recognition of video clarity; for clear and general videos, it is only necessary to call the foreground clarity model for clarity recognition, so that the clarity recognition result can be obtained efficiently and accurately, which improves the recall and precision of clear videos and improves processing efficiency; for blurred videos, the background clarity model is further called for clarity recognition to update the video clarity recognition result. In this way, those videos that are judged to be blurred due to the blurred foreground can be recalled, which improves the precision of blurred videos; the obtained video clarity result is input into the recommendation system, so that the recommendation system recommends videos with higher clarity to users, so as to increase the video click rate and user viewing time.

[0196] The above description is only an embodiment of the present invention and is not intended to limit the protection scope of the present invention. Any modification, equivalent replacement and improvement made within the spirit and scope of the present invention are included in the protection scope of the present invention.

Claims

1. A video definition processing method based on artificial intelligence, characterized in that: The method comprises: Extracting multiple image frames to be recognized from the video; Performing clarity recognition on the foreground in the plurality of image frames to obtain the foreground clarity of each of the image frames; the foreground clarity categories include: foreground clarity, foreground general, and foreground blur; Determine the first video definition of the video according to a ratio threshold and a ratio of the number of image frames included in each foreground definition category to the total number, wherein different foreground definition categories correspond to different ratio thresholds, and the total number is a count of the plurality of image frames to be identified; When the foreground clarity category of the video is the foreground clarity or the foreground general, using the first video clarity as the clarity recognition result of the video; When the foreground clarity category of the video is the foreground blur, performing clarity recognition on the backgrounds of the multiple image frames to obtain the background clarity of each of the image frames; Based on the background definition of each of the image frames, a second video definition of the video is determined and used as an updated definition recognition result of the video.

2. The method according to claim 1, characterized in that The step of extracting a plurality of image frames to be identified from the video comprises: Extracting frames from the video at equal intervals to obtain a first image frame set; Clustering the image frames in the first image frame set to obtain a plurality of similar image frame subsets, extracting an image frame from each of the similar image frame subsets, and combining images that are not clustered into any of the similar image frame subsets to form a second image frame set; Image frames meeting the blur condition are filtered out from the second image frame set, and the remaining multiple image frames in the second image frame set are used as image frames to be identified.

3. The method according to claim 1, characterized in that The step of identifying the clarity of the foreground in the plurality of image frames to obtain the foreground clarity of each image frame includes: The following processing is performed on each of the image frames: The image features of the image frame are mapped into confidences corresponding to different foreground clarity categories, and the foreground clarity category corresponding to the maximum confidence is used as the foreground clarity of the image frame.

4. The method according to claim 1, characterized in that: The foreground clarity category of clear foreground corresponds to a first ratio threshold, the foreground clarity category of average foreground corresponds to a second ratio threshold, the foreground clarity category of blurred foreground corresponds to a third ratio threshold, and the second ratio threshold, the first ratio threshold and the third ratio threshold are arranged in descending order; Determining the first video definition of the video according to the ratio threshold and the ratio of the number of image frames included in each foreground definition category to the total number includes: When the proportion of the number of image frames included in the foreground clarity category with clear foreground in the total number is greater than the first proportion threshold, determining that the first video clarity of the video is clear; When the proportion of the number of image frames included in the foreground clarity category with a general foreground in the total number is greater than the second ratio threshold, and the proportion of the number of image frames included in the foreground clarity category with a blurred foreground in the total number is less than the third ratio threshold, it is determined that the first video clarity of the video is general; When the proportion of the number of image frames included in the foreground clarity category with a clear foreground in the total number is less than the first proportion threshold, and the proportion of the number of image frames included in the foreground clarity category with a blurred foreground in the total number is zero, determining that the first video clarity of the video is fair; When the proportion of the number of image frames included in the foreground definition category with blurred foreground in the total number is greater than a third proportion threshold, it is determined that the first video definition of the video is blurred.

5. The method according to claim 1, characterized in that The step of performing clarity recognition on the backgrounds of the plurality of image frames to obtain the background clarity of each of the image frames includes: The following processing is performed on each of the image frames: Mapping the image features of the image frame into confidences of different background definition categories; The background clarity categories include: clear background and blurred background.

6. The method according to claim 5, characterized in that The determining the second video definition of the video based on the background definition of each of the image frames comprises: Accumulating the confidence that each of the image frames belongs to the background clarity category of background blur and taking the average to obtain a mean confidence; When the mean confidence is greater than the confidence threshold, the second video clarity of the video is determined to be blurred; when the mean confidence is less than or equal to the confidence threshold, the second video clarity of the video is determined to be average.

7. The method according to claim 6, characterized in that The method further comprises: Obtaining category information of the video; In the correspondence between multiple video categories and confidence thresholds, the confidence threshold corresponding to the video category information is searched.

8. The method according to any one of claims 1 to 7, characterized in that: The method further comprises: The definition recognition result of the video is sent to the recommendation system, so that the recommendation system performs a corresponding recommendation operation according to the definition of the video.

9. A video definition processing device based on artificial intelligence, characterized in that: include: An extraction module, used for extracting a plurality of image frames to be identified from a video; A first recognition module, configured to perform clarity recognition on the foreground in the plurality of image frames to obtain the foreground clarity of each of the image frames; The foreground clarity categories include: clear foreground, general foreground and blurred foreground; A first determination module determines a first video definition of the video according to a ratio threshold and a ratio of the number of image frames included in each foreground definition category to a total number, wherein different foreground definition categories correspond to different ratio thresholds, and the total number is a count of the plurality of image frames to be identified; when the foreground definition category of the video is the foreground clear or the foreground general, the first video definition is used as a definition identification result of the video; A second recognition module is used for, when the foreground clarity category of the video is the foreground blur, performing clarity recognition on the backgrounds of the multiple image frames to obtain the background clarity of each of the image frames; The second determination module is used to determine the second video definition of the video based on the background definition of each of the image frames, and use it as an updated definition recognition result of the video.

10. An electronic device, comprising: A memory for storing computer executable instructions; A processor, used to implement the artificial intelligence-based video clarity processing method described in any one of claims 1 to 8 when executing the computer executable instructions stored in the memory.

11. A computer-readable storage medium storing computer-executable instructions or a computer program, wherein the computer-executable instructions or the computer program, when executed by a processor, implements the video clarity processing method based on artificial intelligence as described in any one of claims 1 to 8.