Video processing method and device, computer equipment, computer readable storage medium and computer program product

Abstract

The invention provides a video processing method and device, computer equipment, a computer readable storage medium and a computer program product. The method comprises the steps of obtaining to-be-played video data in response to a received video playing instruction, and performing video playing based on the to-be-played video data; in a video playing process, in response to a received playing progress adjustment instruction, determining a first video frame corresponding to the playing progress adjustment instruction; when the first video frame is a recommended video frame, metadata corresponding to the first video frame is acquired, the metadata comprises quality scores of a plurality of candidate video frames, and the candidate video frames comprise the first video frame and a plurality of second video frames after the first video frame; and based on the quality scores of the plurality of candidate video frames, obtaining a target video frame from the plurality of candidate video frames, and continuing to play the video from the target video frame. Through the method, the exposure effect of the recommended video can be improved, so that the recommendation conversion rate is improved.

Description

Technical Field

[0001] The present application relates to video processing technology, and in particular to a video processing method, apparatus, device, computer-readable storage medium, and computer program product. Background Art

[0002] With the continuous development of internet technology, various video applications have sprung up like mushrooms after rain. Through these applications, people can watch a wide variety of movies, TV series, sports events, news and information anytime and anywhere. To balance the operating costs of video platforms and meet the promotion needs of product suppliers, inserting recommended videos before or during videos has become a widely used and effective operation method. However, when a user triggers a progress adjustment operation during the playback of a recommended video, the relevant technology will continue playback from the video frame corresponding to the progress adjustment operation. However, this video frame may be blurred and poorly correlated with the recommended video, thus failing to effectively convey the recommendation information, thereby reducing the exposure of the recommended video. Summary of the Invention

[0003] The embodiments of the present application provide a video processing method, apparatus, computer device, computer-readable storage medium, and computer program product, which can improve the exposure effect of recommended videos, thereby improving the recommendation conversion rate.

[0004] The technical solution of the embodiment of the present application is implemented as follows:

[0005] The present invention provides a video processing method, which includes:

[0006] In response to the received video play instruction, obtaining to-be-played video data, and playing the video based on the to-be-played video data, wherein the to-be-played video data includes the recommended video data;

[0007] During video playback, in response to receiving a playback progress adjustment instruction, determining a first video frame corresponding to the playback progress adjustment instruction;

[0008] When the first video frame is a recommended video frame, obtaining metadata corresponding to the first video frame from the recommended video data, the metadata including quality scores of multiple candidate video frames, the candidate video frames including the first video frame and multiple second video frames following the first video frame;

[0009] Based on the quality scores of the plurality of candidate video frames, a target video frame is obtained from the plurality of candidate video frames, and the video is played continuously from the target video frame. An embodiment of the present application provides a video processing method, the method comprising:

[0010] Obtaining a recommended video to be inserted, and determining a quality score corresponding to each recommended video frame in the recommended video;

[0011] For each of the recommended video frames, generating metadata of the recommended video frame based on the quality score of the recommended video frame and the quality scores of multiple recommended video frames following the recommended video frame;

[0012] generating recommended video data based on each of the recommended video frames and each of the recommended video frames;

[0013] The first video corresponding to the recommended video is obtained, and the recommended video data and the first video are spliced ​​and encoded to obtain video data to be played.

[0014] The present invention provides a video processing device, including:

[0015] a video playing module, configured to obtain video data to be played in response to a received video playing instruction, and play the video based on the video data to be played, wherein the video data to be played includes recommended video data;

[0016] A first determining module is configured to, in response to receiving a playback progress adjustment instruction during video playback, determine a first video frame corresponding to the playback progress adjustment instruction;

[0017] a first acquisition module configured to, when the first video frame is a recommended video frame, acquire metadata corresponding to the first video frame from the recommended video data, the metadata including quality scores of multiple candidate video frames, the candidate video frames including the first video frame and multiple second video frames following the first video frame;

[0018] The video playback module is further configured to determine a target video frame from the plurality of candidate video frames based on the quality scores of the plurality of candidate video frames, and continue playing the video starting from the target video frame.

[0019] The present invention provides a video processing device, including:

[0020] A score determination module, configured to obtain a recommended video to be inserted and determine a quality score corresponding to each recommended video frame in the recommended video;

[0021] a first generating module, configured to generate metadata of each recommended video frame based on the quality score of the recommended video frame and the quality scores of multiple recommended video frames following the recommended video frame;

[0022] a second generating module, configured to generate recommended video data based on each of the recommended video frames and each of the recommended video frames;

[0023] The splicing and encoding module is used to obtain the first video corresponding to the recommended video, and splice and encode the recommended video data and the first video to obtain video data to be played.

[0024] An embodiment of the present application provides a computer device, comprising:

[0025] a memory for storing computer-executable instructions or computer programs;

[0026] The processor is used to implement the video processing method provided in the embodiment of the present application when executing the computer-executable instructions or computer programs stored in the memory.

[0027] An embodiment of the present application provides a computer-readable storage medium storing computer-executable instructions or a computer program for implementing the video processing method provided in the embodiment of the present application when executed by a processor.

[0028] An embodiment of the present application provides a computer program product, including computer-executable instructions or a computer program. When the computer-executable instructions or the computer program are executed by a processor, the video processing method provided in the embodiment of the present application is implemented.

[0029] The embodiments of the present application have the following beneficial effects:

[0030] During the video playback process, in response to the playback progress adjustment instruction, the first video frame corresponding to the playback progress adjustment instruction is determined. If the first video frame is a recommended video frame, metadata corresponding to the recommended video frame is obtained. The metadata includes quality scores of multiple candidate video frames (including the first video frame and multiple second video frames after the first video frame). In this way, a high-quality target video frame within a certain length can be determined based on the metadata corresponding to the current first video frame without decoding subsequent video frames, thereby improving the real-time performance of video processing. Then, the target video frame is determined based on the quality scores of the candidate video frames, and the video continues to be played from the target video frame. This not only improves the picture presentation quality and exposure effect of the recommended video, but also improves the recommendation conversion rate. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 This is a schematic diagram of the structure of the video processing system architecture provided by an embodiment of the present application;

[0032] Figure 2A 4 is a schematic diagram of the structure of the terminal 400 provided in an embodiment of the present application;

[0033] Figure 2B 2 is a schematic diagram of the structure of the server 200 provided in an embodiment of the present application;

[0034] Figure 3 This is a schematic diagram of an implementation flow of the video processing method provided in an embodiment of the present application;

[0035] Figure 4 This is a schematic diagram of an implementation flow for determining a target video frame provided by an embodiment of the present application;

[0036] Figure 5 This is another implementation flow diagram of the video processing method provided in an embodiment of the present application;

[0037] Figure 6 This is a schematic diagram of an implementation flow for determining a quality score corresponding to a recommended video frame provided in an embodiment of the present application;

[0038] Figure 7A This is a schematic diagram of a first implementation flow of determining the clarity score of a recommended video frame provided in an embodiment of the present application;

[0039] Figure 7B This is a schematic diagram of a second implementation flow of determining the clarity score of a recommended video frame provided in an embodiment of the present application;

[0040] Figure 8A This is a schematic diagram of a first implementation flow for determining the content relevance score of a recommended video frame provided by an embodiment of the present application;

[0041] Figure 8B This is a schematic diagram of a second implementation flow for determining the content relevance score of a recommended video frame provided in an embodiment of the present application;

[0042] Figure 9A This is another implementation flow diagram of the video processing method provided in the embodiment of the present application;

[0043] Figure 9B This is another implementation flow diagram of the video processing method provided in the embodiment of the present application;

[0044] Figure 10A is a schematic diagram of a first video frame determined based on a progress adjustment operation provided by an embodiment of the present application;

[0045] Figure 10B This is a schematic diagram of a target video frame determined based on the video processing method provided in an embodiment of the present application.

[0046] It should be pointed out that the above-mentioned "first" and "second" are only used to distinguish different solutions, and do not represent the degree of distinction between the advantages and disadvantages of the solutions or the priority in the implementation process. DETAILED DESCRIPTION

[0047] In order to make the purpose, technical solutions and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limiting this application. All other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0048] In the following description, reference is made to “some embodiments”, which describes a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0049] In the following description, the terms "first\second\third" involved are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first\second\third" can be interchanged with a specific order or sequence where permitted, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0050] In the embodiments of the present application, the term "module" or "unit" refers to a computer program or a part of a computer program that has a predetermined function and works together with other related parts to achieve a predetermined goal, and can be implemented in whole or in part by using software, hardware (such as processing circuits or memories) or a combination thereof. Similarly, a processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be part of an overall module or unit that includes the function of the module or unit.

[0051] Unless otherwise defined, all technical and scientific terms used in the embodiments of the present application have the same meanings as those commonly understood by those skilled in the art. The terms used in the embodiments of the present application are only for the purpose of describing the embodiments of the present application and are not intended to limit the present application.

[0052] The relevant data collection and processing in the embodiments of this application should be strictly in accordance with the requirements of relevant laws and regulations when applied in examples, and the informed consent or separate consent of the personal information subject should be obtained. Subsequent data use and processing should be carried out within the scope of authorization of laws and regulations and the personal information subject.

[0053] Before further describing the embodiments of the present application in detail, the nouns and terms involved in the embodiments of the present application are explained. The nouns and terms involved in the embodiments of the present application are subject to the following interpretations.

[0054] Response: Used to indicate the conditions or states on which the executed operations depend. When the dependent conditions or states are met, one or more operations executed can be in real time or with a set delay. Unless otherwise specified, there is no restriction on the order in which the multiple operations executed can be executed.

[0055] In related technologies, when a user fast-forwards a recommended video clip, it is processed according to normal fast-forward logic and displayed as a frame corresponding to the fast-forward duration. This may result in a video frame with poor clarity, a dark picture, and difficult-to-recognize text, resulting in poor exposure of the recommended video.

[0056] The embodiments of the present application provide a video processing method, apparatus, computer device, computer-readable storage medium, and computer program product, which can improve the exposure effect of recommended videos, thereby improving the recommendation conversion rate. The following describes exemplary applications of the computer device provided in the embodiments of the present application. The computer device provided in the embodiments of the present application can be implemented as various types of terminals, such as laptops, tablet computers, desktop computers, set-top boxes, smartphones, smart speakers, smart watches, smart TVs, and in-vehicle terminals, and can also be implemented as a server. The following describes exemplary applications when the device is implemented as a terminal or server.

[0057] See also Figure 1 , Figure 1 This is a schematic diagram of the architecture of the video playback system 100 provided in an embodiment of the present application. To support a video playback application, the terminal 400 is connected to the server 200 via the network 300. The network 300 can be a wide area network or a local area network, or a combination of the two.

[0058] Terminal 400 is used to decode the video data received from server 200, thereby playing the video and displaying the video screen on a graphical interface. During the video playback process, in response to receiving a playback progress adjustment instruction, the first video frame corresponding to the playback progress adjustment instruction is determined. When the first video frame is a recommended video frame, metadata corresponding to the first video frame is obtained from the recommended video data, and the metadata includes quality scores of multiple candidate video frames. The candidate video frames include the first video frame and multiple second video frames after the first video frame; based on the quality scores of the multiple candidate video frames, a target video frame is obtained from the multiple candidate video frames, and the video is continued to be played from the target video frame. The picture quality of the target video frame is high, that is, the picture is clear and has a high correlation with the recommended content. This can improve the exposure effect of the recommended video, thereby improving the recommendation conversion rate.

[0059] In some embodiments, the server 200 may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers. It may also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms. The terminal and the server may be connected directly or indirectly via wired or wireless communication, which is not limited in the embodiments of the present application.

[0060] See also Figure 2A , Figure 2A is a schematic diagram of the structure of the terminal 400 provided in an embodiment of the present application, Figure 2A The terminal 400 shown includes: at least one processor 410, a memory 450, at least one network interface 420, and a user interface 430. The various components in the terminal 400 are coupled together via a bus system 440. It is understood that the bus system 440 is used to achieve connection and communication between these components. In addition to including a data bus, the bus system 440 also includes a power bus, a control bus, and a status signal bus. However, for the sake of clarity, the bus system 440 is not shown in FIG. Figure 2A Various buses are labeled as bus system 440 .

[0061] The processor 410 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., where the general-purpose processor can be a microprocessor or any conventional processor, etc.

[0062] The user interface 430 includes one or more output devices 431 that enable presentation of media content, including one or more speakers and / or one or more visual display screens. The user interface 430 also includes one or more input devices 432, including user interface components that facilitate user input, such as a keyboard, mouse, microphone, touch screen display, camera, other input buttons and controls.

[0063] The memory 450 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state memory, hard drives, optical drives, etc. The memory 450 may optionally include one or more storage devices that are physically remote from the processor 410.

[0064] The memory 450 includes volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory may be a read-only memory (ROM), and the volatile memory may be a random access memory (RAM). The memory 450 described in the embodiments of the present application is intended to include any suitable type of memory.

[0065] In some embodiments, the memory 450 can store data to support various operations, examples of which include programs, modules, and data structures, or a subset or superset thereof, as exemplified below.

[0066] Operating system 451, including system programs for processing various basic system services and performing hardware-related tasks, such as the framework layer, core library layer, and driver layer, which are used to implement various basic services and process hardware-based tasks;

[0067] A network communication module 452 for reaching other computer devices via one or more (wired or wireless) network interfaces 420 , exemplary network interfaces 420 including Bluetooth, Wireless LAN (WiFi), and Universal Serial Bus (USB);

[0068] a presentation module 453 for enabling presentation of information via one or more output devices 431 (e.g., a display screen, a speaker, etc.) associated with the user interface 430 (e.g., a user interface for operating peripheral devices and displaying content and information);

[0069] The input processing module 454 is configured to detect one or more user inputs or interactions from one of the one or more input devices 432 and to translate the detected inputs or interactions.

[0070] In some embodiments, the apparatus provided in the embodiments of the present application may be implemented in software. Figure 2A The video processing device 455 stored in the memory 450 is shown. This device can be software in the form of a program or plug-in, and includes the following software modules: a video playback module 4551, a first determination module 4552, and a first acquisition module 4553. These modules are logical and can be arbitrarily combined or further separated according to the functions they implement. The functions of each module will be described below.

[0071] In other embodiments, the apparatus provided in the embodiments of the present application may be implemented in hardware. As an example, the apparatus provided in the embodiments of the present application may be a processor in the form of a hardware decoding processor, which is programmed to execute the video processing method provided in the embodiments of the present application. For example, the processor in the form of a hardware decoding processor may be one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), or other electronic components.

[0072] See also Figure 2B , Figure 2B is a structural diagram of the server 200 provided in an embodiment of the present application, Figure 2B The server 200 shown includes: at least one processor 210, a memory 230 and at least one network interface 220. The various components in the server 200 are coupled together via a bus system 240. It is understood that the bus system 240 is used to achieve connection and communication between these components. In addition to including a data bus, the bus system 240 also includes a power bus, a control bus and a status signal bus. However, for the sake of clarity, the bus system 240 is not described in detail. Figure 2B Various buses are labeled as bus system 240 .

[0073] The processor 210 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., where the general-purpose processor can be a microprocessor or any conventional processor, etc.

[0074] The memory 230 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state memory, a hard drive, an optical drive, etc. The memory 230 may optionally include one or more storage devices that are physically remote from the processor 210.

[0075] The memory 230 includes volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory may be a read-only memory (ROM), and the volatile memory may be a random access memory (RAM). The memory 230 described in the embodiments of the present application is intended to include any suitable type of memory.

[0076] In some embodiments, memory 230 can store data to support various operations, examples of which include programs, modules, and data structures, or a subset or superset thereof, as exemplarily described below.

[0077] Operating system 231, including system programs for processing various basic system services and performing hardware-related tasks, such as the framework layer, core library layer, driver layer, etc., for implementing various basic services and processing hardware-based tasks;

[0078] The network communication module 232 is used to reach other computer devices via one or more (wired or wireless) network interfaces 220. Exemplary network interfaces 220 include Bluetooth, Wireless LAN (WiFi), and Universal Serial Bus (USB).

[0079] In some embodiments, the apparatus provided in the embodiments of the present application may be implemented in software. Figure 2B The video processing device 233 stored in the memory 230 is shown. This device can be software in the form of a program or plug-in, and includes the following software modules: a score determination module 2331, a first generation module 2332, a second generation module 2333, and a splicing encoding module 2334. These modules are logical and can be arbitrarily combined or further separated according to the functions they implement. The functions of each module will be described below.

[0080] In other embodiments, the apparatus provided in the embodiments of the present application may be implemented in hardware. As an example, the apparatus provided in the embodiments of the present application may be a processor in the form of a hardware decoding processor, which is programmed to execute the video processing method provided in the embodiments of the present application. For example, the processor in the form of a hardware decoding processor may be one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), or other electronic components.

[0081] In some embodiments, the terminal or server can implement the video processing method provided by the embodiment of the present application by running various computer executable instructions or computer programs. For example, computer executable instructions can be microprogram-level commands, machine instructions or software instructions. The computer program can be a native program or software module in the operating system; it can be a local (Native) application (APPlication, APP), that is, a program that needs to be installed in the operating system to run, such as a live broadcast APP or a video playback APP; it can also be a small program that can be embedded in any APP, that is, a program that can be run only by downloading it to a browser environment. In short, the above-mentioned computer executable instructions can be instructions in any form, and the above-mentioned computer program can be an application, module or plug-in in any form.

[0082] The video processing method provided in the embodiment of the present application will be described in conjunction with the exemplary application and implementation of the terminal provided in the embodiment of the present application.

[0083] The following describes the video processing method provided by the embodiment of the present application. As mentioned above, the computer device that implements the video processing method of the embodiment of the present application can be a terminal, a server, or a combination of the two. Therefore, the execution entity of each step will not be repeated below.

[0084] See also Figure 3 , Figure 3 This is a schematic diagram of an implementation flow of the video processing method provided in the embodiment of the present application, which will be combined with Figure 3 The steps shown are explained, Figure 3 The execution subject of the steps is described using a terminal as an example.

[0085] In step 101, in response to a received video play instruction, video data to be played is acquired, and video play is performed based on the video data to be played.

[0086] Among them, the video data to be played includes recommended video data. The video play instruction can be triggered by clicking or touching the video cover of the video to be played, or by receiving a play voice for the video to be played, or by receiving a gesture for the video to be played. In some embodiments, in response to receiving the video play instruction, the terminal sends a video play request to the server, and the video play request carries a video identifier. The server parses the video play request, obtains the video identifier, and searches for the video data to be played based on the video identifier, and then sends the video data to be played to the terminal in a streaming manner. After receiving the video data to be played, the terminal decodes the video data to be played to obtain multiple video frames, and then plays each video frame to achieve video playback.

[0087] In step 102, during video playback, in response to receiving a playback progress adjustment instruction, a first video frame corresponding to the playback progress adjustment instruction is determined.

[0088] During video playback, a playback progress bar can be displayed on the video playback interface. The playback progress bar includes a background bar, a progress fill bar, a drag slider and a time label, wherein the background bar is used to indicate the total length of the video, which is usually gray or translucent; the progress fill bar is used to indicate the progress of playback, which is usually more prominent than the color of the background bar, such as white; the drag slider is used to jump to a specified time point in response to the user's drag or click operation; the time label is used to display the current playback time (for example, 00:10) and the total length (for example, 01:02:30).

[0089] In some embodiments, the playback progress adjustment instruction can be triggered by a drag operation on the drag slider or by a click operation on the background bar. When a drag operation on the drag slider is received, the slider position is detected in real time based on the drag operation, and the fill bar and preview time label are dynamically updated. When the end of the drag operation is detected, the playback time corresponding to the end position of the drag operation is determined, and the video frame corresponding to the playback time is determined as the first video frame. When the playback progress adjustment instruction is triggered by a user clicking on a certain position of the progress bar, the proportion of the click position to the total length is calculated, converted into a target time, and the video frame corresponding to the target time is determined as the first video frame.

[0090] In step 103, when the first video frame is a recommended video frame, metadata corresponding to the first video frame is obtained from the recommended video data.

[0091] In some embodiments, the video data to be played includes recommended video data and main video data. The main video data is the video that the user actually wants to watch, such as a film and television drama video, a sports competition video, a game commentary video, etc. The recommended video data can be a product promotion video, where the product can be an actual commodity, or an online training course, a housekeeping service, etc. That is, before playing the main video or during playing the main video, the recommended video will be played. In an embodiment of the present application, in order to display a recommended video with higher picture quality when the user adjusts the video progress, the server generates metadata for each recommended video frame and embeds the metadata into the Supplemental Enhancement Information (SEI) unit of the video coding layer, thereby binding it to the frame data of the recommended video frame. The terminal can determine a target video frame with high clarity and high relevance to the recommended video content based on the metadata. Therefore, when the first video frame is a recommended video frame, the terminal obtains metadata from the SEI unit, wherein the metadata includes quality scores of multiple candidate video frames, and the quality scores of the candidate video frames are used to characterize the picture clarity of the candidate video frame and the relevance of the picture content to the recommended video. The candidate video frames include a first video frame and multiple second video frames after the first video frame. In some embodiments, the metadata also includes a video identifier for each candidate video frame. The video identifier can be a frame index of the candidate video frame or a timestamp. In order to reduce the storage space occupied by metadata, the video identifier in the embodiment of the present application is a frame index.

[0092] In step 104 , based on the quality scores of the multiple candidate video frames, a target video frame is obtained from the multiple candidate video frames, and the video is continuously played starting from the target video frame.

[0093] In some embodiments, the number of second video frames in the candidate video frames can be determined by the video frame rate F. For example, the number of second video frames can be obtained by subtracting one from the video frame rate F. That is, the metadata includes the quality scores of F candidate video frames. In this way, it can be ensured that the time difference between the target video frame determined from multiple candidate video frames and the first video frame is within 1 second, and the matching of the target video frame with the playback progress adjustment instruction can be ensured, thereby achieving the effect of imperceptible adjustment of the video frame.

[0094] In some embodiments, the quality score includes at least a clarity score and a content relevance score, see Figure 4 In step 104, "obtaining a target video frame from multiple candidate video frames based on the quality scores of the multiple candidate video frames" can be implemented by steps 1041 to 1043. Figure 4 Provide explanation.

[0095] In step 1041 , at least one third video frame that meets a definition condition is determined from a plurality of candidate video frames based on the definition scores.

[0096] In some embodiments, the clarity score of the candidate video frame is used to measure the clarity of the candidate video frame. The clarity score can be an integer value. For example, the clarity score can be an integer between 0 and 7, so that a clarity score occupies 3 bits when stored. The clarity score can also be an integer between 0 and 3. In this case, a clarity score occupies 2 bits when stored. In the embodiment of the present application, the clarity score is an integer between 0 and 15 as an example for explanation, that is, a clarity score occupies 4 bits. The clarity condition can be the highest clarity score, or it can be the clarity score being greater than the clarity threshold. In the embodiment of the present application, the clarity condition is explained as the highest clarity score. In this step, at least one third video frame with the highest clarity score is determined from multiple candidate video frames. For example, the highest clarity score of multiple candidate video frames is 14, then the candidate video frame with a clarity score of 13 is determined as the third video frame, assuming that there are 4 third video frames.

[0097] In step 1042 , based on the content relevance score, at least one fourth video frame that meets the relevance condition is determined from the at least one third video frame.

[0098] Here, similar to the clarity score, the content relevance score can also be an integer value. Similarly, the content relevance score can be an integer between 0 and 3, an integer between 0 and 7, or an integer between 0 and 15. The relevance condition can be that the content relevance score is the highest, or that the content relevance score is greater than the relevance threshold. The relevance threshold is pre-set, for example, it can be 13, 12, etc. In the embodiment of the present application, the relevance condition is that the content relevance score is the highest as an example for explanation. In this step, at least one fourth video frame with the highest content relevance score is determined from at least one third video frame. For example, if the highest content relevance score of the four third video frames is 15, then the third video frame with a clarity score of 15 is determined as the fourth video frame, assuming that there are three fourth video frames.

[0099] In step 1043 , a target video frame is determined from the at least one fourth video frame.

[0100] In some embodiments, if the number of fourth video frames is 1, the fourth video frame is directly determined as the target video frame. If the number of fourth video frames is greater than 1, a video frame identifier of each of the fourth video frames is obtained, and based on each video frame identifier, the target video frame is determined from multiple fourth video frames, wherein the fourth video frame corresponding to the smallest video frame identifier can be determined as the target video frame; or the fourth video frame corresponding to the largest video frame identifier can be determined as the target video frame, or the absolute value of the identifier difference between each of the video frame identifiers and the first identifier of the first video frame is determined, and the fourth video frame corresponding to the smallest absolute value of the identifier difference is determined as the target video frame. In some embodiments, a target video frame identifier can also be randomly determined from multiple video frame identifiers, and the fourth video frame corresponding to the target video frame identifier is determined as the target video frame.

[0101] For example, the video frame identifiers of the three fourth video frames are 120, 124, and 125, respectively. The fourth video frame corresponding to the minimum video frame identifier 120 can be determined as the target video frame, or the fourth video frame corresponding to the maximum video frame identifier 125 can be determined as the target video frame. Assuming that the first identifier of the first video frame is 123, then the absolute values ​​of the identifier differences between the three video frame identifiers and the first identifier are 3, 1, and 2, respectively, then the fourth video frame corresponding to the video frame identifier 124 is determined as the target video frame.

[0102] In the above steps 1041 to 1043, a third video frame that meets the clarity condition is first determined from multiple candidate video frames based on the clarity score, a fourth video frame is determined from the third video frame based on the content relevance score, and then a target video frame is finally determined from the fourth video frame based on the video frame identifier of the fourth video frame. A variety of methods for determining the target video frame are provided, thereby improving the flexibility and diversity of video frame selection. In addition, by using clarity as the first screening condition and content relevance as the second screening condition, the picture quality of the determined target video frame can be prioritized. In some embodiments, content relevance can also be used as the first screening condition and clarity as the second screening condition to determine the target video frame from multiple candidate video frames. In this way, it can be prioritized to ensure that the determined target video frame is strongly content-related to the recommended video, which can improve the recommendation exposure rate.

[0103] In some embodiments, if the quality score is a comprehensive score obtained by the clarity score and the content relevance score, then when step 104 is implemented, based on the quality score, at least one fifth video frame that meets the quality conditions can be determined from multiple candidate video frames, and the target video frame can be determined from the at least one fifth video frame.

[0104] Here, the quality score can be obtained by weighted summation of the clarity score and the content relevance score, and the quality score can be an integer or a floating point number. The quality condition can be the highest quality score, or the quality score can be greater than the quality threshold. In the embodiment of the present application, the quality condition is taken as an example of the highest quality score. At this time, at least one fifth video frame with the highest quality score is determined from the candidate video frames. The implementation process of determining the target video frame from at least one fifth video frame is similar to the implementation process of determining the target video frame from at least one fourth video frame. Based on the video frame identifier of the fifth video frame, the fifth video frame corresponding to the minimum video frame identifier or the maximum video frame identifier is determined as the target video frame, or the fifth video frame with the smallest absolute value of the difference between the video frame identifier and the first identifier is determined as the target video frame.

[0105] In some embodiments, if the first video frame is not a recommended video frame, that is, the first video frame is a video frame in the main video that the user wants to watch, then the video can continue to be played starting from the first video frame.

[0106] In the video processing method provided in the embodiment of the present application, during the process of playing the video, the terminal responds to the playback progress adjustment instruction and determines the first video frame corresponding to the playback progress adjustment instruction. If the first video frame is a recommended video frame, the metadata corresponding to the recommended video frame is obtained, and the metadata includes the quality scores of multiple candidate video frames (including the first video frame and multiple second video frames after the first video frame). In this way, a high-quality target video frame within a certain length can be determined based on the metadata corresponding to the current first video frame without decoding subsequent video frames, thereby improving the real-time performance of video processing. Then, the target video frame is determined based on the quality score of the candidate video frame, and the video continues to be played from the target video frame. Since the clarity score and content relevance score of the target video frame meet the corresponding screening conditions, it means that the picture display quality of the target video frame is high and is strongly related to the recommended video content. This not only improves the exposure effect of the recommended video, but also improves the recommendation conversion rate.

[0107] Based on the above embodiments, the present application further provides a video processing method. Figure 5 This is another implementation flow diagram of the video processing method provided in the embodiment of the present application, which will be combined with Figure 5 The steps shown are explained, Figure 5 The execution subject of the steps is described using a server as an example.

[0108] In step 201, a recommended video to be inserted is obtained, and a quality score corresponding to each recommended video frame in the recommended video is determined.

[0109] Here, the recommended video includes multiple recommended video frames. In this step, it is necessary to determine the quality score corresponding to each recommended video frame in the recommended video. The quality score is used to represent the clarity of the recommended video frame and the content relevance to the recommended video theme. The "determining the quality score corresponding to each recommended video frame in the recommended video" in step 201 can be achieved by Figure 6 Steps 2011 to 2013 shown in FIG. 2 are implemented by combining Figure 6 Provide explanation.

[0110] In step 2011 , the clarity score of each recommended video frame is determined.

[0111] Here, the clarity of the recommended video frame is an important indicator to measure the ability to present details and the sharpness of edges in the image, which directly affects the human eye's perception of image quality. It reflects the degree of retention of high-frequency information (such as texture, edges, and tiny details) in the image. The higher the clarity, the "sharper" the image and the richer the details; the lower the clarity, the "fuzzier" the image and the lost details. In an embodiment of the present application, the clarity score of the recommended video frame can be determined by the gradient of the pixel points in the recommended video frame, or it can be determined based on the spectrum diagram of the recommended video frame. Two implementation methods of the clarity score are described below.

[0112] See also Figure 7A Step 2011 can be implemented by the following steps 111A to 114A, and the following is combined Figure 7A accomplish.

[0113] In step 111A, the recommended video is segmented to obtain a plurality of recommended video segments.

[0114] Here, assuming that the duration of the recommended video is T seconds and the frame rate is T frames per second, the recommended video is segmented to obtain multiple recommended video segments. In some embodiments, the recommended video can be segmented according to a preset duration, which can be N seconds. Then, through the segmentation process, K recommended video segments can be obtained, where Indicates rounding x upwards. If T is an integer multiple of N, then the number of recommended video frames contained in each recommended video segment is N×F. For example, T is 30, N is 5, and F is 30. Then, the recommended video is segmented to obtain 6 recommended video segments, and the number of recommended video frames contained in each recommended video segment is 5×30=150 frames.

[0115] If T is not an integer multiple of N, then except for the last recommended video segment, the number of frames contained in each recommended video segment is N×F, and the number of frames contained in the last recommended video segment is Indicates rounding x down. For example, T is 33, N is 5, and F is 30. Then, the recommended video is segmented to obtain 7 recommended video segments. The number of recommended video frames contained in the first 6 recommended video segments is 5×30=150 frames, and the number of recommended video frames contained in the 7th recommended video segment is 3×30=90 frames.

[0116] In some embodiments, the recommended video can be segmented according to a preset number of segments K, where the preset number of segments K is an integer greater than 1. For example, the preset number of segments can be 5, 6, 10, etc. When segmenting the recommended video, the total number of frames S of the recommended video can be determined based on the duration T and frame rate F of the recommended video, and then the total number of frames S of the recommended video can be determined based on the duration T and frame rate F of the recommended video. The number of frames of each recommended video segment is determined, and then based on the number of frames of each recommended video segment, the recommended video is segmented to obtain K recommended video segments.

[0117] For example, T is 30, F is 30, and K is 5, so the total number of frames S is 900, and the number of frames of each recommended video segment is 180. Based on this, the recommended video is segmented to obtain 5 recommended video segments.

[0118] In this step, the recommended video is segmented to obtain multiple recommended video segments. In this way, content analysis (such as clarity detection, theme matching, etc.) can be performed independently for each recommended video segment to avoid resource waste in global calculation of long videos. In addition, through fine-grained segmentation (such as preset duration or preset number of segments), key frames or abnormal segments (such as blurred or inconsistent with the theme) can be more accurately located, thereby enhancing the targeted nature of video processing.

[0119] In step 112A, for each recommended video frame in each recommended video segment, a gradient magnitude of each pixel in the recommended video frame is determined.

[0120] In some embodiments, the gradient magnitude measures the intensity of local variations in pixel values ​​within an image (e.g., the sharpness of edges or textures). Higher gradient magnitudes indicate greater differences between adjacent pixels, i.e., sharper edges and richer textures; lower gradient magnitudes indicate smaller differences between adjacent pixels, i.e., smoother edges.

[0121] In some embodiments, the gradient magnitude of each pixel point in the recommended video frame is determined based on a preset operator (such as a Sobel operator or a Prewitt operator). The following is an explanation of determining the gradient magnitude of each pixel point based on the Sobel operator.

[0122] In some embodiments, the Sobel operator (including the horizontal convolution kernel and the vertical convolution kernel) is used to calculate the horizontal gradient value and the vertical gradient value of the pixel point respectively, and then the gradient amplitude of the pixel point is determined based on the horizontal gradient value and the vertical gradient value of the pixel point. Among them, the horizontal convolution kernel is used to detect vertical edges, and the vertical convolution kernel is used to detect horizontal edges. For example, the horizontal convolution kernel can be The vertical convolution kernel can be Then, for the pixel point, convolution is performed on the surrounding 3×3 neighboring pixels using the horizontal convolution kernel and the vertical convolution kernel respectively to obtain the horizontal gradient value Gx and the vertical gradient value Gy of the pixel point. Then, the gradient amplitude G(x,y) of the pixel point is determined based on the horizontal gradient value Gx and the vertical gradient value Gy using the following Euclidean distance formula (1):

[0123]

[0124] In the above, the dimensions of the horizontal convolution kernel and the vertical convolution kernel are 3×3. In the actual implementation process, 5×5 dimensional horizontal convolution kernel and vertical convolution kernel, 7×7 dimensional horizontal convolution kernel and vertical convolution kernel can also be used. When the dimensions of the convolution kernel are different, the degree of refinement of the edge detection is different. The higher the dimension of the convolution kernel, the lower the degree of refinement of the edge detection.

[0125] In some embodiments, the Prewitt operator can also be used to determine the horizontal gradient value and the vertical gradient value of the pixel point, so as to determine the gradient amplitude of the pixel point based on the horizontal gradient value and the vertical gradient value. The implementation method of using the Prewitt operator to determine the horizontal gradient value and the vertical gradient value of the pixel point is similar to the implementation process of using the Sobel operator to calculate the horizontal gradient value and the vertical gradient value of the pixel point. The difference is that the horizontal convolution kernel and the vertical convolution kernel used are different. In the convolution kernel of the Sobel operator, the center pixel weight is higher, so it can better resist noise, thereby obtaining a smoother edge detection effect. The convolution kernel of the Prewitt operator has a uniform weight, so the calculation is faster, more efficient, and more sensitive to small edges. In practical applications, different operators can be selected according to needs to determine the gradient amplitude of the pixel.

[0126] In step 113A, the gradient variance corresponding to the recommended video frame is determined based on the gradient magnitude of each pixel.

[0127] The variance of the recommended video frames measures the degree of dispersion in the edge strength distribution of the recommended video frames. A high variance indicates a large difference in edge strength. Generally, clear images captured in complex scenes or with fast motion have high variance. A low variance indicates uniform or flat edges, and blurry images have low variance.

[0128] In some embodiments, for a recommended video frame, after determining the gradient magnitude of each pixel point of the recommended video frame, the variance of the recommended video frame is determined by the following formula (2):

[0129]

[0130] Where W is the image width, H is the image height, and μ is the mean value of the gradient amplitude of the recommended video frame.

[0131]

[0132] In step 114A, for each recommended video segment, a clarity score of each recommended video frame is determined based on the gradient variance corresponding to each recommended video frame in the recommended video segment.

[0133] Here, for a recommended video clip, each recommended video frame in the recommended video clip corresponds to a gradient variance. Assuming that the recommended video clip contains M recommended video frames, then M gradient variances {Δ1, Δ2, …, Δ M}.

[0134] In some embodiments, for each recommended video clip, when step 114A is implemented, the gradient variance corresponding to each recommended video frame in the recommended video clip is sorted to obtain a variance sequence, the variance sequence is divided into a preset number of variance intervals, and the first value corresponding to each variance interval is obtained. For each variance interval, the first value corresponding to the variance interval is determined as the clarity score of the recommended video frame corresponding to each gradient variance located in the variance interval.

[0135] In some embodiments, the gradient variances corresponding to each recommended video frame in the recommended video segment can be sorted in ascending order to obtain a variance sequence, and the variance sequence can be divided into a preset number of variance intervals, each variance interval corresponding to a first value, and the first value can be obtained by subtracting one from the position of the variance interval. For example, the variance sequence includes 160 gradient variances, and the preset number is 16. Then, every 10 gradient variances in the variance sequence constitute a variance interval. The first value corresponding to the first variance interval is 0, and the clarity score of the recommended video frame corresponding to each gradient variance in the first variance interval is 0. The first value corresponding to the second variance interval is 1, and the clarity score of the recommended video frame corresponding to each gradient variance in the second variance interval is 1. The first value corresponding to the third variance interval is 2, and the clarity score of the recommended video frame corresponding to each gradient variance in the third variance interval is 2. Similarly, the first value corresponding to the 16th variance interval is 15, and the clarity score of the recommended video frame corresponding to each gradient variance in the 16th variance interval is 15.

[0136] In the above steps 111A to 114A, the gradient variance of the recommended video frame can be determined only through convolution operation and statistical processing, and then the clarity score of the recommended video frame is determined by the gradient variance. This has low computational complexity, thereby improving video processing efficiency.

[0137] In some embodiments, a clear image generally contains more high-frequency components (such as details and edges), while a blurred image has lower high-frequency energy. Therefore, the clarity of a video frame can also be determined based on frequency domain analysis, see Figure 7B Step 2011 can be implemented by the following steps 111B to 115B, and the following is combined Figure 7B accomplish.

[0138] In step 111B, the recommended video is segmented to obtain a plurality of recommended video segments.

[0139] Here, the implementation process of step 111B is the same as the implementation process of step 111A, and the implementation process of step 111A may be referred to during implementation.

[0140] In step 112B, for each recommended video frame in each recommended video segment, frequency domain conversion is performed on the recommended video frame to obtain a frequency spectrum of the recommended video frame.

[0141] Here, the video frame can be processed by Fourier transform or fast Fourier transform to obtain a spectrum diagram of the video frame. In the spectrum diagram, the frequency distribution follows the following rules: the low-frequency area is located in the center of the frequency domain image (corresponding to the slow changes in brightness / color in the image, such as the smooth area), and the low-frequency area represents the overall structure and lighting information of the image; the high-frequency area is located at the edge of the frequency domain image (that is, the part far away from the center), and the high-frequency area corresponds to the details, edges, noise and other rapidly changing components of the image.

[0142] In step 113B, the spectrum diagram is divided into a first region and a second region.

[0143] Among them, the amount of change in image information in the first area is higher than the amount of change in image information in the second area. That is, the first area is a high-frequency area, and the second area is a low-frequency area. In some embodiments, when dividing the first area and the second area, the center of the spectrum graph can be used as the origin, and a radius threshold (such as 1 / 4 of the image width) can be set. The area in the spectrum graph that is less than or equal to the radius threshold from the origin is determined as the second area, and the area in the spectrum graph that is greater than the radius threshold from the origin is determined as the first area. In some embodiments, an annular mask (such as a Butterworth high-pass filter) can also be used to smoothly transition the high- and low-frequency boundary to obtain the first area and the second area.

[0144] In step 114B, the energy proportion of the first region corresponding to the recommended video frame is determined.

[0145] In some embodiments, when implementing step 114B, the total energy of the spectrum graph is first determined, and then the first energy corresponding to the first area is determined. The ratio of the first energy to the total energy is determined as the energy proportion of the first area. If the energy proportion of the first area of ​​the recommended video frame is higher, it means that there are more high-frequency components and the image is clearer.

[0146] In step 115B, for each recommended video segment, a clarity score of each recommended video frame is determined based on the energy proportion corresponding to each recommended video frame in the recommended video segment.

[0147] In some embodiments, when determining the clarity score of a recommended video frame based on the energy proportion, the energy proportion can be directly normalized to obtain the clarity score of the recommended video frame. The clarity score can also be obtained by performing a logarithmic transformation on the energy proportion. After determining the energy proportion of the first area of ​​each recommended video frame, the energy proportions can be sorted from small to large to obtain a proportion sequence, and the proportion sequence can be divided into multiple proportion intervals to determine the second value corresponding to each proportion interval. For each proportion interval, the clarity score of the recommended video frame corresponding to each energy proportion in the proportion interval is determined as the second value corresponding to the proportion interval.

[0148] In the above steps 111B to 115B, the recommended video frame is subjected to frequency domain conversion to obtain a spectrum diagram, and the first area (high-frequency area) and the second area (low-frequency area) are determined through the spectrum diagram, so that the clarity score of the recommended video frame is determined based on the energy proportion of the first area. In this way, the image clarity is measured by the frequency domain characteristics of the recommended video frame, and the influence of noise can be suppressed by frequency domain filtering, so that the obtained clarity score is more accurate.

[0149] In step 2012, the theme information of the recommended video is obtained, and based on the theme information, the content relevance score of each recommended video frame is determined.

[0150] Here, the theme information of the recommended video refers to the core content information that the recommended video wants to convey. The theme information may include theme text. The theme text refers to the text content that can directly convey the core information of the recommended video, the key features of the product or service, the promotional focus, etc. For example, slogans, product names, descriptions of key selling points, etc. For example, the texts such as "High-definition pixels, record beautiful moments" and "XX mobile phone, excellent performance" in the recommended video of a certain mobile phone are theme texts. In addition to theme text, theme information may also include visual elements, audio elements and other content. The content relevance score is used to measure the similarity between the recommended video frame and the recommended video theme. In some embodiments, the content relevance score of the recommended video frame can be determined by the similarity between the text contained in the recommended video frame and the theme text in the theme information of the recommended video, see Figure 8A In step 2012, "determining the content relevance score of each recommended video frame based on the subject information" can be implemented by the following steps 121A to 124A. Figure 8A Provide explanation.

[0151] In step 121A, the recommended video is segmented to obtain a plurality of recommended video segments.

[0152] In step 122A, text recognition is performed on each recommended video frame in each recommended video segment to obtain a recognition result.

[0153] In some embodiments, when performing text recognition on a recommended video frame, text region detection is first performed to determine the region in the recommended video frame that may contain text. This can be achieved through image analysis and edge detection algorithms. The edge detection algorithm can identify edge features in the image and then determine the outline of the text region based on the edge features to obtain the text region. Then, text recognition can be performed on the text region using a trained text recognition model or a preset text recognition algorithm. The following uses a trained text recognition model as an example to illustrate text recognition: First, the convolution module in the text recognition model extracts local texture, edge and other visual features, then performs spatial compression, and reduces the feature map size through a pooling layer (such as Max Pooling) to retain key information. The feature map is then treated as a sequence in the width direction, with each time step corresponding to a column of feature vectors, and finally a feature sequence is output. The feature sequence includes A time steps, each time step is a C'-dimensional vector. The Long Short-Term Memory (LSTM) / Gated Recurrent Unit (GRU) module processes the feature sequence in sequence according to the time step, using the hidden state to transmit context information. The input of each time step is the current feature vector and the hidden state of the previous step. A fully connected layer is connected after each time step, and the character probability (such as 62 types of letters and numbers + blank characters) is output through the fully connected layer. After obtaining the character probability corresponding to each time step, the final first text is obtained through the preset decoding strategy through the continuous temporal classification (CTC) decoding module. The preset decoding strategy can be greedy search, beam search, etc., among which greedy search selects the character with the highest probability at each time step and ignores blank marks. This method is computationally efficient, but may not be able to obtain the global optimal solution; beam search retains the several candidate sequences with the highest probability at each time step and expands these sequences at subsequent time steps.

[0154] In some implementations, when no text region is detected in the recommended video frame, the recognition result may be that no text is detected.

[0155] The above is an exemplary description of the text recognition implementation process, and the embodiments of the present application do not limit the text recognition implementation process.

[0156] In step 123A, when it is determined based on the recognition result that the first text is recognized, a first similarity is determined based on the first text and the topic information.

[0157] When determining the first similarity between the first text and the topic information, the first language of the first text and the second language of the topic text in the topic information are first obtained, and it is determined whether the first language and the second language are the same. If the first language and the second language are the same, a first text vector corresponding to the first text and a second text vector corresponding to the topic text are generated, and the first similarity between the first text and the topic text is determined based on the first text vector and the second text vector; if the first language and the second language are different, the first text is translated to obtain a second text, the language of the second text is the second language, and then a third text vector corresponding to the second text and a second text vector corresponding to the topic text are generated, and the first similarity between the first text and the topic text is determined based on the second text vector and the third text vector.

[0158] In some embodiments, when generating a first text vector corresponding to a first text, the first text is first preprocessed to remove noise (such as garbled characters, special symbols) and irrelevant characters (such as timestamps in subtitles) in the first text, and then the preprocessed first text is segmented to obtain a segmentation result, and then the first text vector is generated through a trained network model. For example, the segmentation result can be word embedded through a trained network model to obtain the first text vector. The implementation process of generating the second text vector corresponding to the subject text is similar to the implementation process of generating the first text vector.

[0159] When determining the first similarity between the first text and the subject text based on the first text vector and the second text vector, the cosine value of the angle between the first text vector and the second text vector can be determined, and the first similarity between the first text and the subject text can be determined based on the cosine value of the angle, wherein the closer the cosine value of the angle is to 1, the higher the similarity between the first text and the subject text. In some embodiments, the distance between the first text vector and the second text vector (such as Euclidean distance, Manhattan distance, etc.) can also be determined, and the similarity between the first text and the subject text can be determined based on the distance between the two vectors. The greater the distance, the lower the similarity.

[0160] In some embodiments, when it is determined based on the recognition result that the first text is not recognized, the first similarity is determined as a third value. For example, the third value may be 0.

[0161] In step 124A, for each of the recommended video segments, a content relevance score of each recommended video frame is determined based on the first similarity corresponding to each recommended video frame in the recommended video segment.

[0162] In some embodiments, when determining the content relevance score of the recommended video frame based on the first similarity, after determining the first similarity of each recommended video frame, the first similarities can be sorted from small to large to obtain a first similarity sequence, and then the first similarity sequence can be divided into multiple first similarity intervals, and the fourth value corresponding to each first similarity interval is determined. For each first similarity interval, the content relevance score of the recommended video frame corresponding to each first similarity in the first similarity interval is determined as the fourth value corresponding to the first similarity interval.

[0163] For example, the first similarity sequence can be divided into 4 first similarity intervals, or 8 first similarity intervals, or 16 first similarity intervals. If the first similarity sequence is divided into 4 first similarity intervals, the fourth value corresponding to each first similarity interval is 0, 1, 2, and 3, respectively.

[0164] In some embodiments, the first similarity may be directly normalized to obtain a content relevance score for the recommended video frame, or the first similarity may be logarithmically transformed to obtain a content relevance score.

[0165] In the above embodiment, when determining the content relevance score of the recommended video frame, text recognition is first performed to obtain the recognition result. When the first text is obtained through the recognition result, it is first determined whether the first text and the subject text are in the same language. If they are not in the same language, the first text needs to be translated into a second text that is consistent with the language of the subject text. Therefore, the first similarity between the recommended video frame and the subject text is determined based on the text vector between the two texts, and the content relevance score of the recommended video frame is further determined based on the first similarity. This is not only simple to calculate, but also can capture the semantic association between the first text and the subject text, avoiding the limitations of simple keyword matching, and therefore has a high accuracy rate.

[0166] In some embodiments, semantic understanding of the recommended video frame can be performed to determine the content relevance score between the recommended video frame and the recommended video theme. Figure 8B Step 2012 can be implemented by the following steps 121B to 124B, and the following is combined Figure 8B Provide explanation.

[0167] In step 121B, the recommended video is segmented to obtain a plurality of recommended video segments.

[0168] In step 122B, for each recommended video frame in each recommended video segment, semantic understanding is performed on the recommended video frame to obtain a semantic understanding result.

[0169] In some embodiments, semantic understanding of recommended video frames involves using computer vision technology to enable a machine to understand high-level semantic information such as objects, scenes, actions, and relationships within the recommended video frames. During implementation, step 122B first extracts image features from the recommended video frames. For example, a convolutional neural network (CNN) can be used to automatically extract image features. Through operations such as convolutional layers and pooling layers, CNN can learn features at different levels in the image, from low-level edges and corners to high-level semantic features, and then perform target detection and positioning. The sliding window method or deep learning-based methods can be used to determine the target object in the recommended video frame as well as the position and category of the target object. The extracted features can then be input into classifiers such as support vector machines and decision trees, or the extracted features can be input into a trained deep neural network model for image classification to obtain the image classification results of the recommended video frames. The recommended video frames can then be semantically segmented. A fully convolutional neural network can be used to output a semantic segmentation map of the same size as the input image, with each pixel corresponding to a category label, to achieve fine segmentation of different objects in the image. Finally, the results of different tasks such as target detection, image classification, and semantic segmentation are integrated to obtain a more comprehensive and accurate semantic understanding result, which can be represented in vectorized form.

[0170] In step 123B, a second similarity is determined based on the semantic understanding result and the topic information.

[0171] In some embodiments, the topic text can be obtained from the topic information, and then the topic text can be word embedded to obtain a second text vector of the topic text, and then the second similarity between the semantic understanding result and the second text vector is determined, where the semantic understanding result can be a semantic understanding vector. At this time, the second similarity can be determined by a process similar to step 123A, by determining the cosine value of the angle between the semantic understanding vector and the second text vector, or the distance between the semantic understanding vector and the second text vector.

[0172] In step 124B, for each recommended video segment, a content relevance score of each recommended video frame in the recommended video segment is determined based on the second similarity corresponding to each recommended video frame.

[0173] In some embodiments, when determining the content relevance score of the recommended video frame based on the second similarity, after determining the second similarity of each recommended video frame, the second similarities can be sorted from small to large to obtain a second similarity sequence, and then the second similarity sequence can be divided into multiple second similarity intervals, and the fifth value corresponding to each second similarity interval is determined. For each second similarity interval, the content relevance score of the recommended video frame corresponding to each second similarity in the second similarity interval is determined as the fifth value corresponding to the second similarity interval.

[0174] In the above steps 121B to 124B, semantic understanding is performed on the recommended video frame to obtain a semantic understanding result, and a second similarity is determined based on the semantic understanding result and the second text vector of the theme text of the recommended video, thereby determining the content relevance score of the recommended video frame based on the second similarity. In this way, through multimodal semantic understanding, a fused semantic understanding vector is obtained, which realizes a deep, flexible and robust association analysis between the picture content of the recommended video frame and the theme text, and can improve the accuracy of the content relevance score.

[0175] In step 2013, a quality score corresponding to each recommended video frame is determined based on the clarity score and content relevance score of each recommended video frame.

[0176] In some embodiments, for each recommended video frame, the clarity score and content relevance score can be directly combined to determine the quality score corresponding to the recommended video frame, or the clarity score and content relevance score can be fused to obtain the quality score corresponding to the recommended video frame. For example, a first weight for the clarity score and a second weight for the content relevance score can be obtained, and then the clarity score and content relevance score can be weighted and summed using the first and second weights to obtain the quality score corresponding to the recommended video frame. The first and second weights can be pre-set. When the first weight is greater than the second weight, the clarity score has a greater impact on the quality score than the content relevance score. When the first weight is less than the second weight, the clarity score has a smaller impact on the quality score than the content relevance score. For example, if the clarity score is 14 points and the content relevance score is 12 points, the first weight is 0.6, and the second weight is 0.4, the quality score is 13.2. In some embodiments, if the quality score is a decimal, the quality score can be approximated to obtain an integer quality score. This approximation can be performed by rounding up, rounding down, or rounding up.

[0177] In step 202 , for each recommended video frame, metadata of the recommended video frame is generated based on the quality score of the recommended video frame and the quality scores of multiple recommended video frames following the recommended video frame.

[0178] In some embodiments, the video frame identifier of the current recommended video frame is obtained, and based on the video frame identifier, a preset number of recommended video frames after the recommended video frame is determined. For example, the preset number can be the frame rate of the recommended video, that is, multiple recommended video frames within 1 second after the recommended video frame are obtained at this time. The preset number can also be the frame rate minus 1. For example, if the frame rate is 60, the preset number can be 59, and the metadata includes the quality scores of the 60 recommended video frames. In addition to the quality scores of the recommended video frames, the metadata also includes the video frame identifiers of each recommended video frame. The video frame identifier can be represented by a display timestamp or a video frame index represented by an integer. In the embodiment of the present application, the video frame identifier is used as a video frame index for explanation.

[0179] For example, the current recommended video frame has a video frame identifier of 21, and its quality scores include a clarity score of 14 and a content relevance score of 12. The quality scores of the recommended video frame with a video frame identifier of 22 include a clarity score of 15 and a content relevance score of 14. The quality scores of the recommended video frame with a video frame identifier of 23 include a clarity score of 10 and a content relevance score of 8, ..., and the quality scores of the recommended video frame with a video frame identifier of 80 include a clarity score of 8 and a content relevance score of 4. Then, the metadata of the recommended video frame with a video frame identifier of 21 is {{21, 14, 12}, {22, 15, 14}, {23, 10, 8}, ..., {80, 8, 4}}.

[0180] In step 203 , recommended video data is generated based on the recommended video frames and the metadata.

[0181] In some embodiments, when step 203 is implemented, the metadata of the recommended video frame is first formatted to obtain structured metadata, such as structured metadata in Josn format or structured metadata in binary format, and then each recommended video frame is bound to its corresponding structured metadata to obtain each recommended video frame bound with structured metadata. During implementation, the structured metadata is added to the SEI unit and timestamp alignment is performed to ensure that the structured metadata in the SEI unit is synchronized with the corresponding recommended video frame timestamp, that is, the binding of metadata and recommended video frames is achieved. SEI is an additional data unit in the H.264 / H.265 video coding standard, which allows uncompressed metadata to be inserted between video frames without affecting video decoding. Each recommended video frame bound with structured metadata constitutes recommended video data.

[0182] In step 204, the first video corresponding to the recommended video is obtained, and the recommended video data and the first video are spliced ​​and encoded to obtain video data to be played.

[0183] In some embodiments, the first video may be a film or television drama video, a sports game video, a news video, etc. When implementing step 204, the recommended video data and the first video are first spliced ​​together, and then the spliced ​​video is encoded to obtain the video data to be played. When splicing the recommended video data and the first video, the recommended video data can be spliced ​​before the first video, or in the middle of the first video to obtain the spliced ​​video. The encoder then applies a preset encoding algorithm to compress and encode the spliced ​​video. For example, in video encoding, commonly used encoding algorithms include H.264, H.265, etc. These algorithms achieve efficient compression by removing redundant information in the video signal. After encoding, the encoder generates a compressed bit stream, which is the video data to be played.

[0184] In the video processing method provided in the embodiment of the present application, the quality of the recommended video is first evaluated to obtain the clarity score and content relevance score of each recommended video frame in the recommended video, and the quality score of each recommended video frame is determined based on the clarity score and content relevance score. For each recommended video frame, metadata of the recommended video frame is generated based on the quality score of the recommended video frame and the quality scores of multiple recommended video frames after the recommended video frame. In this way, the target video frame can be determined in real time when the player plays the video to ensure smooth playback. After obtaining the metadata of each recommended video frame, the recommended video frame and the metadata are bound, and the recommended video data and the main video data are spliced ​​and encoded to obtain the video data to be played, thereby providing the necessary data basis for determining the target video frame during the video playback process.

[0185] The present application further provides a video processing method for Figure 1 The video processing system shown, Figure 9A This is another implementation flow diagram of the video processing method provided in the embodiment of the present application. Figure 9A Provide explanation.

[0186] In step 301, the server obtains a recommended video to be inserted, and determines a quality score corresponding to each recommended video frame in the recommended video.

[0187] In step 302 , the server generates metadata of each recommended video frame based on the quality score of the recommended video frame and the quality scores of multiple recommended video frames following the recommended video frame.

[0188] In step 303 , the server generates recommended video data based on each recommended video frame and each metadata.

[0189] In step 304, the server obtains the first video corresponding to the recommended video, and performs splicing and encoding processing on the recommended video data and the first video to obtain video data to be played.

[0190] It should be noted that the implementation process of the above steps 301 to 304 can refer to steps 101 to 104.

[0191] In step 305, in response to receiving the video play instruction, the terminal sends a video play request to the server.

[0192] In step 306, the server sends the video data to be played to the terminal in response to the video play request.

[0193] In step 307, the terminal plays the video based on the video data to be played.

[0194] In step 308 , during the video playback process, in response to receiving the playback progress adjustment instruction, the terminal determines a first video frame corresponding to the playback progress adjustment instruction.

[0195] In step 309, the terminal determines whether the first video frame is a recommended video frame.

[0196] When the first video frame is a recommended video frame, the process proceeds to step 310 ; when the first video frame is not a recommended video frame, the process proceeds to step 312 .

[0197] In step 310 , the terminal obtains metadata corresponding to the first video frame from the recommended video data.

[0198] The metadata includes quality scores of multiple candidate video frames, and the candidate video frames include the first video frame and multiple second video frames following the first video frame.

[0199] In step 311 , the terminal obtains a target video frame from the multiple candidate video frames based on the quality scores of the multiple candidate video frames, and continues to play the video starting from the target video frame.

[0200] In step 312, the terminal continues playing the video starting from the first video frame.

[0201] It should be noted that the implementation process of the above steps 305 to 312 can refer to steps 101 to 104.

[0202] In the video processing method provided in the embodiment of the present application, the server performs a quality evaluation on the recommended video, obtains the clarity score and content relevance score of each recommended video frame in the recommended video, and determines the quality score of each recommended video frame based on the clarity score and content relevance score. For each recommended video frame, the server generates metadata for the recommended video frame based on the quality score of the recommended video frame and the quality scores of multiple recommended video frames after the recommended video frame. In this way, the target video frame can be determined in real time when the player plays the video to ensure smooth playback. After obtaining the metadata of each recommended video frame, the server binds the recommended video frame and the metadata, and splices and encodes the recommended video data and the main video data to obtain the video data to be played. After receiving the video playback request from the terminal, the server sends the data to be played to the terminal, so that when the terminal receives a progress adjustment instruction during the video playback process, it can determine a high-quality target video frame based on the metadata and continue to play the video from the target video frame. This not only improves the display effect of the recommended video, but also improves the recommendation conversion rate.

[0203] The following describes an exemplary application of the embodiments of the present application in a practical application scenario.

[0204] An embodiment of the present application provides a video processing method, the purpose of which is to output recommended video frames that allow users to clearly identify product brands, thereby improving the exposure effect of recommended videos. At the same time, in order to ensure a seamless user experience, the playback terminal needs to be able to make display decisions in real time.

[0205] Figure 9B This is another implementation flow diagram of the video processing method provided in the embodiment of the present application, such as Figure 9B As shown, the implementation process includes:

[0206] Step 401: The server obtains recommended videos.

[0207] In step 402 , the server determines metadata of each recommended video frame in the recommended video through metadata analysis.

[0208] Here, in order to ensure the exposure effect of the recommended video, the embodiment of the present application evaluates the quality of the recommended video frame based on the clarity of the picture and the correlation between the text in the picture and the recommended topic. Therefore, the metadata analysis includes clarity analysis and text correlation analysis, wherein the picture clarity score of the recommended video frame is obtained through clarity analysis (corresponding to the clarity score in other embodiments), and the text correlation score between the text in the recommended video frame and the recommended topic text is obtained through text correlation analysis (corresponding to the content correlation score in other embodiments).

[0209] The picture clarity score can range from 0 to 15, so the picture clarity score of a recommended video frame can be represented by a 4-bit binary number. When performing clarity analysis on the recommended video frame, the recommended video is first segmented according to a preset duration, with each segment being N seconds long. Then, for each recommended video segment, the picture clarity score of each recommended video frame in the recommended video segment is determined based on the variance statistics of the edge detection algorithm. In some embodiments, the recommended video frame is filtered using a Sobel operator or other operators (i.e., the image is convolved) to obtain the gradient amplitude of each pixel in the recommended video frame, and the gradient variance of the recommended video frame is determined based on the amplitudes of each pixel. The gradient variance corresponding to the i-th recommended video frame in the recommended video segment is denoted as Δ i The gradient variances of each recommended video frame in the recommended video clip are sorted from smallest to largest to obtain a variance sequence. This variance sequence is then divided into 16 variance intervals, each corresponding to an integer value. For each variance interval, the picture clarity score of the recommended video frame corresponding to the gradient variance in that variance interval is determined as the integer value corresponding to that variance interval. For example, if the first variance interval corresponds to the integer value 0, then the picture clarity score of the recommended video frame corresponding to each gradient variance in the first variance interval is 0.

[0210] The value range of the text relevance score can also be 0 to 15, so the text relevance score of a recommended video frame can be represented by a 4-bit binary number. When performing text relevance analysis on the recommended video frame, the text in the recommended video frame is first identified, and then the degree of relevance between the text in the recommended video frame and the recommended video theme is determined (corresponding to the first similarity in other embodiments). The degree of relevance can be determined by comparing methods such as text vector similarity. The degree of relevance corresponding to each recommended video frame is then sorted to obtain a degree of relevance sequence, and the relevance program sequence is divided into 16 degree of relevance intervals, each of which has an integer value from 0 to 15. The text relevance scores of the video frames corresponding to each degree of relevance in a degree of relevance interval are recorded as the integer value corresponding to the degree of relevance interval.

[0211] In order to ensure the real-time decision-making of the player, the metadata of a recommended video frame not only includes its own picture clarity score and text relevance score, but also needs to carry the picture clarity scores and text relevance scores of multiple recommended video frames after itself. For example, it can carry the picture clarity scores and text relevance scores of multiple recommended video frames within one second after the recommended video frame, thereby ensuring that the player can adjust in real time. Therefore, in order to ensure the correspondence between the recommended video frames and their respective picture clarity scores and text relevance scores, the metadata also needs to include the frame index of each recommended video frame. The frame index is generally a display timestamp, and can also be a number starting from 0. For a frame index of findex For the recommended video frame, the metadata of the recommended video frame can be expressed as: ad metadata :{{f index ,p score ,t score},{f index+1 ,p score ,t score},…,{f index+60 ,p score ,t score}}

[0212] Step 403: The server writes metadata.

[0213] In some embodiments, metadata can be written into the SEI unit to achieve the binding of metadata with the recommended video frame. The server then splices the recommended video with metadata and the main video to be played, and encodes the spliced ​​video to obtain the video data to be transmitted.

[0214] Step 404: The server transmits video data.

[0215] Here, after receiving the video play request sent by the terminal, the server may obtain video data corresponding to the video play request and send the video data to the terminal.

[0216] Step 405: The terminal decodes the received video data.

[0217] Step 406: The terminal determines whether a progress adjustment operation is detected.

[0218] Here, if the terminal detects the progress adjustment operation, the process proceeds to step 407 ; if the terminal does not detect the progress adjustment operation, the process proceeds to step 405 .

[0219] Step 407: The terminal determines whether the first video frame contains metadata.

[0220] Here, the first video frame is the video frame corresponding to the progress adjustment operation. If the first video frame contains metadata, it means that the first video frame is a recommended video frame, and step 408 is entered at this time. If the first video frame does not contain metadata, it means that the first video frame is not a recommended video frame, and step 410 is entered.

[0221] Step 408: The terminal parses the metadata and determines the target video frame.

[0222] In some embodiments, the terminal parses metadata to obtain a candidate video frame with the highest clarity score, and then obtains a target video frame with the highest text relevance score from the candidate video frame with the highest clarity score.

[0223] Figure 10A This is a schematic diagram of the first video frame determined based on the progress adjustment operation provided by the embodiment of the present application. Figure 10A It can be seen that the first video frame is not a clear image at this time. Figure 10B The target video frame is determined based on the video processing method provided in the embodiment of the present application. Figure 10A and Figure 10B It can be seen that the quality of the target video frame displayed to the user is better than Figure 10A The first video frame in .

[0224] Step 409: The terminal continues playing the video starting from the target video frame.

[0225] In some embodiments, it is determined whether the first video frame is the target video frame. If the first video frame is not the target video frame, the first video frame is discarded and the video is jumped to the target video frame to start playing. If the first video frame is the target video frame, the video is played starting from the first video frame (that is, the target video frame).

[0226] In step 410, the terminal starts playing the video from the first video frame.

[0227] In some embodiments, if the first video frame is not a recommended video frame, it means that the first video frame is the main video that the user wants to watch, and the video starts playing from the first video frame.

[0228] The video processing method provided in the embodiment of the present application can improve the exposure effect of recommended videos, improve the impression of recommended videos on users without affecting the use of user functions, and thus improve the recommendation conversion rate.

[0229] The following continues to describe the exemplary structure of the video processing device 455 provided in the embodiment of the present application implemented as a software module. In some embodiments, such as Figure 2A As shown, the software modules stored in the video processing device 455 of the memory 450 may include:

[0230] The video playing module 4551 is configured to obtain video data to be played in response to a received video playing instruction, and play the video based on the video data to be played, wherein the video data to be played includes recommended video data;

[0231] A first determining module 4552 is configured to, in response to receiving a playback progress adjustment instruction during video playback, determine a first video frame corresponding to the playback progress adjustment instruction;

[0232] A first acquisition module 4553 is configured to, when the first video frame is a recommended video frame, acquire metadata corresponding to the first video frame from the recommended video data, the metadata including quality scores of multiple candidate video frames, the candidate video frames including the first video frame and multiple second video frames following the first video frame;

[0233] The video playback module 4551 is further configured to determine a target video frame from the plurality of candidate video frames based on the quality scores of the plurality of candidate video frames, and continue playing the video starting from the target video frame.

[0234] In some embodiments, the quality score includes at least a clarity score and a content relevance score, and the video playback module 4551 is further used to: determine at least one third video frame that meets the clarity condition from multiple candidate video frames based on the clarity score; determine at least one fourth video frame that meets the relevance condition from at least one of the third video frames based on the content relevance score; and determine the target video frame from at least one of the fourth video frames.

[0235] In some implementations, the video playback module 4551 is further used to determine the fourth video frame as the target video frame if the number of the fourth video frames is 1; obtain the video frame identifier of each of the fourth video frames if the number of the fourth video frames is greater than 1; and determine the target video frame from multiple fourth video frames based on each of the video frame identifiers.

[0236] In some implementations, the video playback module 4551 is further used to determine the fourth video frame corresponding to the smallest video frame identifier as the target video frame; or, to determine the fourth video frame corresponding to the largest video frame identifier as the target video frame, or, to determine the absolute value of the identifier difference between each of the video frame identifiers and the first identifier of the first video frame; and to determine the fourth video frame corresponding to the smallest absolute value of the identifier difference as the target video frame.

[0237] In some embodiments, the quality score is a comprehensive score obtained by the clarity score and the content relevance score. The video playback module 4551 is also used to determine at least one fifth video frame that meets the quality conditions from multiple candidate video frames based on the quality score; and determine the target video frame from at least one fifth video frame.

[0238] The following continues to describe the exemplary structure of the video processing device 233 provided in the embodiment of the present application implemented as a software module. In some embodiments, such as Figure 2B As shown, the software modules stored in the video processing device 233 of the memory 230 may include:

[0239] The score determination module 2331 is used to obtain the recommended video to be inserted and determine the quality score corresponding to each recommended video frame in the recommended video;

[0240] A first generating module 2332 is configured to generate metadata of each recommended video frame based on the quality score of the recommended video frame and the quality scores of multiple recommended video frames following the recommended video frame.

[0241] A second generating module 2333 is configured to generate recommended video data based on each of the recommended video frames and each of the recommended video frames;

[0242] The splicing and encoding module 2334 is used to obtain the first video corresponding to the recommended video, and splice and encode the recommended video data and the first video to obtain video data to be played.

[0243] In some embodiments, the score determination module 2331 is further used to determine the clarity score of each of the recommended video frames; obtain the theme information of the recommended video, and determine the content relevance score of each of the recommended video frames based on the theme information; and determine the quality score corresponding to each of the recommended video frames based on the clarity score and the content relevance score of each of the recommended video frames.

[0244] In some embodiments, the score determination module 2331 is further used to segment the recommended video to obtain multiple recommended video segments; for each recommended video frame in each of the recommended video segments, determine the gradient amplitude of each pixel in the recommended video frame; based on the gradient amplitude of each pixel, determine the gradient variance corresponding to the recommended video frame; for each of the recommended video segments, determine the clarity score of each of the recommended video frames based on the gradient variance corresponding to each of the recommended video frames in the recommended video segment.

[0245] In some embodiments, the score determination module 2331 is further used to sort the gradient variances corresponding to each of the recommended video frames in the video clip to obtain a variance sequence; divide the variance sequence into a preset number of variance intervals, and obtain a first value corresponding to each of the variance intervals; for each of the variance intervals, determine the first value corresponding to the variance interval as the clarity score of the recommended video frame corresponding to each gradient variance located in the variance interval.

[0246] In some embodiments, the score determination module 2331 is further used to segment the recommended video to obtain multiple recommended video segments; for each recommended video frame in each of the recommended video segments, perform frequency domain conversion on the recommended video frame to obtain a spectrum diagram of the recommended video frame; divide the spectrum image into a first area and a second area, and the image information change amount in the first area is higher than the image information change amount in the second area; determine the energy proportion of the first area corresponding to the recommended video frame; for each of the recommended video segments, determine the clarity score of each of the recommended video frames based on the energy proportion corresponding to each of the recommended video frames in the recommended video segment.

[0247] In some embodiments, the score determination module 2331 is further used to segment the recommended video to obtain multiple recommended video segments; for each recommended video frame in each of the recommended video segments, text recognition is performed on the recommended video frame to obtain a recognition result; when a first text is determined to be identified based on the recognition result, a first similarity is determined based on the first text and the topic information; for each of the recommended video segments, a content relevance score of each of the recommended video frames is determined based on the first similarity corresponding to each of the recommended video frames in the recommended video segment.

[0248] In some embodiments, the score determination module 2331 is further used to obtain the first language of the first text and the second language of the topic text in the topic information; when the first language and the second language are the same, determine the first text vector of the first text and the second text vector of the topic text; and determine a first similarity based on the first text vector and the second text vector.

[0249] In some embodiments, the score determination module 2331 is further used to, when the first language and the second language are different, translate the first text based on the second language to obtain a second text; determine a third text vector of the second text; and determine a first similarity based on the third text vector and the second text vector.

[0250] In some embodiments, the score determination module 2331 is further used to segment the recommended video to obtain multiple recommended video segments; for each recommended video frame in each of the recommended video segments, perform semantic understanding on the recommended video frame to obtain a semantic understanding result; based on the semantic understanding result and the topic information, determine the second similarity; for each of the recommended video segments, determine the content relevance score of each of the recommended video frames based on the second similarity corresponding to each of the recommended video frames in the recommended video segment.

[0251] In some embodiments, the score determination module 2331 is further used to determine, for each of the recommended video frames, the clarity score and the content relevance score as the quality score corresponding to the recommended video frame, or to fuse the clarity score and the content relevance score to obtain the quality score corresponding to the recommended video frame.

[0252] The present invention provides a computer program product comprising a computer program or computer-executable instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer program or computer-executable instructions from the computer-readable storage medium and executes the computer program or computer-executable instructions, causing the computer device to perform the video processing method described in the present invention.

[0253] The embodiment of the present application provides a computer-readable storage medium in which computer-executable instructions or computer programs are stored. When the computer-executable instructions or computer programs are executed by a processor, the processor will execute the video processing method provided by the embodiment of the present application, for example, Figure 3 The video processing method is shown.

[0254] In some embodiments, the computer-readable storage medium may be a memory such as RAM, ROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or may be various devices including one or any combination of the above memories.

[0255] In some embodiments, computer-executable instructions may be in the form of a program, software, software module, script, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.

[0256] As an example, computer-executable instructions may, but need not, correspond to a file in a file system, may be stored as part of a file that stores other programs or data, such as in one or more scripts in a HyperText Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple coordinating files (e.g., files storing one or more modules, subroutines, or code portions).

[0257] By way of example, computer-executable instructions may be deployed to be executed on one computer device or on multiple computer devices at one site or on multiple computer devices distributed across multiple sites and interconnected by a communication network.

[0258] The above description is merely an embodiment of the present application and is not intended to limit the scope of protection of the present application. Any modifications, equivalent replacements, and improvements made within the spirit and scope of the present application are included in the scope of protection of the present application.

Claims

1. A video processing method, characterized in that: The method comprises: In response to the received video playback instruction, obtaining video data to be played, and playing the video based on the video data to be played, wherein the video data to be played includes recommended video data; During video playback, in response to receiving a playback progress adjustment instruction, determining a first video frame corresponding to the playback progress adjustment instruction; When the first video frame is a recommended video frame, obtaining metadata corresponding to the first video frame from the recommended video data, the metadata including quality scores of multiple candidate video frames, the candidate video frames including the first video frame and multiple second video frames following the first video frame; Based on the quality scores of the multiple candidate video frames, a target video frame is obtained from the multiple candidate video frames, and the video is continuously played starting from the target video frame.

2. The method according to claim 1, characterized in that The quality score includes at least a clarity score and a content relevance score, and obtaining a target video frame from the plurality of candidate video frames based on the quality scores of the plurality of candidate video frames includes: Determining, based on the clarity score, at least one third video frame that meets a clarity condition from the plurality of candidate video frames; determining, based on the content relevance score, at least one fourth video frame that meets a relevance condition from at least one of the third video frames; The target video frame is determined from at least one of the fourth video frames.

3. The method according to claim 2, characterized in that The determining the target video frame from at least one of the fourth video frames includes: If the number of the fourth video frame is 1, determining the fourth video frame as the target video frame; If the number of the fourth video frames is greater than 1, obtaining a video frame identifier of each of the fourth video frames; Based on each of the video frame identifiers, the target video frame is determined from the plurality of fourth video frames.

4. The method according to claim 3, characterized in that The determining the target video frame from the plurality of fourth video frames based on each of the video frame identifiers includes: Determine the fourth video frame corresponding to the smallest video frame identifier as the target video frame; or, The fourth video frame corresponding to the largest video frame identifier is determined as the target video frame, or, Determine an absolute value of a difference between each of the video frame identifiers and the first identifier of the first video frame; The fourth video frame corresponding to the minimum absolute value of the identification difference is determined as the target video frame.

5. The method according to claim 1, characterized in that The quality score is a comprehensive score obtained by combining a clarity score and a content relevance score. The step of obtaining a target video frame from the plurality of candidate video frames based on the quality scores of the plurality of candidate video frames includes: determining, based on the quality scores, at least one fifth video frame satisfying a quality condition from the plurality of candidate video frames; The target video frame is determined from at least one of the fifth video frames.

6. A video processing method, characterized in that: The method comprises: Obtaining a recommended video to be inserted, and determining a quality score corresponding to each recommended video frame in the recommended video; For each of the recommended video frames, generating metadata of the recommended video frame based on the quality score of the recommended video frame and the quality scores of multiple recommended video frames following the recommended video frame; generating recommended video data based on each of the recommended video frames and each of the metadata; The first video corresponding to the recommended video is obtained, and the recommended video data and the first video are spliced ​​and encoded to obtain video data to be played.

7. The method according to claim 6, characterized in that Determining the quality score corresponding to each recommended video frame in the recommended video includes: Determining a clarity score for each of the recommended video frames; Obtaining theme information of the recommended video, and determining a content relevance score for each of the recommended video frames based on the theme information; Based on the clarity score and the content relevance score of each recommended video frame, a quality score corresponding to each recommended video frame is determined.

8. The method according to claim 7, characterized in that Determining the clarity score of each of the recommended video frames includes: Segmenting the recommended video to obtain multiple recommended video segments; For each recommended video frame in each recommended video segment, determining a gradient magnitude of each pixel in the recommended video frame; Determining a gradient variance corresponding to the recommended video frame based on the gradient magnitude of each pixel; For each of the recommended video segments, a clarity score of each of the recommended video frames in the recommended video segment is determined based on a gradient variance corresponding to each of the recommended video frames.

9. The method according to claim 8, characterized in that The determining, based on the gradient variance corresponding to each of the recommended video frames in the recommended video segment, a clarity score of each of the recommended video frames includes: Sorting the gradient variance corresponding to each of the recommended video frames in the video clip to obtain a variance sequence; Dividing the variance sequence into a preset number of variance intervals, and obtaining a first value corresponding to each of the variance intervals; For each variance interval, a first value corresponding to the variance interval is determined as the clarity score of the recommended video frame corresponding to each gradient variance in the variance interval.

10. The method according to claim 7, characterized in that Determining the clarity score of each of the recommended video frames includes: Segmenting the recommended video to obtain multiple recommended video segments; For each recommended video frame in each recommended video segment, performing frequency domain conversion on the recommended video frame to obtain a frequency spectrum of the recommended video frame; dividing the spectrum image into a first region and a second region, wherein a change in image information of the first region is greater than a change in image information of the second region; Determining an energy proportion of a first region corresponding to the recommended video frame; For each of the recommended video segments, a clarity score of each of the recommended video frames is determined based on an energy proportion corresponding to each of the recommended video frames in the recommended video segment.

11. The method according to claim 7, characterized in that Determining the content relevance score of each of the recommended video frames based on the topic information includes: Segmenting the recommended video to obtain multiple recommended video segments; For each recommended video frame in each of the recommended video clips, performing text recognition on the recommended video frame to obtain a recognition result; When it is determined based on the recognition result that a first text is recognized, determining a first similarity based on the first text and the subject information; For each of the recommended video segments, a content relevance score of each of the recommended video frames in the recommended video segment is determined based on the first similarity corresponding to each of the recommended video frames.

12. The method according to claim 11, characterized in that The determining a first similarity based on the first text and the subject information includes: Acquire a first language of the first text, and acquire a second language of the subject text in the subject information; When the first language and the second language are the same, determining a first text vector of the first text and a second text vector of the subject text; A first similarity is determined based on the first text vector and the second text vector.

13. The method according to claim 12, characterized in that The determining a first similarity based on the first text and the subject information includes: When the first language and the second language are different, translating the first text based on the second language to obtain a second text; determining a third text vector for the second text; A first similarity is determined based on the third text vector and the second text vector.

14. The method according to claim 7, wherein: Determining the content relevance score of each of the recommended video frames based on the topic information includes: Segmenting the recommended video to obtain multiple recommended video segments; For each recommended video frame in each of the recommended video clips, performing semantic understanding on the recommended video frame to obtain a semantic understanding result; determining a second similarity based on the semantic understanding result and the topic information; For each of the recommended video segments, a content relevance score of each of the recommended video frames in the recommended video segment is determined based on the second similarity corresponding to each of the recommended video frames.

15. The method according to any one of claims 7 to 14, characterized in that The determining, based on the clarity score and the content relevance score of each recommended video frame, a quality score corresponding to each recommended video frame includes: For each of the recommended video frames, the clarity score and the content relevance score are determined as the quality score corresponding to the recommended video frame, or, The clarity score and the content relevance score are fused to obtain a quality score corresponding to the recommended video frame.

16. A video processing device, characterized in that: The device comprises: a video playing module, configured to obtain video data to be played in response to a received video playing instruction, and play the video based on the video data to be played, wherein the video data to be played includes recommended video data; A first determining module is configured to, in response to receiving a playback progress adjustment instruction during video playback, determine a first video frame corresponding to the playback progress adjustment instruction; a first acquisition module configured to, when the first video frame is a recommended video frame, acquire metadata corresponding to the first video frame from the recommended video data, the metadata including quality scores of multiple candidate video frames, the candidate video frames including the first video frame and multiple second video frames following the first video frame; The video playback module is further configured to determine a target video frame from the plurality of candidate video frames based on the quality scores of the plurality of candidate video frames, and continue playing the video starting from the target video frame.

17. A video processing device, characterized in that: The device comprises: A score determination module, configured to obtain a recommended video to be inserted and determine a quality score corresponding to each recommended video frame in the recommended video; a first generating module, configured to generate metadata of each recommended video frame based on the quality score of the recommended video frame and the quality scores of multiple recommended video frames following the recommended video frame; a second generating module, configured to generate recommended video data based on each of the recommended video frames and each of the recommended video frames; The splicing and encoding module is used to obtain the first video corresponding to the recommended video, and splice and encode the recommended video data and the first video to obtain video data to be played.

18. A computer device, characterized in that: The computer device comprises: a memory for storing computer-executable instructions or computer programs; The processor is configured to implement the video processing method according to any one of claims 1 to 5 or any one of claims 6 to 15 when executing the computer-executable instructions or computer program stored in the memory.

19. A computer-readable storage medium storing computer-executable instructions or a computer program, characterized in that: When the computer executable instructions or computer program are executed by a processor, the video processing method according to any one of claims 1 to 5 or any one of claims 6 to 15 is implemented.

20. A computer program product comprising computer executable instructions or a computer program, characterized in that When the computer executable instructions or computer program are executed by a processor, the video processing method according to any one of claims 1 to 5 or any one of claims 6 to 15 is implemented.

Images