Video processing method, device, computer equipment and storage medium

Abstract

The embodiment of the present application provides a video processing method, device, computer equipment and storage medium, the method comprising: obtaining an initial business video to be processed; performing video recognition processing on the initial business video to obtain transformation reference information of the initial business video; generating multiple candidate videos based on the transformation reference information and the initial business video, the multiple candidate videos are obtained by performing video transformation processing on the initial business video using the transformation reference information; and determining a target business video from the multiple candidate videos. By using the embodiment of the present application, video transformation can be performed flexibly.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a video processing method, apparatus, computer equipment and storage medium. Background Art

[0002] In some video application scenarios, the video needs to be resized before it is published; for example, in the publishing scenario of an advertising video, the advertising video is a horizontal video. When the advertising video needs to be published to a device with a vertical screen (such as a smart phone, tablet computer, etc.), the advertising video can be resized and the horizontal advertising video can be converted into a vertical video before publishing; similarly, in the publishing scenario of an advertising video, the advertising video is a vertical video. When the advertising video needs to be published to a device with a horizontal screen (such as a desktop computer, laptop computer, etc.), the advertising video can be resized and the vertical advertising video can be converted into a horizontal video before publishing. At present, the video size conversion method is mainly concentrated in the fixed size conversion method of horizontal or vertical screen display. The fixed size conversion method makes the converted video style single and inflexible. Summary of the invention

[0003] The embodiments of the present application provide a video processing method, apparatus, computer equipment and storage medium, which can flexibly perform video transformation.

[0004] On the one hand, an embodiment of the present application provides a video processing method, the video processing method comprising:

[0005] Obtain the initial business video to be processed;

[0006] Performing video recognition processing on the initial business video to obtain transformation reference information of the initial business video;

[0007] Generate multiple candidate videos according to the transformation reference information and the initial business video, where the multiple candidate videos are obtained by performing video transformation processing on the initial business video using the transformation reference information;

[0008] Determine a target business video from multiple candidate videos.

[0009] On the other hand, an embodiment of the present application provides a video processing device, the video processing device comprising:

[0010] An acquisition unit, used for acquiring an initial business video to be processed;

[0011] A processing unit, used to perform video recognition processing on the initial business video to obtain transformation reference information of the initial business video;

[0012] The processing unit is further used to generate a plurality of candidate videos according to the transformation reference information and the initial service video, wherein the plurality of candidate videos are obtained by performing video transformation processing on the initial service video using the transformation reference information;

[0013] The processing unit is also used to determine a target business video from multiple candidate videos.

[0014] In one embodiment, the transformation reference information includes classification label information of the initial business video; the processing unit is used to perform video recognition processing on the initial business video, and when the transformation reference information of the initial business video is obtained, it is specifically used to perform the following steps:

[0015] Obtaining audio and video data and text data of the initial business video;

[0016] Acquire audio and video features of audio and video data and text features of text data;

[0017] Perform feature fusion processing on the audio and video features and the text features to obtain fusion features;

[0018] The initial business video is classified and identified based on the fusion features to obtain classification label information of the initial business video.

[0019] In one embodiment, the transformation reference information includes M scene time periods corresponding to the initial business video, where M is an integer greater than 1; the processing unit is used to perform video recognition processing on the initial business video, and when the transformation reference information of the initial business video is obtained, it is specifically used to perform the following steps:

[0020] Determine a plurality of target image frames from an initial business video;

[0021] Performing scene switching detection on each target image frame, and determining a scene switching image frame that meets the scene switching condition from each target image frame;

[0022] According to the time points corresponding to the image frames of each scene switching, the time range corresponding to the initial business video is divided into M scene time periods;

[0023] Among them, the M scene time periods correspond to the M scene video segments in the initial business video, one scene time period corresponds to one scene video segment, there are N scene video segments among the M scene video segments belonging to different scenes, and N is an integer greater than 1 and less than or equal to M.

[0024] In one embodiment, the transformation reference information includes target filling area information of the initial business video; the processing unit is used to perform video recognition processing on the initial business video, and when the transformation reference information of the initial business video is obtained, it is specifically used to perform the following steps:

[0025] Extracting a plurality of reference image frames from an initial business video;

[0026] Performing image filling detection on each reference image frame to obtain reference filling area information of each reference image frame;

[0027] If the areas indicated by the reference filling area information of each reference image frame match, the target filling area information is determined according to the reference filling area information of each reference image frame.

[0028] In one embodiment, the transformation reference information includes scene reference information of M scene video segments of the initial business video, where M is an integer greater than 1; the processing unit is used to perform video recognition processing on the initial business video, and when the transformation reference information of the initial business video is obtained, it is specifically used to perform the following steps:

[0029] Obtain M scene video segments from the initial business video;

[0030] Performing subject object detection on the first image frame of each scene video segment in the M scene video segments;

[0031] Determining scene reference information of each scene video segment according to a main object detection result of a first image frame of each scene video segment;

[0032] Wherein, for a target scene video segment among the M scene video segments, if a subject object detection result of a first image frame of the target scene video segment indicates that the first image frame of the target scene video segment contains a subject object, then the scene reference information of the target scene video segment includes: subject position area information of the subject object in each image frame of the target scene video segment, where the subject position area information is obtained after tracking the subject object in the target scene video segment;

[0033] If the main object detection result of the first image frame of the target scene video segment indicates that the first image frame of the target scene video segment does not contain the main object, the scene reference information of the target scene video segment includes: the second image frame of the target scene video segment or preset cropping area information of each image frame in the target scene video segment.

[0034] In one embodiment, the transformation reference information includes classification label information of the initial business video; the processing unit is used to generate multiple candidate videos according to the transformation reference information and the initial business video, and is specifically used to perform the following steps:

[0035] Obtain a video filling template that matches the classification label information of the initial business video;

[0036] The initial business video is filled with a video filling template to obtain a first candidate video among multiple candidate videos.

[0037] In one embodiment, the transformation reference information includes M scene time periods corresponding to the initial business video, where M is an integer greater than 1; the processing unit is configured to generate multiple candidate videos according to the transformation reference information and the initial business video, and is specifically configured to perform the following steps:

[0038] Perform video segmentation processing on the initial business video according to M scene time periods to obtain M scene video segments;

[0039] Using a video splicing template to perform video segment splicing processing on the M scene video segments to obtain a second candidate video from multiple candidate videos;

[0040] When the video splicing template is used to perform video segment splicing processing on the M scene video segments, there are at least two scene video segments whose positions in the video splicing template are different.

[0041] In one embodiment, the transformation reference information includes M scene time periods corresponding to the initial business video, where M is an integer greater than 1; the processing unit is configured to generate multiple candidate videos according to the transformation reference information and the initial business video, and is specifically configured to perform the following steps:

[0042] Perform video segmentation processing on the initial business video according to M scene time periods to obtain M scene video segments;

[0043] Filling image frames are extracted from the M scene video segments, and filling processing is performed on the initial business video according to the extracted filling image frames and the video filling template to obtain a third candidate video from the multiple candidate videos.

[0044] In one embodiment, the transformation reference information includes target filling area information of the initial business video; the processing unit is used to generate multiple candidate videos according to the transformation reference information and the initial business video, and is specifically used to perform the following steps:

[0045] Based on the target filling area information, defilling is performed on each image frame of the initial business video;

[0046] Performing image frame splicing processing on each image frame after the defilling processing, to obtain a fourth candidate video among the multiple candidate videos.

[0047] In one embodiment, the initial service video includes a first type of scene video segment and a second type of scene video segment, the first type of scene video segment includes a subject object, and the second type of scene video segment does not include a subject object; the transformation reference information includes first scene reference information and second scene reference information;

[0048] The processing unit is used to generate multiple candidate videos according to the transformation reference information and the initial business video, and is specifically used to perform the following steps:

[0049] According to the first scene reference information corresponding to each first type of scene video segment, a first transformation process is performed on each first type of scene video segment to obtain each first type of scene video segment after processing; and according to the second scene reference information corresponding to each second type of scene video segment, a second transformation process is performed on each second type of scene video segment to obtain each second type of scene video segment after processing;

[0050] The processed scene video segments of the first type and the processed scene video segments of the second type are spliced ​​in time sequence to obtain a fifth candidate video from the multiple candidate videos.

[0051] In one embodiment, the first scene reference information includes subject position area information of the subject object in each image frame of the first type of scene video segment; the processing unit is used to perform a first transformation process on each first type of scene video segment according to the first scene reference information corresponding to each first type of scene video segment, and when obtaining each processed first type of scene video segment, specifically to execute the following steps:

[0052] For any first-type scene video segment, determining target cropping area information of each image frame of the first-type scene video segment according to subject position area information of the subject object in each image frame of the first-type scene video segment;

[0053] Cropping each image frame of the first type of scene video segment according to target cropping region information of each image frame of the first type of scene video segment to obtain cropped image frames;

[0054] The cropped image frames are subjected to image frame splicing processing to obtain a processed first type of scene video segment.

[0055] In one embodiment, the second scene reference information includes a second image frame in a second type of scene video segment or preset cropping area information of an image frame in a second type of scene video segment; the processing unit is used to perform a second transformation process on each second type of scene video segment according to the second scene reference information corresponding to each second type of scene video segment, and when obtaining each second type of scene video segment after the process, specifically to execute the following steps:

[0056] For any second type of scene video segment, the second image frame in the second type of scene video segment is used to fill the second type of scene video segment to obtain a processed second type of scene video segment; or, according to preset cropping area information of each image frame in the second type of scene video segment, each image frame in the second type of scene video segment is cropped to obtain a processed second type of scene video segment.

[0057] On the other hand, an embodiment of the present application provides a computer device, which includes: a memory and a processor; the memory is used to store a computer program; the processor is used to execute the computer program to implement the above-mentioned video processing method.

[0058] Accordingly, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program. When the computer program is read and executed by a processor of a computer device, the computer device executes the above-mentioned video processing method.

[0059] Accordingly, an embodiment of the present application provides a computer program product or a computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device performs the above-mentioned video processing method.

[0060] In an embodiment of the present application, a video can be subjected to video recognition processing, and a video transformation processing can be dynamically performed on the video based on the result of the video recognition processing to obtain multiple candidate videos, so as to determine one or more transformed videos from the multiple candidate videos. The video transformation processing method is adapted to the video, and multiple videos can be flexibly transformed for selection based on the video recognition result of a video, thereby improving the diversity of video transformation. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0062] Figure 1a is a schematic diagram of a horizontal video provided in an embodiment of the present application;

[0063] Figure 1b is a schematic diagram of a vertical video provided in an embodiment of the present application;

[0064] Figure 2 It is a flowchart of a video processing method provided by an embodiment of the present application;

[0065] Figure 3 is a flowchart of another video processing method provided in an embodiment of the present application;

[0066] Figure 4 is a flowchart of another video processing method provided in an embodiment of the present application;

[0067] Figure 5a It is a flowchart of a video scene classification process provided by an embodiment of the present application;

[0068] Figure 5b It is a structural diagram of a scene switching detection model provided in an embodiment of the present application;

[0069] Figure 5c is a result schematic diagram of an image filling detection model provided in an embodiment of the present application;

[0070] Figure 5d It is a flowchart of a matching degree detection process provided by an embodiment of the present application;

[0071] Figure 5e It is a structural diagram of a subject object detection model provided in an embodiment of the present application;

[0072] Figure 5f It is a structural diagram of a subject object tracking model provided in an embodiment of the present application;

[0073] Figure 5g It is a flowchart of a video subject object recognition process provided by an embodiment of the present application;

[0074] Figure 6 is a flowchart of another video processing method provided in an embodiment of the present application;

[0075] Figure 7a This is a schematic diagram of the effect of a video filling strategy provided by an embodiment of the present application;

[0076] Figure 7b This is a schematic diagram of the effect of another video filling strategy provided by an embodiment of the present application;

[0077] Figure 7c This is a schematic diagram of the effect of another video filling strategy provided by an embodiment of the present application;

[0078] Figure 7d This is a schematic diagram of the effect of a video splicing strategy provided by an embodiment of the present application;

[0079] Figure 7e It is a schematic diagram of the effect of another video splicing strategy provided by an embodiment of the present application;

[0080] Figure 7f This is a schematic diagram of the effect of a video defilling strategy provided by an embodiment of the present application;

[0081] Figure 7g It is a schematic diagram of the effect of a video subject object tracking strategy provided by an embodiment of the present application;

[0082] Figure 7h This is a schematic diagram of the effect of a video cropping strategy provided by an embodiment of the present application;

[0083] Figure 8 is a scene schematic diagram of a video processing method provided in an embodiment of the present application;

[0084] Fig. 9 is a structural schematic diagram of a video processing device provided in an embodiment of the present application;

[0085] Fig.10 It is a structural diagram of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0086] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0087] A video is a sequence of image frames consisting of multiple image frames that are played continuously. According to the video layout, videos can include horizontal videos and vertical videos. The so-called horizontal video refers to a video whose horizontal size is larger than its vertical size. Figure 1a is a schematic diagram of a horizontal video provided in an embodiment of the present application, such as Figure 1a As shown, video 101 is displayed in the video playback interface 10, and video 101 is a horizontal video. A vertical video refers to a video whose horizontal size is smaller than its vertical size. Figure 1b is a schematic diagram of a vertical video provided in an embodiment of the present application, such as Figure 1b As shown, video 102 is displayed in the playback interface 10, and video 102 is a vertical video.

[0088] The video transformation process includes the process of transforming the video format, for example, performing video transformation process on a horizontal video to obtain a vertical video, performing video transformation process on a vertical video to obtain a horizontal video, etc. In the embodiment of the present application, the video to be transformed can be specifically subjected to analysis and processing such as content recognition and understanding, and a suitable video transformation template is determined according to the analysis result, and the video to be transformed and the suitable video transformation template are combined for processing to obtain one or more candidate videos for the user to select.

[0089] Based on the above-mentioned description of video and video transformation processing, an embodiment of the present application proposes a video processing scheme, which performs video transformation processing on the video based on the result of video recognition processing, and can flexibly perform video transformation. In one embodiment, the video processing scheme can be executed by a computer device, which can be a terminal or a server. The terminal mentioned here can be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, a car terminal, a smart TV, etc., but is not limited to this; the server mentioned here can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), and basic cloud computing services such as big data and artificial intelligence platforms.

[0090] Based on the above description, the video processing solution provided in the embodiment of the present application is introduced in more detail below in combination with the accompanying drawings.

[0091] Figure 2 1 is a flow chart of a video processing method provided by an embodiment of the present application. The video processing method can be executed by the aforementioned computer device. The embodiment of the present application introduces a video recognition processing strategy and a video transformation processing strategy. Figure 2 As shown, the video processing method may include the following steps S201-S204:

[0092] S201, obtain the initial business video to be processed. According to the content of the video, the initial business video can be any type of video, such as film and television video, variety video, self-media video, game video, advertising video, etc. Film and television video refers to: in a specified shooting scene, the performance process of characters and / or animals and the surrounding environment are recorded according to a pre-made script, and audio, special effects, etc. are added later; variety video refers to: a video that integrates multiple art forms and is entertaining; self-media video refers to: a video that the general public uses a camera to shoot a certain scene and publishes it through the Internet and other channels, such as vlog (video blog, video record / video blog / video network log); game video refers to: in the process of one or more player users playing the game, the game screen displayed on the terminal screen of any player user, or the game screen displayed on the terminal screen of the viewing user who watches the game process of any player user, is screen recorded; advertising video refers to: a video with advertising content. The initial business video may include multiple image frames with continuous playback time.

[0093] S202, perform video recognition processing on the initial business video to obtain transformation reference information of the initial business video. The video recognition processing process of the initial business video can be a video recognition processing process of the initial business video from multiple video recognition angles using multiple video recognition processing strategies. In one embodiment, one video recognition processing strategy among the multiple video recognition processing strategies corresponds to a video recognition angle; multiple video recognition processing strategies can be obtained, and each video recognition processing strategy can be used to perform video recognition processing on the initial business video according to the corresponding video recognition angle, and then the transformation reference information of the initial business video can be determined based on the video recognition results of the initial business video under each video recognition processing strategy.

[0094] The video recognition processing strategy refers to a specific strategy or method for performing video recognition processing on the initial business video. Multiple video recognition processing strategies may include but are not limited to: video tag classification strategy, video scene segmentation strategy, video filling recognition strategy, video subject object recognition strategy, etc. The introduction of each video recognition processing strategy is as follows:

[0095] (1) The video tag classification strategy corresponding to the video tag classification perspective can be used to identify the classification tag information of the initial business video. The classification tag information may include one or more category tags to which the initial business video belongs. The category tag can be used to indicate the video presentation form of the initial business video, or can also indicate the industry field to which the initial business video belongs.

[0096] For example, the video label classification strategy is used to identify that the category label of the initial business video is "financial dashboard". This category label can indicate that the initial business video belongs to the financial field, and then the video transformation template corresponding to "financial dashboard" or the video transformation template related to the financial field can be selected for video transformation processing; the video label classification strategy is used to identify that the category label of the initial business video is "writing test paper". This category label can indicate that the initial business video belongs to the education field, and then the video transformation template corresponding to "writing test paper" or the video transformation template related to the education field can be selected for video transformation processing.

[0097] For another example, the video label classification strategy is used to identify the category label of the initial business video as "filling", and this category label can indicate that the initial business video is a filled video; the video label classification strategy is used to identify the category label of the initial business video as "tail frame", and this category label can indicate that the video content presented by the initial business video contains the tail image frame of the initial business video (that is, the last image frame of the initial business video); the video label classification strategy is used to identify the category label of the initial business video as "mouth broadcast", and this category label can indicate that the initial business video is a broadcast video containing an announcer; the video label classification strategy is used to identify the category label of the initial business video as "situational drama", and this category label can indicate that the video presentation form of the initial business video is a situational drama. In some embodiments, the above-mentioned "filling" label, "tail frame" label, "mouth broadcast" label, etc. can be used in the subsequent video analysis and understanding process.

[0098] The above example describes the situation where a single category label is identified. In actual application scenarios, the number of identified category labels can also be two or more (for example, a combination of two or more). For example, the video label classification strategy is used to identify the category labels of the initial business video as "financial dashboard" and "filling", which indicates that the initial business video has been filled and the industry field described in the initial business video is the financial field.

[0099] (2) The video scene segmentation strategy corresponds to the video scene segmentation angle. The video scene segmentation strategy can be used to identify whether the initial business video is a single-scene video or a multi-scene video; a single-scene video refers to a video without scene switching, that is, the scene appearing in the video is a single scene, such as the scene appearing in the video is a single scene of the classroom; a multi-scene video refers to a video with scene switching, that is, there are at least two scenes appearing in the video, such as the scenes appearing in the video are the classroom and the playground, and the video switches between the two scenes of the classroom and the playground. If the video scene segmentation strategy is used to identify the initial business video as a multi-scene video, the video scene segmentation strategy can further determine the scene time period of each scene appearing in the initial business video, and can also divide the initial business video into multiple scene video segments according to the scene time period of each scene, and one scene time period corresponds to one scene video segment. If the initial business video is a multi-scene video, the video recognition result of the initial business video under the video scene segmentation strategy may include: the scene time point of each scene appearing in the initial business video.

[0100] (3) The video filling identification strategy corresponds to the video filling identification angle. The video filling identification strategy can be used to identify whether the initial business video is an unfilled video or a filled video; an unfilled video refers to a video without any filling content (such as pictures, color blocks, etc.), that is, the unfilled video only contains the video itself; a filled video refers to a video that has been filled with filling content, that is, the filled video contains two parts: the video and the filling content. If the video filling identification strategy is used to identify the initial business video as a filled video, the video filling identification strategy can further determine the target filling area information of the initial business video. The target filling area information refers to the information of the area occupied by the filling content in the initial business video. For example, the target filling area information may include the coordinate parameters of the area occupied by the filling content in the initial business video. If the initial business video is a filled video, the video identification result of the initial business video under the video filling identification strategy may include: the target filling area information of the initial business video.

[0101] (4) The video subject object recognition strategy corresponds to the video subject object recognition angle. The video subject object recognition strategy can be used to recognize that the initial business video is a subject object video or a non-subject object video; a subject object video refers to a video with a subject object, that is, the video contains a subject object, and the subject object can be, for example, a person, an animal, etc.; a non-subject object video refers to a video without a subject object, that is, the video does not contain a subject object. If the video subject object recognition strategy is used to recognize that the initial business video is a subject object video, the video subject object recognition strategy can further track the subject object in the initial business video to determine the subject position area information of the subject object in each image frame of the initial business video. The subject position area information refers to the information of the area occupied by the image block containing the subject object in the image frame. The subject position area information may include the coordinate parameters of the area occupied by the image block containing the subject object in the image frame. If the initial business video is a subject object video, the video recognition result of the initial business video under the video subject object recognition strategy may include: the subject position area information of the subject object in each image frame of the initial business video. If the initial business video is a non-subject object video, the video recognition result of the initial business video under the video subject recognition strategy may include: the preset cropping area information of the specified image frame of the initial business video or each image frame in the initial business video; the specified image frame may include the tail image frame (i.e., the last image frame) in the initial business video, any image frame in the initial business video or the first image frame of the initial business video, etc. The preset cropping area information refers to the information of the preset area that needs to be cropped, and the preset cropping area information may include the coordinate parameters of the preset area that needs to be cropped.

[0102] It should be noted that the above-mentioned video label classification strategy, video scene segmentation strategy, video filling identification strategy, and video subject object identification strategy can be independent of each other, and the above-mentioned four video identification processing strategies can be used to perform video identification processing for any initial business video to be processed. Alternatively, the above-mentioned video label classification strategy, video scene segmentation strategy, video filling identification strategy, and video subject object identification strategy can be interrelated, and other video identification processing strategies that can be used to perform video identification processing on the initial business video can be determined based on the video identification result of the initial business video under the video label classification strategy (i.e., the classification label information of the initial business video); for example, if the classification label information of the initial business video is "filling", indicating that the initial business video may be a filled video, it is determined that the video filling identification strategy can be triggered to perform video identification processing on the initial business video; if the classification label information of the initial business video is "oral broadcast", indicating that there may be a subject object in the initial business video, it is determined that the video subject object identification strategy can be triggered to perform video identification processing; if the classification label information of the initial business video is "situational drama", indicating that there may be scene switching in the initial business video, it is determined that the video scene segmentation strategy can be triggered to perform video identification processing.

[0103] The transformation reference information may be determined based on the video recognition results of the initial business video under each video recognition processing strategy. In one embodiment, the transformation reference information may include the video recognition results of the initial business video under one or more video recognition processing strategies, and the transformation reference information may be obtained by combining the video recognition results of the initial business video under each video recognition processing strategy.

[0104] For example, if the initial business video is a multi-scene video, the transformation reference information may include: the video recognition result of the initial business video under the video label classification strategy, and the video recognition result of the initial business video under the video scene segmentation strategy; that is, the transformation reference information may include: the classification label information of the initial business video, and the scene time period of each scene appearing in the initial business video. For another example, if the initial business video is a multi-scene video and a filled video, the transformation reference information may include: the video recognition result of the initial business video under the video label classification strategy, the video recognition result of the initial business video under the video scene segmentation strategy, and the video recognition result of the initial business video under the video filling recognition strategy; that is, the transformation reference information may include: the classification label information of the initial business video, the scene time period of each scene appearing in the initial business video, and the target filling area information of the initial business video. For another example, if the initial business video is a multi-scene video, a filled video, and a subject object video, the transformation reference information may include: the video recognition result of the initial business video under the video label classification strategy, the video recognition result of the initial business video under the video scene segmentation strategy, the video recognition result of the initial business video under the video filling recognition strategy, and the video recognition result of the initial business video under the video subject object recognition strategy; that is, the transformation reference information may include: the classification label information of the initial business video, the scene time period of each scene appearing in the initial business video, the target filling area information of the initial business video, and the subject position area information of the subject object in each image frame of the initial business video.

[0105] S203, generating multiple candidate videos according to the transformation reference information and the initial business video. After performing video recognition processing on the initial business video to obtain the transformation reference information of the initial business video, multiple candidate videos can be generated according to the transformation reference information and the initial business video. The multiple candidate videos can be obtained by performing video transformation processing on the initial business video using the transformation reference information.

[0106] In one embodiment, multiple video transformation processing strategies can be determined based on the transformation reference information, and multiple candidate videos can be obtained by performing multiple video transformation processes on the initial business video using multiple video transformation processing strategies. Multiple video transformation processing strategies may include, but are not limited to: video filling strategy, video splicing strategy, video defilling strategy, video subject object tracking strategy, video cropping strategy, and the like. For example, when the transformation reference information includes the classification label information of the initial business video or the specified image frame of the initial business video, the video transformation processing strategy can be determined to be a video filling strategy; when the transformation reference information includes the scene time point at which each scene appears in the initial business video, the video transformation processing strategy can be determined to be a video splicing strategy; when the transformation reference information includes the subject position area information of the subject object in each image frame of the initial business video, the video transformation processing strategy can be determined to be a video subject object tracking strategy; when the transformation reference information includes the target filling area information of the initial business video, the video transformation processing strategy can be determined to be a video defilling strategy; when the transformation reference information includes the preset cropping area information of the initial business video, the video transformation processing strategy can be determined to be a video cropping strategy. The introduction of each video transformation processing strategy is as follows:

[0107] (1) Video filling strategy refers to a strategy for filling the initial business video using a video filling template.

[0108] (2) The video stitching strategy can be applied to perform video transformation processing on multi-scene videos; when the initial business video is a multi-scene video, the initial business video can be divided into multiple scene video segments according to the scene time periods of each scene appearing in the initial business video; the video stitching strategy refers to a strategy of performing video segment stitching processing on each scene video segment of the initial business video using a video stitching template.

[0109] (3) The video defilling strategy can be applied to perform video transformation processing on the padded video; the video defilling strategy refers to a strategy for defilling the padded content in each image frame of the initial business video within the area indicated by the target filling area information, thereby removing the padded content in each image frame of the initial business video.

[0110] (4) The video subject object tracking strategy is applicable to performing video transformation processing on the subject object video; the video subject object tracking strategy refers to a strategy for retaining the image content located in the area indicated by the subject position area information in each image frame of the initial business video, and removing the image content located outside the area indicated by the subject position area information in each image frame of the initial business video.

[0111] (5) Video cropping strategy refers to a strategy for retaining the image content in each image frame of the initial business video that is located within the area indicated by the preset cropping area information, and removing the image content in each image frame of the initial business video that is located outside the area indicated by the preset cropping area information, that is, a strategy for cropping each image frame according to the preset cropping area information of each image frame of the initial business video.

[0112] It should be noted that the process of performing video transformation processing on the initial business video using the video transformation processing strategy to obtain the candidate video may include a front-end video transformation processing process (also referred to as a front-end rendering process) and a background video transformation processing process (also referred to as a background rendering process). Among them, the front-end video transformation processing process refers to the process of performing video transformation rendering on the initial business video using the player technology, which does not synthesize the real candidate video; for example, for the video filling strategy, the front-end video transformation processing is to use the video filling template to fill and display the initial business video, but the video filling template is not actually used to fill the initial business video to generate the candidate video; for another example, for the video splicing strategy, the front-end video transformation processing is to splice and display the scene video segments of each scene, but the scene video segments of each scene are not actually spliced ​​to generate the candidate video; the player technology can specifically be a rendering technology based on PAG (a rendering tool); through the front-end video transformation processing, the effect of the candidate video can be quickly previewed without generating the candidate video. Compared with the front-end video transformation processing, the background video transformation processing refers to the process of actually generating candidate videos through video transformation processing. That is, the candidate video does not exist as a preview or display, but as a video. In this way, the generated candidate video is more stable, which is conducive to the storage of the candidate video.

[0113] S204, determining a target business video from multiple candidate videos. The target business video may be one or more target candidate videos determined from multiple candidate videos, for example, a preview effect of multiple candidate videos may be displayed on an interface, and the user may select and determine the target business video on the interface. After the target business video is determined, business processing may be performed on the target business video, for example, publishing and playing the target business video.

[0114] In one embodiment, a computer device interface may be used to present multiple candidate videos (such as the preview of the candidate videos obtained by the front-end video conversion processing described above), and the target business video is one or more target candidate videos selected from the computer device interface, for example, the user selects one or more target candidate videos in the computer device interface; in this way, the candidate videos are presented in the interface for the user to select, and the user can select the target candidate video that meets the video conversion processing requirements from the candidate videos, thereby better meeting the user's video conversion processing requirements. In another embodiment, the target business video can be one or more target candidate videos randomly selected by the computer device from multiple candidate videos; in this way, the target business video can be quickly determined from the candidate videos, thereby effectively improving the video conversion efficiency.

[0115] In summary, the video processing method described in steps S201-S204 can be summarized as follows: Figure 3 The process diagram shown is as follows: Figure 3 As shown, the video processing method may include four stages, namely, video recognition processing, video transformation processing strategy determination, video transformation processing stage, and target business video determination; in the video recognition processing stage, multiple video recognition processing strategies may be used to perform multi-angle video recognition processing on the initial business video to obtain the video recognition results of the initial business video under each video recognition processing strategy, and the transformation reference information of the initial business video is determined according to the video recognition results of the initial business video under each video recognition processing strategy; in the video transformation processing strategy determination stage, multiple video transformation processing strategies may be determined according to the transformation reference information of the initial business video; in the video transformation processing stage, multiple video transformation processing strategies may be used to perform multiple video transformation processing on the initial business video to generate multiple candidate videos, and the video transformation processing stage may include a front-end video transformation processing process and a background video transformation processing process; in the target business video determination stage, one or more target candidate videos may be selected or randomly determined from multiple candidate videos, and the one or more target candidate videos may be determined as the target business video.

[0116] In the embodiment of the present application, by adopting multiple video recognition processing strategies to perform multi-angle video recognition processing on the initial business video, the multi-angle video recognition results of the initial business video can be obtained. Further, through the multi-angle video recognition results, a variety of and rich video transformation processing strategies can be determined, so that video transformation can be performed flexibly, thereby effectively improving the video transformation effect. In addition, the generation process of the candidate video can include two processes, front-end rendering and background rendering. The front-end rendering can quickly preview the transformation effect of the initial business video, and the background rendering is conducive to the stable storage of the candidate video.

[0117] Figure 4FIG. 1 is a flow chart of another video processing method provided by an embodiment of the present application. The video processing method can be executed by the aforementioned computer device. The embodiment of the present application mainly introduces the video recognition processing process of each video recognition processing strategy. Figure 4 As shown, the video processing method may include the following steps S401-S406:

[0118] S401, obtaining an initial business video to be processed.

[0119] The execution process of step S401 in the embodiment of the present application is similar to Figure 2 The execution process of step S201 in the embodiment shown is the same. For the specific execution process, please refer to Figure 2 The description of step S201 in the illustrated embodiment will not be repeated here.

[0120] S402, obtaining multiple video recognition processing strategies.

[0121] From the foregoing content, it can be seen that multiple video recognition processing strategies may include but are not limited to: video tag classification strategy, video scene segmentation strategy, video filling recognition strategy, video subject object recognition strategy, etc.; one video recognition processing strategy corresponds to a video recognition angle; the video tag classification strategy corresponds to the video tag classification angle, the video scene segmentation strategy corresponds to the video scene segmentation angle, the video filling recognition strategy corresponds to the video filling recognition angle, and the video subject object recognition strategy corresponds to the video subject object recognition angle.

[0122] S403: adopt various video recognition processing strategies to perform video recognition processing on the initial business video according to the corresponding video recognition angle.

[0123] S404: Determine transformation reference information of the initial business video according to the video recognition results of the initial business video under each video recognition processing strategy.

[0124] In steps S403-S404, after obtaining multiple video recognition processing strategies, each video recognition processing strategy can be used to perform video recognition processing on the initial business video according to the corresponding video recognition angle, and then the transformation reference information of the initial business video can be determined according to the video recognition results of the initial business video under each video recognition processing strategy. The video recognition processing process of each video recognition processing strategy is introduced below:

[0125] (1) The video recognition processing strategy may include a video label classification strategy. Using the video label classification angle corresponding to the video label classification strategy, the process of performing video recognition processing on the initial business video may include: obtaining the audio and video data and text data of the initial business video; obtaining the audio and video features of the audio and video data and the text features of the text data; performing feature fusion processing on the audio and video features and the text features to obtain fusion features; performing classification and recognition processing on the initial business video based on the fusion features to obtain classification label information of the initial business video. Here, the video recognition result of the initial business video under the video label classification strategy may include the classification label information of the initial business video; the transformation reference information may include the video recognition result of the initial business video under the video label classification strategy; that is, the transformation reference information may include the classification label information of the initial business video.

[0126] Among them, audio and video data refers to the images and audio and other contents contained in the initial business video, and the audio and video data may include: multiple scene recognition image frames determined from the initial business video, the cover image of the initial business video, and the audio extracted from the initial business video; the multiple scene recognition image frames may be each image frame in the initial business video, or the multiple scene recognition image frames may be image frames extracted from the initial business video according to the first image frame extraction strategy (for example, one image frame is extracted every preset time period (for example, it can be 1 second)), and the embodiments of the present application do not limit this. Text data refers to the text content appearing in the initial business video, and the text data may be obtained by identifying the initial business video.

[0127] In one embodiment, obtaining audio-visual features of audio-visual data may refer to obtaining features of each part of the content contained in the audio-visual data. The process of obtaining audio-visual features of audio-visual data may include: performing image recognition processing on each scene recognition image frame to obtain image features of each scene recognition image frame, and determining the video features of the initial business video according to the image features of each scene recognition image frame, and the video features may be specifically represented by a video feature representation vector; performing image recognition processing on the cover image to obtain the cover features of the initial business video, and the cover features may be specifically represented by a cover feature representation vector; and performing audio recognition processing on the audio contained in the initial business video to obtain the audio features of the initial business video, and the audio features may be specifically represented by an audio feature representation vector; the audio-visual features may include video features, cover features, and audio features. The process of obtaining text features of text data may include: performing text recognition processing on the text content appearing in the initial business video to obtain the text features of the initial business video, and the text features may be represented by a text feature representation vector.

[0128] It should be noted that the image recognition processing process can be specifically performed using an image recognition model. The image recognition processing model can be, for example, a residual network (for example, Resnet50). The residual network is a convolutional neural network for learning image features. The audio recognition processing process can be specifically performed using an audio recognition processing model. The audio recognition processing model can be, for example, a VGGish network. The VGGish network is a convolutional neural network for learning audio features. The text recognition processing process can be specifically performed using a text recognition processing model. The text recognition processing model can be, for example, a Bert network. The Bert network is an NLP (Natural Language Processing) model for learning text features. The image recognition network, the audio recognition network, and the text recognition network can be trained using multiple training videos annotated with classification label information.

[0129] The video recognition processing process based on the video label classification strategy can be found in Figure 5a ,like Figure 5a As shown, for the initial business video, the scene recognition image frame, cover image, audio and text content of the initial business video can be obtained; then the video feature representation vector of the initial business video can be obtained based on the scene recognition image frames of the initial business video, the cover feature representation vector of the initial business video can be obtained based on the cover image, the audio feature representation vector of the initial business video can be obtained based on the audio, and the text feature representation vector of the initial business video can be obtained based on the text content; then the video feature representation vector, cover feature representation vector, audio feature representation vector and text feature representation vector of the initial business video can be fused to obtain the fused feature representation vector of the initial business video; then classification and recognition processing can be performed based on the fused feature representation vector of the initial business video to obtain classification label information of the initial business video.

[0130] (2) The video recognition processing strategy may include a video scene segmentation strategy. Using the video scene segmentation angle corresponding to the video scene segmentation strategy, the process of performing video recognition processing on the initial business video may include: determining multiple target image frames from the initial business video; performing scene switching detection on each target image frame, and determining a scene switching image frame that meets the scene switching condition from each target image frame; dividing the time range corresponding to the initial business video into M scene time periods according to the time points corresponding to each scene switching image frame, where M is an integer greater than 1. Here, the video recognition result of the initial business video under the video scene segmentation strategy may include the M scene time periods corresponding to the initial business video; the transformation reference information may include the video recognition result of the initial business video under the video scene segmentation strategy; that is, the transformation reference information may include the M scene time periods corresponding to the initial business video.

[0131] The multiple target image frames may be image frames in the initial business video, or the multiple target image frames may be image frames extracted from the initial business video according to a second image frame extraction strategy (e.g., extracting an image frame every preset time period (e.g., 1 second)). The M scene time periods correspond to the M scene video segments in the initial business video, one scene time period corresponds to one scene video segment, and N of the M scene video segments belong to different scenes, that is, at least two of the M scene video segments belong to different scenes, that is, at least two of the M scene video segments have different classification label information, and N is an integer greater than 1 and less than or equal to M.

[0132] In one embodiment, a process of performing scene switching detection on each target image frame and determining a scene switching image frame that meets a scene switching condition from each target image frame may include: for any target image frame among each target image frame, a scene similarity between the target image frame and a previous target image frame of the target image frame may be calculated; the greater the scene similarity between the two target image frames, the smaller the possibility of a scene switching between the two target image frames, and the smaller the scene similarity between the two target image frames, the greater the possibility of a scene switching between the two target image frames; if the scene similarity is less than a scene similarity threshold, the target image frame may be determined as a scene switching image frame that meets the scene switching condition.

[0133] Based on the above content, the process of dividing the time range corresponding to the initial business video into M scene time periods according to the time points corresponding to each scene switching image frame may include: the time point corresponding to the scene switching image frame may be used as the starting time point, and the time point corresponding to the previous image frame of the scene switching image frame may be used as the ending time point, and further, the time range corresponding to the initial business video may be divided into M scene time periods according to the determined starting time points and the ending time points. For example, the target image frames extracted from the initial business video include target image frame 1, target image frame 2, target image frame 3, target image frame 4 and target image frame 5. Target image frame 3 is a scene switching image frame. The time point corresponding to the scene switching image frame can be used as the starting time point, that is, the time point corresponding to target image frame 3 is used as the starting time point, and the time point corresponding to the previous image frame of target image frame 3 (that is, target image frame 2) is used as the ending time point; thereby, the time range corresponding to the initial business video can be divided into two scene time periods according to the starting time point and the ending time point; the first scene time period starts with the time point corresponding to target image frame 1 and ends with the time point corresponding to target image frame 2; the second scene time period starts with the time point corresponding to target image frame 3 and ends with the time point corresponding to target image frame 5.

[0134] It should be noted that the process of performing scene switching detection on each target image frame can be performed using a scene switching detection model. The input data of the scene switching detection model is each target image frame. The output of the scene switching detection model is the scene switching label information of each target image frame. The scene switching label information can be used to indicate whether the corresponding target image frame meets the scene switching condition. The scene switching detection model can be, for example, a TransNet network, which is a convolutional neural network for learning image features. The structure of an exemplary scene switching detection model can be found in Figure 5b ,like Figure 5b As shown, the scene switching detection model may include multiple convolutional layers ( Figure 5b Taking 4 convolutional layers as an example), concatenation layers, pooling layers, multiple fully connected layers ( Figure 5b Taking two fully connected layers as an example), and an activation layer, multiple convolutional layers can perform feature learning on the target image frame respectively, the splicing layer can perform feature splicing on the features learned by each convolutional layer, the pooling layer (for example, it can be a maximum pooling layer) can downsample the features spliced ​​by the splicing layer, and multiple fully connected layers can perform feature integration processing on the features after the downsampling processing. The activation layer can activate the features after the feature integration processing and determine the scene switching label information of the target image frame according to the activated features.

[0135] (3) The video recognition processing strategy may include a video filling recognition strategy. Using the video filling recognition angle corresponding to the video filling recognition strategy, the process of performing video recognition processing on the initial business video may include: extracting multiple reference image frames from the initial business video; performing image filling detection on each reference image frame to obtain reference filling area information of each reference image frame; if the areas indicated by the reference filling area information of each reference image frame match, determining the target filling area information according to the reference filling area information of each reference image frame. Here, the video recognition result of the initial business video under the video filling recognition strategy may include the target filling area information of the initial business video; the transformed reference information may include the video recognition result of the initial business video under the video filling recognition strategy; that is, the transformed reference information may include the target filling area information of the initial business video.

[0136] Among them, multiple reference image frames can be image frames in the initial business video, or multiple reference image frames can be image frames extracted from the initial business video according to a third image frame extraction strategy (for example, one image frame is extracted every preset time period, and a total of P image frames are extracted, where P is a positive integer); it should be noted that the third image frame extraction strategy here and the aforementioned first image frame extraction strategy and second image frame extraction strategy can be the same extraction strategy or different extraction strategies, and the embodiments of the present application are not limited to this.

[0137] In one embodiment, the process of performing image filling detection on each reference image frame to obtain the reference filling area information of each reference image frame may be performed using an image filling detection model, and the image filling detection model may be, for example, a Pixel-Link model. An exemplary model structure of an image filling detection model and an image filling detection process may be referred to in Figure 5c ,like Figure 5c As shown, the video filling recognition model may include a convolutional network and a classification network; the number of convolutional networks may be multiple, and the convolutional network may be used for feature learning; the number of classification networks may be two, for example, they may be a first classification network and a second classification network, respectively, and the first classification network and the second classification network may perform feature classification from different angles, the first classification network may perform feature classification from a pixel filling angle (Pixel angle), and the second classification network may perform feature classification from a pixel connection angle (Link angle).

[0138] based on Figure 5c The model structure of the image filling detection model shown in the figure, using the image filling detection model to perform image filling detection on each reference image frame, and the process of obtaining the reference filling area information of each reference image frame may include: for any reference image frame 501 in each reference image frame, multiple convolution networks may be used to perform feature learning on the reference image frame to obtain reference features; further, a first classification network may be used to perform feature classification on the reference features to obtain the feature classification results of the first classification network, and a second classification network may be used to perform feature classification on the reference features to obtain the feature classification results of the second classification network; then the feature classification results of the first classification network and the feature classification results of the second classification network are fused to obtain the reference filling area information of the reference image frame. The reference filling area information of the reference image frame can be used to indicate the information of the area occupied by the filling content in the reference image frame, for example, the reference filling area information of the reference image frame may include the coordinate parameters of the area occupied by the filling content in the reference image frame. After obtaining the reference filling area information of the reference image frame, the reference filling area information can also be used to perform cropping processing on the reference image frame to obtain the cropped reference image frame 502.

[0139] Further, after determining the reference filling area information of each reference image frame, the area indicated by the reference filling area information of each reference image frame can be detected for matching. If the areas indicated by the reference filling area information of each reference image frame match, the target filling area information of the initial business video can be determined based on the areas indicated by the reference filling area information of each reference image frame. The process of performing matching detection on the areas indicated by the reference filling area information of each reference image frame can be found in Figure 5d ,like Figure 5d As shown:

[0140] The regional similarity between the regions indicated by the reference filling region information of each reference image frame can be calculated, and the regional similarity here can be determined according to the mean and / or variance of the region size indicated by the reference filling region of each reference image frame. If the regional similarity between the regions indicated by the reference filling region information of each reference image frame is less than the regional similarity threshold, it can be determined that the regions indicated by the reference filling region information of each reference image frame do not match. If the regional similarity between the regions indicated by the reference filling region information of each reference image frame is greater than or equal to the regional similarity threshold, the reference filling region information of each reference image frame can be voted, and the number of occurrences of each reference filling region information in the reference filling region information of each reference image frame can be counted; if the voting result shows that multiple reference filling region information appears, it is determined that the regions indicated by the reference filling region information of each reference image frame do not match; if the voting result shows that only one reference filling region information appears, it can be determined that the regions indicated by the reference filling region information of each reference image frame match. Here, determining the target filling area information of the initial service video according to the area indicated by the reference filling area information of each reference image frame may include: determining the reference filling area information that only appears in the voting result as the target filling area information.

[0141] (4) The video recognition processing strategy may include a video subject object recognition strategy. When the initial business video is a multi-scene video, the subject objects appearing in the various scene video segments contained in the initial business video may be inconsistent. For example, a first subject object appears in a first scene video segment, and a second subject object appears in a second scene video segment. The first subject object and the second subject object are not the same subject object. Alternatively, a part of the scene video segments contained in the initial business video may contain the subject object, while another part of the scene video segments may not contain the subject object. Based on this, the initial business video may be divided into multiple scene video segments, and then the video subject object recognition strategy may be used to perform video recognition processing on each scene video segment.

[0142] In one embodiment, the video subject object recognition angle corresponding to the video subject object recognition strategy, and the process of performing video recognition processing on the initial business video using the video subject object recognition strategy may include: obtaining M scene video segments from the initial business video; performing subject object detection on the first image frame of each scene video segment in the M scene video segments; and determining the scene reference information of each scene video segment based on the subject object detection result of the first image frame of each scene video segment. Here, the video recognition result of the initial business video under the video subject object recognition strategy may include the scene reference information of each scene video segment; the transformation reference information may include the video recognition result of the initial business video under the video subject object recognition strategy; that is, the transformation reference information may include the scene reference information of each scene video segment.

[0143] Among them, for a target scene video segment among the M scene video segments, the target scene video segment is any scene video segment among the M scene video segments; if the subject object detection result of the first image frame of the target scene video segment indicates that the first image frame of the target scene video segment contains the subject object, then the scene reference information of the target scene video segment may include: subject position area information of the subject object in each image frame of the target scene video segment, and the subject position area information is obtained after tracking the subject object in the target scene video segment; if the subject object detection result of the first image frame of the target scene video segment indicates that the first image frame of the target scene video segment does not contain the subject object, then the scene reference information of the target scene video segment may include: the second image frame of the target scene video segment or preset cropping area information of each image frame in the target scene video segment.

[0144] The first image frame may be the first image frame (i.e., the first image frame) of each scene video segment. Subject object detection for the first image frame of each scene video segment may be performed using a subject object detection model. For example, the subject object detection model may be a BMPM model, which is a convolutional neural network for salient subject detection. An exemplary structure of a subject object detection model can be found in Figure 5e ,like Figure 5eAs shown, the subject object detection model may include a convolutional layer, MCFEM (Multi-scale Context-aware Feature Extraction Module), and GBMPM (Gated Bi-directional Message Passing Module). The input data of the subject object detection model is the first image frame, the convolutional layer can be used to perform feature learning on the first image frame, the MCFEM can be used to extract context information from the multi-level feature map learned by the convolutional layer, the GBMPM can be used to propagate information between the multi-level feature maps obtained by the context information extraction, and the gate function in the GBMPM can be used to control the information passing rate, so that the feature map after the information transmission encodes the semantic information and combines the spatial details, and finally the feature maps are fused from the deep layer to the shallow layer to obtain the final saliency feature map, which may include the subject position area information of the subject object in the first image frame. Among them, Indicates that the feature map propagates from shallow to deep layers. Indicates that the feature map propagates from deep layers to shallow layers. Indicates combination and The obtained feature map.

[0145] The process of tracking the subject object in the target scene video segment and obtaining the subject position area information of the subject object in each image frame of the target scene video segment can be performed using a subject object tracking model. For example, the subject object tracking model can be an ATOM model, which is a network model that can be used for subject object tracking. An exemplary structure of the subject object tracking model and the training process of the subject object tracking model can be found in Figure 5f ,like Figure 5fAs shown, the subject object tracking model includes a residual network (such as ResNet-18, ResNet-16, etc.), an IOU (Intersection-over-Union) prediction module (IOU Modulation and IOU Predictor) and a classification network (Classifier); wherein the residual network is used to perform feature learning on the reference image and the test image, the IOU prediction module can be used to calculate the IOU value between the marked box of the reference image and the candidate box of the test image, the classification network can be used to enhance the discrimination of the classifier to distinguish the subject object from other objects in the scene, and can output a confidence map of the position of the subject object; the area selected by the marked box is the area indicated by the subject position area information of the subject object in the reference image, and the area selected by the candidate box is the area indicated by the subject position area information of the subject object in the test image.

[0146] If the main object detection result of the first image frame of the target scene video segment indicates that the first image frame of the target scene video segment does not contain the main object, the video label classification strategy can be used to perform video recognition processing on the target scene video segment to obtain the classification label information of the target scene video segment; if the classification label information of the target scene video segment indicates that the target scene video segment contains a second image frame, the scene reference information of the target scene video segment may include the second image frame of the target scene video segment, and the second image frame may be the tail image frame (i.e., the last image frame) of each scene video segment; if the classification label information of the target scene video segment indicates that the target scene video segment contains a second image frame If the segment does not contain the second image frame, the scene reference information of the target scene video segment may include preset cropping area information of each image frame in the target scene video segment; wherein the area sizes indicated by the preset cropping area information of each image frame are the same, that is, the horizontal size and vertical size of the area indicated by the preset cropping area information of each image frame are the same, but there are at least two area positions indicated by the preset cropping area information in the preset cropping area information of each image frame that are different, for example, the area indicated by the first preset cropping area information is located on the left side of the image frame, and the area indicated by the second preset cropping area information is located on the right side of the image frame.

[0147] In summary, for a single-scene video, the main object detection can be directly performed on the first image frame of the single-scene video. If the main object detection result of the first image frame of the single-scene video indicates that the first image frame of the single-scene video contains the main object, the main object can be tracked in the single-scene video to determine the main position area information of the main object in each image frame of the single-scene video. If the main object detection result of the first image frame of the single-scene video indicates that the first image frame of the single-scene video does not contain the main object, the video label classification strategy can be used to obtain the classification label information of the single-scene video; if the classification label information of the single-scene video indicates that the single-scene video contains a second image frame, the scene reference information of the single-scene video may include the second image frame of the single-scene video, and the second image frame may be the tail image frame (i.e., the last image frame) of each scene video segment; if the classification label information of the single-scene video indicates that the second image frame is not contained in the single-scene video, the scene reference information of the single-scene video may include the preset cropping area information of each image frame in the target scene video segment. For multi-scene videos, the video recognition process of the video main object recognition strategy can be referred to. Figure 5g The flowchart shown in the figure can divide the multi-scene video into multiple scene video segments, and then perform video recognition processing on each scene video segment according to the video recognition processing process of the single scene video.

[0148] S405: Generate multiple candidate videos according to the transformation reference information and the initial service video.

[0149] The execution process of step S405 in the embodiment of the present application is similar to Figure 2 The execution process of step S203 in the illustrated embodiment is the same. For the specific execution process, please refer to Figure 2 The description of step S203 in the illustrated embodiment will not be repeated here.

[0150] S406: Determine a target business video from multiple candidate videos.

[0151] The execution process of step S406 in the embodiment of the present application is similar to Figure 2 The execution process of step S204 in the illustrated embodiment is the same. For the specific execution process, please refer to Figure 2 The description of step S204 in the illustrated embodiment will not be repeated here.

[0152] In an embodiment of the present application, multiple video recognition and processing strategies are used to perform multi-angle video recognition processing on the initial business video. This can accurately and comprehensively identify and understand the video content of the initial business video. Therefore, when video transformation processing is performed according to the reference transformation information obtained from the video recognition, the video transformation processing method is more compatible with the video, which can improve the video transformation effect.

[0153] Figure 6 FIG. 1 is a flow chart of another video processing method provided by an embodiment of the present application. The video processing method can be executed by the aforementioned computer device. The embodiment of the present application mainly introduces the video transformation processing process of each video transformation processing strategy. Figure 6 As shown, the video processing method may include the following steps S601-S607:

[0154] S601, obtaining an initial business video to be processed.

[0155] The execution process of step S401 in the embodiment of the present application is similar to Figure 2 The execution process of step S201 in the embodiment shown is the same. For the specific execution process, please refer to Figure 2 The description of step S201 in the illustrated embodiment will not be repeated here.

[0156] S602, obtaining multiple video recognition processing strategies.

[0157] The execution process of step S602 in the embodiment of the present application is similar to Figure 4 The execution process of step S401 in the illustrated embodiment is the same. For the specific execution process, please refer to Figure 4 The description of step S401 in the illustrated embodiment will not be repeated here.

[0158] S603: adopt various video recognition processing strategies to perform video recognition processing on the initial service video according to the corresponding video recognition angle.

[0159] The execution process of step S603 in the embodiment of the present application is similar to Figure 4 The execution process of step S403 in the illustrated embodiment is the same. For the specific execution process, please refer to Figure 4 The description of step S403 in the illustrated embodiment will not be repeated here.

[0160] S604: Determine transformation reference information of the initial business video according to the video recognition results of the initial business video under each video recognition processing strategy.

[0161] The execution process of step S604 in the embodiment of the present application is similar to Figure 4 The execution process of step S404 in the illustrated embodiment is the same. For the specific execution process, please refer to Figure 4 The description of step S404 in the illustrated embodiment will not be repeated here.

[0162] S605: Determine multiple video transformation processing strategies according to the transformation reference information.

[0163] In one embodiment, multiple video transformation processing strategies can be determined according to the transformation reference information. The multiple video transformation processing strategies can include but are not limited to: video filling strategy, video splicing strategy, video defilling strategy, video main object tracking strategy, video cropping strategy, etc.

[0164] S606: Use multiple video transformation processing strategies to perform video transformation processing on the initial service video, thereby obtaining multiple candidate videos.

[0165] The following introduces the video transformation processing processes using video filling strategy, video splicing strategy, video defilling strategy, video subject object tracking strategy, video cropping strategy, etc., as follows:

[0166] (1) Video filling strategy.

[0167] As can be seen from the foregoing, the video filling strategy refers to a strategy for performing filling processing on the initial service video using a video filling template.

[0168] In one embodiment, the transformation reference information may include classification label information of the initial business video. In this case, the video filling strategy is used to perform video transformation processing on the initial business video. The process of obtaining the first candidate video among multiple candidate videos may include: obtaining a video filling template that matches the classification label information of the initial business video; the video filling template here may be a first video filling template, and the first video filling template may include a first filling area, a second filling area, and a third filling area; the first filling area and the third filling area may contain preset filling images, and the preset filling images match the classification label information of the initial business video, and the second filling area can be used to fill the initial business video; the initial business video is filled using the video filling template to obtain the first candidate video among multiple candidate videos; wherein, the classification label information of the initial business video matches the video filling template means that the classification label information of the initial business scene matches the classification label information of the video filling template.

[0169] The video filling processing process based on the first video filling template can be found in Figure 7a , Figure 7aThe category label included in the classification label information of the initial business video 701 on the left is "writing a test paper", indicating that the initial business video belongs to the field of education. The first video filling template matching the classification label information of the initial business video includes an upper area (i.e., a first filling area), a middle area (i.e., a second filling area), and a lower area (i.e., a third filling area). The upper area and the lower area contain a preset filling image 702. The image content of the preset filling image 702 includes a student, that is, the preset filling image matches the classification label information of the initial business video. After the initial business video 701 is filled in the middle area of ​​the first video filling template, a first candidate video 703 is obtained;

[0170] Figure 7a The category label included in the classification label information of the initial business video 701 on the right is "Financial Dashboard", indicating that the initial business video belongs to the financial field. In the first video filling template that matches the classification label information of the initial business video, the upper and lower areas contain preset filling images 702. The image content of the preset filling image 702 is related to the financial field, that is, the preset filling image matches the classification label information of the initial business video. After the initial business video 701 is filled in the middle area of ​​the first video filling template, the first candidate video 703 is obtained.

[0171] In one embodiment, the transformation reference information may include M scene time periods corresponding to the initial business video, where M is an integer greater than 1; in this case, the process of performing video transformation processing on the initial business video using a video filling strategy to obtain a third candidate video from multiple candidate videos may include: performing video segmentation processing on the initial business video according to the M scene time periods to obtain M scene video segments; extracting filling image frames from the M scene video segments, and performing filling processing on the initial business video according to the extracted filling image frames and a video filling template to obtain a third candidate video from multiple candidate videos; the video filling template here may be a second video filling template; the second video filling template may include a first filling area, a second filling area, and a third filling area; the first filling area and the third filling area may be filled with a filling image frame, and the second filling area may be filled with the initial business video. Among them, the filling image frame may be any image frame in the scene video segment, or the filling image frame may be the first image frame in the scene video segment, or the filling image frame may be the last image frame in the scene video segment, and so on.

[0172] The filling process based on the filling image frame and the second video filling template can be found in Figure 7bThe second video filling template includes an upper area (i.e., a first filling area), a middle area (i.e., a second filling area) and a lower area (i.e., a third filling area). The filling image frame 704 and the filling image frame 705 are extracted from the initial business video 701. The filling image frame 704 is filled in the upper area of ​​the second video filling template, the filling image frame 705 is filled in the lower area of ​​the second video filling template, and the initial business video 701 is filled in the middle area of ​​the second video filling template to obtain the third candidate video 706.

[0173] In another embodiment, the transformation reference information may include M scene time periods corresponding to the initial business video, where M is an integer greater than 1; in this case, the process of performing video transformation processing on the initial business video using a video filling strategy to obtain a third candidate video from multiple candidate videos may include: performing video segmentation processing on the initial business video according to the M scene time periods to obtain M scene video segments; extracting filling image frames from the M scene video segments, and performing filling processing on the initial business video according to the extracted filling image frames and the video filling template to obtain the third candidate video from the multiple candidate videos. The video filling template here may be a third video filling template; performing filling processing on the initial business video according to the extracted filling image frames and the third video filling template to obtain the third candidate video from the multiple candidate videos may include: performing Gaussian blur processing on the extracted filling image frames using the third video filling template, and superimposing the initial business video on the upper layer of the filling image frames after Gaussian blur processing to obtain the third candidate video. Among them, the filling image frame may be any image frame in the scene video segment, or the filling image frame may be the first image frame in the scene video segment, or the filling image frame may be the last image frame in the scene video segment, and so on.

[0174] The video filling process based on the third video filling template and the filled image frame can be found in Figure 7c The candidate video 708 is obtained by superimposing the initial business video 701 on the upper layer of the filling image frame 707 after Gaussian blur processing. The filling image frame 707 is extracted from the initial business video 701. The filling image frame 707 is Gaussian blurred using the third video filling template to obtain the filling image frame 707 after Gaussian blur processing. The initial business video 701 is superimposed on the upper layer of the filling image frame 707 after Gaussian blur processing to obtain the candidate video 708.

[0175] (2) Video splicing strategy.

[0176] From the above content, it can be seen that the video stitching strategy refers to a strategy of using a video stitching template to perform video segment stitching processing on the video segments of each scene of the initial business video. The so-called video segment stitching processing refers to the spatial stitching processing of the video segments of each scene.

[0177] In one embodiment, the transformation reference information may include M scene time periods corresponding to the initial business video, where M is an integer greater than 1; in this case, the process of using a video splicing strategy to perform video transformation processing on the initial business video to obtain a second candidate video from multiple candidate videos may include: performing video segmentation processing on the initial business video according to the M scene time periods to obtain M scene video segments; using a video splicing template to perform video segment splicing processing on the M scene video segments to obtain a second candidate video from the multiple candidate videos; wherein, when using the video splicing template to perform video segment splicing processing on the M scene video segments, there are at least two scene video segments that occupy different positions in the video splicing template.

[0178] It should be noted that the video splicing template may include a hierarchical simulcast template and a wonderful video segment display template. Among them, in the candidate video formed by splicing the video segments of each scene using the hierarchical simulcast template, the video segments of each scene are arranged and displayed in order from top to bottom or from left to right, and the video segments of each scene are hierarchically simulcasted. Hierarchical simulcast means that the videos of each scene are played in order according to the arrangement order. For example, after the first scene video segment is played, the second scene video segment is played, and after the second scene video segment is played, the third scene video segment is played, and so on. An exemplary hierarchical simulcast template can be seen in Figure 7d , 6 scene video segments are arranged vertically and displayed in the hierarchical simulcast template, and the first scene video segment is currently being played in the hierarchical simulcast template. The wonderful video segment display template includes a main playback area and multiple sub-playback areas. The area of ​​the main playback area is larger than that of the sub-playback area. Each scene video segment is displayed in the sub-playback area of ​​the wonderful video segment display template, and each scene video segment is played in sequence in the sub-playback area. The main playback area is used to play the video segment currently being played in the sub-playback area. An exemplary wonderful video segment display template can be found in Figure 7e , the six scene video segments are respectively displayed in the six sub-playback areas of the wonderful video segment display template, the first scene video segment 709 is played simultaneously in the sub-playback area and the main playback area, and the scene video segments in other sub-playback areas are not played.

[0179] (3) Video de-filling strategy.

[0180] From the foregoing, it can be seen that the video defilling processing strategy refers to a strategy for defilling the filling content in each image frame of the initial business video within the area indicated by the target filling area information, thereby removing the filling content in each image frame of the initial business video.

[0181] In one embodiment, the transformation reference information may include target filling area information of the initial business video. In this case, the process of using a video defilling strategy to perform video transformation processing on the initial business video to obtain a fourth candidate video from multiple candidate videos may include: defilling the filling content in each image frame of the initial business video within the area indicated by the target filling area information; performing image frame splicing processing on each image frame after the defilling processing to obtain a fourth candidate video from multiple candidate videos; wherein the image frame splicing processing refers to a splicing method of combining each image frame after the defilling processing to form a new video. An exemplary video defilling processing process can be found in Figure 7f After de-filling the filling content in the area 710 indicated by the target filling area information in the initial business video 701, the de-filled initial business video 711 is obtained. It should be noted that the fourth candidate video obtained by performing video transformation processing on the initial business video using the video de-filling strategy can also continue to be transformed using the video filling strategy, the video splicing strategy, etc.

[0182] (4) Video subject object tracking strategy.

[0183] As can be seen from the foregoing, the video subject object tracking strategy refers to a strategy for retaining the image content in the area indicated by the subject position area information in each image frame of the initial business video, and removing the image content in each image frame of the initial business video that is outside the area indicated by the subject position area information. Using the video subject object tracking strategy to perform video transformation processing on the initial business video may include: determining target cropping area information for each image frame of the initial business video based on the subject position area information of the subject object in each image frame of the initial business video; Figure 7g As shown, the area indicated by the subject position area information in the image frame is area 711, and the area indicated by the determined target cropping area information is area 712; then, the image frames of the initial business video can be cropped according to the target cropping area information of each image frame of the initial business video to obtain cropped image frames; and the cropped image frames can be spliced ​​to obtain the processed initial business video (i.e., the candidate video).

[0184] (5) Video cropping strategy.

[0185] As can be seen from the foregoing, the video cropping strategy refers to a strategy that retains the image content in each image frame of the initial business video that is located in the area indicated by the preset cropping area information, and removes the image content in each image frame of the initial business video that is located outside the area indicated by the preset cropping area information. Using the video cropping strategy to perform video transformation processing on the initial business video may include: cropping each image frame in the initial business video according to the preset cropping area information of each image frame in the initial business video to obtain each cropped image frame; and performing image frame splicing processing on each cropped image frame to obtain a processed initial business video (i.e., a candidate video). Figure 7h As shown, the content in the image frame located within the area 713 indicated by the preset cropping area information is retained, and the content in the image frame located outside the area 713 indicated by the preset cropping area information is cropped.

[0186] In one embodiment, when the initial business video is a multi-scene video, the initial business video may include a first type of scene video segment and a second type of scene video segment, the first type of scene video segment includes a subject object, and the second type of scene video segment does not include a subject object; the transformation reference information may include the first scene reference information and the second scene reference information. In this case, a video subject object tracking strategy may be used to perform a first transformation process on each first type of scene video segment according to the first scene reference information corresponding to each first type of scene video segment, so as to obtain each processed first type of scene video segment; and a video cropping strategy or a video filling strategy may be used to perform a second transformation process on each second type of scene video segment according to the second scene reference information corresponding to each second type of scene video segment, so as to obtain each processed second type of scene video segment; then, the processed first type of scene video segment and the processed second type of scene video segment may be subjected to a time sequence splicing process to obtain a fifth candidate video among multiple candidate videos; the time sequence splicing process refers to combining the processed first type of scene video segment and the processed second type of scene video segment to generate a new video segment splicing method.

[0187] Among them, the first scene reference information may include subject position area information of the main object in each image frame of the first type of scene video segment; for any first type of scene video segment contained in the initial business video, a video subject object tracking strategy is adopted to perform a first transformation processing on each first type of scene video segment according to the first scene reference information corresponding to each first type of scene video segment, and the process of obtaining each processed first type of scene video segment may include: determining target cropping area information of each image frame of the first type of scene video segment according to the subject position area information of the main object in each image frame of the first type of scene video segment; cropping each image frame of the first type of scene video segment according to the target cropping area information of each image frame of the first type of scene video segment to obtain each cropped image frame; performing image frame splicing processing on each cropped image frame to obtain a processed first type of scene video segment.

[0188] The second reference information may include the preset cropping area information of the second image frame in the second type of scene video segment or the image frame in the second type of scene video segment; the process of performing a second transformation process on each second type of scene video segment according to the second scene reference information corresponding to each second type of scene video segment using a video cropping strategy or a video filling strategy to obtain each processed second type of scene video segment may include: for any second type of scene video segment, based on the video filling strategy, the second image frame in the second type of scene video segment (for example, the last image frame of the second type of scene video segment) may be used to fill the second type of scene video segment to obtain the processed second type of scene video segment. The filling processing method here may be the Gaussian blur processing method using the third video filling template mentioned above. Alternatively, the video cropping strategy may be used to crop each image frame in the second type of scene video segment according to the preset cropping area information of each image frame in the second type of scene video segment to obtain the processed second type of scene video segment.

[0189] S607: Determine a target business video from multiple candidate videos.

[0190] The execution process of step S607 in the embodiment of the present application is similar to Figure 2 The execution process of step S204 in the illustrated embodiment is the same. For the specific execution process, please refer to Figure 2 The description of step S204 in the illustrated embodiment will not be repeated here.

[0191] Combined with the contents of the above steps S601-S607, the following Figure 8 The specific application scenarios of the embodiments of the present application are introduced as follows: Figure 8As shown, the initial business video may be an advertisement video 801 to be delivered to a video processing application. When the advertisement video 801 needs to be delivered, the advertisement video 801 may be uploaded to a video processing interface 80 of the video processing application. The video processing interface 80 is an interface for performing video processing in the video processing application. The video processing application runs in a computer device. The video processing interface 80 includes video adjustment options 802 (e.g. Figure 8 When the video adjustment option 802 is selected, a plurality of candidate videos for video transformation processing of the initial business video can be displayed in the video processing interface. The user can select one or more target candidate videos from the video processing interface 80 as target business videos, i.e., processed advertising videos for delivery, such as Figure 8 As shown, the second and fourth candidate videos from the left are selected as target business videos and processed as advertising videos for delivery.

[0192] In an embodiment of the present application, by adopting multiple video transformation processing strategies to perform video transformation processing on the initial business video, various types of video transformation processing can be performed on the initial business video, and the multiple candidate videos obtained by the video transformation processing can present multiple video transformation effects, thereby improving the video transformation effect.

[0193] The method of the embodiment of the present application is described in detail above. In order to facilitate better implementation of the above scheme of the embodiment of the present application, the device of the embodiment of the present application is provided below accordingly.

[0194] See also Fig. 9 , Fig. 9 It is a structural diagram of a video processing device provided in an embodiment of the present application. The video processing device can be set in a computer device. The video processing device can be, for example, a computer program (including program code) running in the computer device. The computer device can include a terminal or a server.

[0195] In one embodiment, the video processing device 90 can be used to perform Figure 2 , Figure 4 or Figure 6 The corresponding steps in the video processing method shown in the figure, the video processing device may include:

[0196] An acquisition unit 901 is used to acquire an initial service video to be processed;

[0197] The processing unit 902 is used to perform video recognition processing on the initial service video to obtain transformation reference information of the initial service video;

[0198] The processing unit 902 is further configured to generate a plurality of candidate videos according to the transformation reference information and the initial service video, wherein the plurality of candidate videos are obtained by performing video transformation processing on the initial service video using the transformation reference information;

[0199] The processing unit 902 is further configured to determine a target service video from a plurality of candidate videos.

[0200] In one embodiment, the transformation reference information includes classification label information of the initial business video; the processing unit 902 is used to perform video recognition processing on the initial business video, and when the transformation reference information of the initial business video is obtained, it is specifically used to perform the following steps:

[0201] Obtaining audio and video data and text data of the initial business video;

[0202] Acquire audio and video features of audio and video data and text features of text data;

[0203] Perform feature fusion processing on the audio and video features and the text features to obtain fusion features;

[0204] The initial business video is classified and identified based on the fusion features to obtain classification label information of the initial business video.

[0205] In one embodiment, the transformation reference information includes M scene time periods corresponding to the initial business video, where M is an integer greater than 1; the processing unit 902 is used to perform video recognition processing on the initial business video, and when the transformation reference information of the initial business video is obtained, it is specifically used to perform the following steps:

[0206] Determine a plurality of target image frames from an initial business video;

[0207] Performing scene switching detection on each target image frame, and determining a scene switching image frame that meets the scene switching condition from each target image frame;

[0208] According to the time points corresponding to the image frames of each scene switching, the time range corresponding to the initial business video is divided into M scene time periods;

[0209] Among them, the M scene time periods correspond to the M scene video segments in the initial business video, one scene time period corresponds to one scene video segment, there are N scene video segments among the M scene video segments belonging to different scenes, and N is an integer greater than 1 and less than or equal to M.

[0210] In one embodiment, the transformation reference information includes target filling area information of the initial business video; the processing unit 902 is used to perform video recognition processing on the initial business video, and when the transformation reference information of the initial business video is obtained, it is specifically used to perform the following steps:

[0211] Extracting a plurality of reference image frames from an initial business video;

[0212] Performing image filling detection on each reference image frame to obtain reference filling area information of each reference image frame;

[0213] If the areas indicated by the reference filling area information of each reference image frame match, the target filling area information is determined according to the reference filling area information of each reference image frame.

[0214] In one embodiment, the transformation reference information includes scene reference information of M scene video segments of the initial business video, where M is an integer greater than 1; the processing unit 902 is configured to perform video recognition processing on the initial business video, and when the transformation reference information of the initial business video is obtained, it is specifically configured to perform the following steps:

[0215] Obtain M scene video segments from the initial business video;

[0216] Performing subject object detection on the first image frame of each scene video segment in the M scene video segments;

[0217] Determining scene reference information of each scene video segment according to a main object detection result of a first image frame of each scene video segment;

[0218] Wherein, for a target scene video segment among the M scene video segments, if a subject object detection result of a first image frame of the target scene video segment indicates that the first image frame of the target scene video segment contains a subject object, then the scene reference information of the target scene video segment includes: subject position area information of the subject object in each image frame of the target scene video segment, where the subject position area information is obtained after tracking the subject object in the target scene video segment;

[0219] If the main object detection result of the first image frame of the target scene video segment indicates that the first image frame of the target scene video segment does not contain the main object, the scene reference information of the target scene video segment includes: the second image frame of the target scene video segment or preset cropping area information of each image frame in the target scene video segment.

[0220] In one embodiment, the transformation reference information includes classification label information of the initial business video; the processing unit 902 is used to generate multiple candidate videos according to the transformation reference information and the initial business video, and is specifically used to perform the following steps:

[0221] Obtain a video filling template that matches the classification label information of the initial business video;

[0222] The initial business video is filled with a video filling template to obtain a first candidate video among multiple candidate videos.

[0223] In one embodiment, the transformation reference information includes M scene time periods corresponding to the initial business video, where M is an integer greater than 1; the processing unit 902 is configured to generate multiple candidate videos according to the transformation reference information and the initial business video, and is specifically configured to perform the following steps:

[0224] Perform video segmentation processing on the initial business video according to M scene time periods to obtain M scene video segments;

[0225] Using a video splicing template to perform video segment splicing processing on the M scene video segments to obtain a second candidate video from multiple candidate videos;

[0226] When the video splicing template is used to perform video segment splicing processing on the M scene video segments, there are at least two scene video segments whose positions in the video splicing template are different.

[0227] In one embodiment, the transformation reference information includes M scene time periods corresponding to the initial business video, where M is an integer greater than 1; the processing unit 902 is configured to generate multiple candidate videos according to the transformation reference information and the initial business video, and is specifically configured to perform the following steps:

[0228] Perform video segmentation processing on the initial business video according to M scene time periods to obtain M scene video segments;

[0229] Filling image frames are extracted from the M scene video segments, and filling processing is performed on the initial business video according to the extracted filling image frames and the video filling template to obtain a third candidate video from the multiple candidate videos.

[0230] In one embodiment, the transformation reference information includes target filling area information of the initial service video; the processing unit 902 is used to generate multiple candidate videos according to the transformation reference information and the initial service video, and is specifically used to perform the following steps:

[0231] Based on the target filling area information, defilling is performed on each image frame of the initial business video;

[0232] Performing image frame splicing processing on each image frame after the defilling processing, to obtain a fourth candidate video among the multiple candidate videos.

[0233] In one embodiment, the initial service video includes a first type of scene video segment and a second type of scene video segment, the first type of scene video segment includes a subject object, and the second type of scene video segment does not include a subject object; the transformation reference information includes first scene reference information and second scene reference information;

[0234] The processing unit 902 is used to generate multiple candidate videos according to the transformation reference information and the initial service video, and is specifically used to perform the following steps:

[0235] According to the first scene reference information corresponding to each first type of scene video segment, a first transformation process is performed on each first type of scene video segment to obtain each first type of scene video segment after processing; and according to the second scene reference information corresponding to each second type of scene video segment, a second transformation process is performed on each second type of scene video segment to obtain each second type of scene video segment after processing;

[0236] The processed scene video segments of the first type and the processed scene video segments of the second type are spliced ​​in time sequence to obtain a fifth candidate video from the multiple candidate videos.

[0237] In one embodiment, the first scene reference information includes subject position area information of the subject object in each image frame of the first type of scene video segment; the processing unit 902 is used to perform a first transformation process on each first type of scene video segment according to the first scene reference information corresponding to each first type of scene video segment, and when obtaining each processed first type of scene video segment, specifically to perform the following steps:

[0238] For any first-type scene video segment, determining target cropping area information of each image frame of the first-type scene video segment according to subject position area information of the subject object in each image frame of the first-type scene video segment;

[0239] Cropping each image frame of the first type of scene video segment according to target cropping region information of each image frame of the first type of scene video segment to obtain cropped image frames;

[0240] The cropped image frames are subjected to image frame splicing processing to obtain a processed first type of scene video segment.

[0241] In one embodiment, the second scene reference information includes a second image frame in a second type of scene video segment or preset cropping area information of an image frame in a second type of scene video segment; the processing unit 902 is configured to perform a second transformation process on each second type of scene video segment according to the second scene reference information corresponding to each second type of scene video segment, and when obtaining each second type of scene video segment after the process, specifically to perform the following steps:

[0242] For any second type of scene video segment, the second image frame in the second type of scene video segment is used to fill the second type of scene video segment to obtain a processed second type of scene video segment; or, according to preset cropping area information of each image frame in the second type of scene video segment, each image frame in the second type of scene video segment is cropped to obtain a processed second type of scene video segment.

[0243] In one embodiment, Fig. 9 The various units in the video processing device 90 shown can be individually or completely combined into one or several other units to form, or one (some) of the units can be further divided into multiple functionally smaller units to form, which can achieve the same operation without affecting the realization of the technical effects of the embodiments of the present application. The above-mentioned units are divided based on logical functions. In practical applications, the function of one unit can also be implemented by multiple units, or the functions of multiple units can be implemented by one unit. In other embodiments of the present application, the video processing device 90 may also include other units. In practical applications, these functions can also be implemented with the assistance of other units, and can be implemented by the collaboration of multiple units. According to another embodiment of the present application, it is possible to execute the following by running on a general-purpose computing device of a general-purpose computer including processing elements and storage elements such as a central processing unit (CPU), a random access storage medium (RAM), and a read-only storage medium (ROM). Figure 2 , Figure 4 or Figure 6 A computer program (including program code) for each step involved in the corresponding method shown in Fig. 9 The video processing device 90 shown in the figure can be used to implement the video processing method of the embodiment of the present application. The computer program can be recorded on a computer-readable storage medium, for example, and loaded into the above-mentioned computing device through the computer-readable storage medium and run therein.

[0244] In an embodiment of the present application, a video can be subjected to video recognition processing, and a video transformation processing can be dynamically performed on the video based on the result of the video recognition processing to obtain multiple candidate videos, so as to determine one or more transformed videos from the multiple candidate videos. The video transformation processing method is adapted to the video, and multiple videos can be flexibly transformed for selection based on the video recognition result of a video, thereby improving the diversity of video transformation.

[0245] See also Fig.10 , Fig.10 1 is a schematic diagram of a computer device provided in an embodiment of the present application, and the computer device may be a terminal or a server. The computer device at least includes a processor 1001 and a memory 1002. The processor 1001 and the memory 1002 may be connected via a bus or other means.

[0246] The processor 1001 may be a central processing unit (CPU). The processor 1001 may further include a hardware chip. The hardware chip may be an application-specific integrated circuit (ASIC), a programmable logic device (PLD), etc. The PLD may be a field-programmable gate array (FPGA), a generic array logic (GAL), etc.

[0247] The memory 1002 may include a volatile memory (Volatile Memory), such as a random-access memory (Random-Access Memory, RAM); the memory 1002 may also include a non-volatile memory (Non-Volatile Memory), such as a flash memory (Flash Memory), a solid-state drive (SSD), etc.; the memory 1002 may also include a combination of the above-mentioned types of memory.

[0248] The memory 1002 is used to store computer programs, which include computer instructions, and the processor 1001 is used to execute computer instructions. The processor 1001 (or CPU (Central Processing Unit)) is the computing core and control core of the computer device, which is suitable for implementing one or more computer instructions, and is specifically suitable for loading and executing one or more computer instructions to implement the corresponding method flow or corresponding function.

[0249] The memory 1002 stores a computer program, which is executed by the processor 1001 to implement Figure 2 , Figure 4 or Figure 6 Corresponding steps in the method embodiment shown; in a specific implementation, the processor 1001 runs the computer program to:

[0250] Obtain the initial business video to be processed;

[0251] Performing video recognition processing on the initial business video to obtain transformation reference information of the initial business video;

[0252] Generate multiple candidate videos according to the transformation reference information and the initial business video, where the multiple candidate videos are obtained by performing video transformation processing on the initial business video using the transformation reference information;

[0253] Determine a target business video from multiple candidate videos.

[0254] In one embodiment, the transformation reference information includes classification label information of the initial service video; the processor 1001 is used to:

[0255] Obtaining audio and video data and text data of the initial business video;

[0256] Acquire audio and video features of audio and video data and text features of text data;

[0257] Perform feature fusion processing on the audio and video features and the text features to obtain fusion features;

[0258] The initial business video is classified and identified based on the fusion features to obtain classification label information of the initial business video.

[0259] In one embodiment, the transformation reference information includes M scene time periods corresponding to the initial service video, where M is an integer greater than 1; the processor 1001 is configured to:

[0260] Determine a plurality of target image frames from an initial business video;

[0261] Performing scene switching detection on each target image frame, and determining a scene switching image frame that meets the scene switching condition from each target image frame;

[0262] According to the time points corresponding to the image frames of each scene switching, the time range corresponding to the initial business video is divided into M scene time periods;

[0263] Among them, the M scene time periods correspond to the M scene video segments in the initial business video, one scene time period corresponds to one scene video segment, there are N scene video segments among the M scene video segments belonging to different scenes, and N is an integer greater than 1 and less than or equal to M.

[0264] In one embodiment, the transformation reference information includes target filling area information of the initial service video; the processor 1001 is used to:

[0265] Extracting a plurality of reference image frames from an initial business video;

[0266] Performing image filling detection on each reference image frame to obtain reference filling area information of each reference image frame;

[0267] If the areas indicated by the reference filling area information of each reference image frame match, the target filling area information is determined according to the reference filling area information of each reference image frame.

[0268] In one embodiment, the transformation reference information includes scene reference information of M scene video segments of the initial service video, where M is an integer greater than 1; the processor 1001 is configured to:

[0269] Obtain M scene video segments from the initial business video;

[0270] Performing subject object detection on the first image frame of each scene video segment in the M scene video segments;

[0271] Determining scene reference information of each scene video segment according to a main object detection result of a first image frame of each scene video segment;

[0272] Wherein, for a target scene video segment among the M scene video segments, if a subject object detection result of a first image frame of the target scene video segment indicates that the first image frame of the target scene video segment contains a subject object, then the scene reference information of the target scene video segment includes: subject position area information of the subject object in each image frame of the target scene video segment, where the subject position area information is obtained after tracking the subject object in the target scene video segment;

[0273] If the main object detection result of the first image frame of the target scene video segment indicates that the first image frame of the target scene video segment does not contain the main object, the scene reference information of the target scene video segment includes: the second image frame of the target scene video segment or preset cropping area information of each image frame in the target scene video segment.

[0274] In one embodiment, the transformation reference information includes classification label information of the initial service video; the processor 1001 is used to:

[0275] Obtain a video filling template that matches the classification label information of the initial business video;

[0276] The initial business video is filled with a video filling template to obtain a first candidate video among multiple candidate videos.

[0277] In one embodiment, the transformation reference information includes M scene time periods corresponding to the initial service video, where M is an integer greater than 1; the processor 1001 is configured to:

[0278] Perform video segmentation processing on the initial business video according to M scene time periods to obtain M scene video segments;

[0279] Using a video splicing template to perform video segment splicing processing on the M scene video segments to obtain a second candidate video from multiple candidate videos;

[0280] When the video splicing template is used to perform video segment splicing processing on the M scene video segments, there are at least two scene video segments whose positions in the video splicing template are different.

[0281] In one embodiment, the transformation reference information includes M scene time periods corresponding to the initial service video, where M is an integer greater than 1; the processor 1001 is configured to:

[0282] Perform video segmentation processing on the initial business video according to M scene time periods to obtain M scene video segments;

[0283] Filling image frames are extracted from the M scene video segments, and filling processing is performed on the initial business video according to the extracted filling image frames and the video filling template to obtain a third candidate video from the multiple candidate videos.

[0284] In one embodiment, the transformation reference information includes target filling area information of the initial service video; the processor 1001 is used to:

[0285] Based on the target filling area information, defilling is performed on each image frame of the initial business video;

[0286] Performing image frame splicing processing on each image frame after the defilling processing, to obtain a fourth candidate video among the multiple candidate videos.

[0287] In one embodiment, the initial service video includes a first type of scene video segment and a second type of scene video segment, the first type of scene video segment includes a subject object, and the second type of scene video segment does not include a subject object; the transformation reference information includes first scene reference information and second scene reference information; the processor 1001 is used to:

[0288] According to the first scene reference information corresponding to each first type of scene video segment, a first transformation process is performed on each first type of scene video segment to obtain each first type of scene video segment after processing; and according to the second scene reference information corresponding to each second type of scene video segment, a second transformation process is performed on each second type of scene video segment to obtain each second type of scene video segment after processing;

[0289] The processed scene video segments of the first type and the processed scene video segments of the second type are spliced ​​in time sequence to obtain a fifth candidate video from the multiple candidate videos.

[0290] In one embodiment, the first scene reference information includes subject position area information of the subject object in each image frame of the first type of scene video segment; the processor 1001 is used to:

[0291] For any first-type scene video segment, determining target cropping area information of each image frame of the first-type scene video segment according to subject position area information of the subject object in each image frame of the first-type scene video segment;

[0292] Cropping each image frame of the first type of scene video segment according to target cropping region information of each image frame of the first type of scene video segment to obtain cropped image frames;

[0293] The cropped image frames are subjected to image frame splicing processing to obtain a processed first type of scene video segment.

[0294] In one embodiment, the second scene reference information includes a second image frame in a second type of scene video segment or preset cropping area information of an image frame in a second type of scene video segment; the processor 1001 is configured to:

[0295] For any second type of scene video segment, the second image frame in the second type of scene video segment is used to fill the second type of scene video segment to obtain a processed second type of scene video segment; or, according to preset cropping area information of each image frame in the second type of scene video segment, each image frame in the second type of scene video segment is cropped to obtain a processed second type of scene video segment.

[0296] In an embodiment of the present application, a video can be subjected to video recognition processing, and a video transformation processing can be dynamically performed on the video based on the result of the video recognition processing to obtain multiple candidate videos, so as to determine one or more transformed videos from the multiple candidate videos. The video transformation processing method is adapted to the video, and multiple videos can be flexibly transformed for selection based on the video recognition result of a video, thereby improving the diversity of video transformation.

[0297] In one embodiment, a computer program product or a computer program is provided, the computer program product or the computer program includes computer instructions, the computer instructions are stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the video processing method provided in the above-mentioned various optional manners.

[0298] A person of ordinary skill in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing related hardware through a program, and the program can be stored in a computer-readable storage medium. The computer-readable storage medium may include: a flash drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0299] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A video processing method, characterized in that: The method comprises: Obtain the initial business video to be processed; Performing video recognition processing on the initial business video to obtain transformation reference information of the initial business video; the transformation reference information includes: classification label information of the initial business video, M scene time periods corresponding to the initial business video, target filling area information of the initial business video, and scene reference information of the M scene video segments of the initial business video, where M is an integer greater than 1; different transformation reference information respectively correspond to respective video transformation processing strategies; Generate multiple candidate videos according to the transformation reference information and the initial business video; wherein, generating multiple candidate videos according to the transformation reference information and the initial business video includes: determining multiple video transformation processing strategies according to the transformation reference information, and using each of the video transformation processing strategies to perform video transformation processing on the initial business video to obtain one candidate video among the multiple candidate videos; A target business video is determined from the multiple candidate videos.

2. The method according to claim 1, characterized in that The transformation reference information includes classification label information of the initial business video; the performing video recognition processing on the initial business video to obtain the transformation reference information of the initial business video includes: Acquire audio and video data and text data of the initial business video; Acquire audio-visual features of the audio-visual data and text features of the text data; Performing feature fusion processing on the audio and video features and the text features to obtain fusion features; The initial business video is classified and identified based on the fusion feature to obtain classification label information of the initial business video.

3. The method according to claim 1, characterized in that The transformation reference information includes M scene time periods corresponding to the initial business video; the performing video recognition processing on the initial business video to obtain the transformation reference information of the initial business video includes: Determining a plurality of target image frames from the initial business video; Performing scene switching detection on each target image frame, and determining a scene switching image frame that meets a scene switching condition from the target image frames; According to the time points corresponding to the scene switching image frames, the time range corresponding to the initial business video is divided into the M scene time periods; Among them, the M scene time periods correspond to M scene video segments in the initial business video, one scene time period corresponds to one scene video segment, and there are N scene video segments among the M scene video segments belonging to different scenes, and N is an integer greater than 1 and less than or equal to M.

4. The method according to claim 1, characterized in that The transformation reference information includes target filling area information of the initial business video; the performing video recognition processing on the initial business video to obtain the transformation reference information of the initial business video includes: Extracting a plurality of reference image frames from the initial business video; Performing image filling detection on each reference image frame to obtain reference filling area information of each reference image frame; If the areas indicated by the reference filling area information of the reference image frames match, the target filling area information is determined according to the reference filling area information of the reference image frames.

5. The method according to claim 1 or 3, characterized in that: The transformation reference information includes scene reference information of M scene video segments of the initial business video; The performing video recognition processing on the initial business video to obtain transformation reference information of the initial business video includes: Acquire the M scene video segments from the initial business video; Performing subject object detection on the first image frame of each scene video segment in the M scene video segments; Determining scene reference information of each scene video segment according to a main object detection result of a first image frame of each scene video segment; Wherein, for a target scene video segment in the M scene video segments, if a subject object detection result of a first image frame of the target scene video segment indicates that a subject object is included in the first image frame of the target scene video segment, the scene reference information of the target scene video segment includes: subject position area information of the subject object in each image frame of the target scene video segment, wherein the subject position area information is obtained after tracking the subject object in the target scene video segment; If the main object detection result of the first image frame of the target scene video segment indicates that the first image frame of the target scene video segment does not contain the main object, the scene reference information of the target scene video segment includes: the second image frame of the target scene video segment or preset cropping area information of each image frame in the target scene video segment.

6. The method according to claim 1 or 2, characterized in that: The transformation reference information includes classification label information of the initial business video; the multiple video transformation processing strategies include a video filling strategy; and the process of performing video transformation processing on the initial business video using the video filling strategy includes: Acquire a video filling template that matches the classification label information of the initial business video; The initial business video is filled using the video filling template to obtain a first candidate video among the multiple candidate videos.

7. The method according to claim 1 or 3, characterized in that: The transformation reference information includes M scene time periods corresponding to the initial business video, where M is an integer greater than 1; the multiple video transformation processing strategies include a video splicing strategy; The process of performing video transformation processing on the initial service video by adopting the video splicing strategy includes: Performing video segmentation processing on the initial business video according to the M scene time periods to obtain M scene video segments; Using a video splicing template to perform video segment splicing processing on the M scene video segments to obtain a second candidate video from the multiple candidate videos; When the video splicing template is used to perform video segment splicing processing on the M scene video segments, there are at least two scene video segments that occupy different positions in the video splicing template.

8. The method according to claim 1 or 3, characterized in that: The transformation reference information includes M scene time periods corresponding to the initial business video, where M is an integer greater than 1; the multiple video transformation processing strategies include a video filling strategy; and the process of performing video transformation processing on the initial business video using the video filling strategy includes: Performing video segmentation processing on the initial business video according to the M scene time periods to obtain M scene video segments; Filling image frames are extracted from the M scene video segments, and filling processing is performed on the initial business video according to the extracted filling image frames and the video filling template to obtain a third candidate video among the multiple candidate videos.

9. The method according to claim 1 or 4, characterized in that: The transformation reference information includes target filling area information of the initial service video; the multiple video transformation processing strategies include video defilling strategies; The process of performing video transformation processing on the initial service video by adopting the video defilling strategy includes: Based on the target filling area information, defilling each image frame of the initial business video; Performing image frame splicing processing on each image frame after the defilling processing to obtain a fourth candidate video among the multiple candidate videos.

10. The method according to claim 1, characterized in that The initial service video includes a first type of scene video segment and a second type of scene video segment, the first type of scene video segment includes a subject object, and the second type of scene video segment does not include a subject object; the transformation reference information includes first scene reference information and second scene reference information; The plurality of video transformation processing strategies include a video subject object tracking strategy; The process of performing video transformation processing on the initial business video by adopting the video subject object tracking strategy includes: performing a first transformation process on each first-type scene video segment according to first scene reference information corresponding to each first-type scene video segment to obtain processed first-type scene video segments; as well as, performing a second transformation process on each of the second-type scene video segments according to the second scene reference information corresponding to each of the second-type scene video segments to obtain processed each of the second-type scene video segments; The processed scene video segments of the first type and the processed scene video segments of the second type are subjected to time-sequential splicing processing to obtain a fifth candidate video among the multiple candidate videos.

11. The method according to claim 10, characterized in that The first scene reference information includes subject position area information of the subject object in each image frame of the first type of scene video segment; performing a first transformation process on each first type of scene video segment according to the first scene reference information corresponding to each first type of scene video segment to obtain each processed first type of scene video segment includes: For any first-type scene video segment, determining target cropping area information of each image frame of the first-type scene video segment according to subject position area information of a subject object in each image frame of the first-type scene video segment; Cropping each image frame of the first type of scene video segment according to target cropping area information of each image frame of the first type of scene video segment to obtain cropped image frames; The cropped image frames are subjected to image frame splicing processing to obtain a processed first type of scene video segment.

12. The method according to claim 10, characterized in that The second scene reference information includes a second image frame in the second type of scene video segment or preset cropping area information of each image frame in the second type of scene video segment; the second transformation processing is performed on each second type of scene video segment according to the second scene reference information corresponding to each second type of scene video segment to obtain each second type of scene video segment after processing, including: For any second type of scene video segment, using the second image frame in the second type of scene video segment, performing filling processing on the second type of scene video segment to obtain a processed second type of scene video segment; or, According to the preset cropping area information of each image frame in the second type of scene video segment, cropping processing is performed on each image frame in the second type of scene video segment to obtain a processed second type of scene video segment.

13. A video processing device, characterized in that: The video processing device comprises: An acquisition unit, used for acquiring an initial business video to be processed; A processing unit, configured to perform video recognition processing on the initial business video to obtain transformation reference information of the initial business video; the transformation reference information includes: classification label information of the initial business video, M scene time periods corresponding to the initial business video, target filling area information of the initial business video, and scene reference information of the M scene video segments of the initial business video, where M is an integer greater than 1; different transformation reference information respectively correspond to respective video transformation processing strategies; The processing unit is further configured to generate a plurality of candidate videos according to the transformation reference information and the initial business video; wherein the generating a plurality of candidate videos according to the transformation reference information and the initial business video comprises: determining a plurality of video transformation processing strategies according to the transformation reference information, and performing video transformation processing on the initial business video using each of the video transformation processing strategies respectively, to obtain one of the plurality of candidate videos; The processing unit is further configured to determine a target business video from the multiple candidate videos.

14. A computer device, characterized in that: The computer device comprises: A memory and a processor; wherein the memory is used to store a computer program; and the processor is used to execute the computer program to implement the video processing method as described in any one of claims 1 to 12.

15. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is read and executed by a processor, the video processing method according to any one of claims 1 to 12 is implemented.

16. A computer program product, characterized in that The computer program product includes computer instructions, which are stored in a computer-readable storage medium; when the computer instructions are read and executed by a processor, the video processing method according to any one of claims 1 to 12 is implemented.

Images