Video processing method, device, computer equipment and storage medium

By performing scene cutting and target object cutting on the video frame sequence, and generating clip videos, the time-consuming and labor-consuming video tracking and editing in the prior art is solved, and efficient video processing and quality improvement is achieved.

CN113689440BActive Publication Date: 2024-11-26TENCENT MUSIC ENTERTAINMENT TECH (SHENZHEN) CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202110971660.4
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-08-23
Publication Date
2024-11-26
Estimated Expiration
2041-08-23

AI Technical Summary

Technical Problem

In the prior art, video tracking and editing is time-consuming and labor-intensive, and is inefficient, and cannot effectively improve the processing efficiency of objects of attention in video.

Method used

By obtaining the video frame sequence, performing scene cutting processing, determining the reference object of the target video frame set, and cropping its image area to generate clip videos to realize automated video processing.

Benefits of technology

It improves the efficiency of video processing, reduces manual participation, improves the quality of video conversion and user experience, and saves manual editing operation costs.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN113689440B_ABST
    Figure CN113689440B_ABST
Patent Text Reader

Abstract

The present application discloses a video processing method, device, computer equipment and storage medium, wherein the method comprises: obtaining a video frame sequence of a processed video, then performing scene cutting processing on the video frame sequence, removing scene switching video frames, obtaining N video frame sets, each video frame set corresponding to a scene, then determining a reference object of a target video frame set in the N video frame sets, and obtaining a cropped image set according to an image region where the reference object is located in the target video frame included in the target video frame set, and finally generating a video clip of the video to be processed according to the cropped image set. Through this method, the processing efficiency of the video is improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a video processing method, apparatus, computer equipment and storage medium. Background Art

[0002] With the rapid development of information technology, various types of videos have emerged. Compared with non-mobile terminals, mobile terminals are more popular. People like to use mobile terminals to listen to music and watch videos. When watching videos, people often focus on a special object. Videos with a focus are more popular. Therefore, tracking objects in videos to obtain videos of the focused object is one of the current research hotspots.

[0003] At present, most of the solutions for video tracking and editing of the objects of interest in the video are obtained through manual editing. This method is time-consuming, labor-intensive and inefficient. Summary of the invention

[0004] The embodiments of the present invention provide a video processing method, apparatus, computer equipment and storage medium, which can improve the efficiency of video processing.

[0005] On the one hand, an embodiment of the present application discloses a video processing method, the method comprising:

[0006] Get the video frame sequence of the video to be processed;

[0007] Performing scene segmentation processing on the video frame sequence to obtain N video frame sets, each video frame set corresponds to a scene, and N is an integer greater than 1;

[0008] Determine a reference object of a target video frame set among the N video frame sets, where the target video frame set is any one of the N video frame sets;

[0009] For the target video frame set, cropping an image region where a reference object is located in the target video frame included in the target video frame set to obtain a cropped image set corresponding to the target video frame set, wherein the target video frame is a video frame in the target video frame set that includes the reference object;

[0010] A video segment of the video to be processed is generated according to the cropped image set.

[0011] The present application discloses a video processing device, which includes:

[0012] An acquisition unit, used for acquiring a video frame sequence of a video to be processed;

[0013] A processing unit, configured to perform scene segmentation processing on the video frame sequence to obtain N video frame sets, each video frame set corresponding to a scene, where N is an integer greater than 1;

[0014] a determining unit, configured to determine a reference object of a target video frame set among the N video frame sets, wherein the target video frame set is any one of the N video frame sets;

[0015] The processing unit is further used to crop the image area where the reference object is located in the target video frame set, to obtain a cropped image set corresponding to the target video frame set, wherein the target video frame is a video frame in the target video frame set that includes the reference object, and to generate a video clip of the video to be processed based on the cropped image set.

[0016] On the one hand, an embodiment of the present application discloses a computer device, which includes: a processor, suitable for implementing one or more computer programs; and a computer storage medium, wherein the computer storage medium stores one or more computer programs, and the one or more computer programs are suitable for being loaded by the processor and executing the above-mentioned video processing method.

[0017] In one aspect, the present application discloses a computer-readable storage medium, wherein the computer storage medium stores one or more computer programs, and the one or more computer programs are suitable for being loaded by a processor and executing the above-mentioned video processing method.

[0018] The present application discloses a computer program product, which includes a computer program stored in a computer-readable storage medium. A processor of a computer device reads the computer program from the computer-readable storage medium, and the processor executes the computer program, so that the computer device executes the above-mentioned video processing method.

[0019] In the embodiment of the present application, the computer device processes the video to be processed to obtain a sequence of video frames to be processed, and then performs scene cutting processing on the video frame sequence, removes the scene switching video frames, and obtains N video frame sets, each of which corresponds to a scene, and then determines the reference object of the target video frame set in the N video frame sets, and crops the image area where the reference object is located in the target video frame included in the target video frame set to obtain a cropped image set, and finally generates a video clip of the video to be processed based on the cropped image set. This method can automatically crop the video frames to obtain video clips by cutting the video scenes, and then according to the reference objects in each scene and the image area including the reference objects, thereby improving the processing efficiency of the video. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0021] Figure 1 is a schematic diagram of the architecture of a video processing system disclosed in an embodiment of the present application;

[0022] Figure 2 It is a flowchart of a video processing method disclosed in an embodiment of the present application;

[0023] Figure 3a is a schematic diagram of a priority order disclosed in an embodiment of the present application;

[0024] Figure 3b is a schematic diagram of a reference object confirmation process disclosed in an embodiment of the present application;

[0025] Figure 3c is a schematic diagram of a cutting method disclosed in an embodiment of the present application;

[0026] Figure 4 It is a flowchart of another video processing method disclosed in an embodiment of the present application;

[0027] Figure 5 is a schematic diagram of a target motion trajectory disclosed in an embodiment of the present application;

[0028] Figure 6 is a flowchart of another video processing method disclosed in an embodiment of the present application;

[0029] Figure 7 is a structural schematic diagram of a video processing device disclosed in an embodiment of the present application;

[0030] Figure 8 It is a structural schematic diagram of a computer device disclosed in an embodiment of the present application. DETAILED DESCRIPTION

[0031] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0032] In order to effectively improve the quality and efficiency of vertical video conversion, the embodiment of the present application proposes a video processing solution and a corresponding video processing system; Figure 1 This is a schematic diagram of the architecture of a video processing system disclosed in an embodiment of the present application. The video processing system 100 may include at least: a terminal device 101 and a computer device 102, wherein the terminal device 101 may be mainly used to send a video to be processed to the computer device 102, or in some feasible embodiments, the video to be processed may also be obtained from a database by the computer device 102 when receiving a video clip search; the computer device 102 may be mainly used to process the video to be processed to obtain the corresponding video clip. Furthermore, the terminal device 101 and the computer device 102 may realize a communication connection, and the connection mode includes a wired connection and a wireless connection, which is not limited.

[0033] It should be noted that the terminal device 101 mentioned above can be a smart phone, tablet computer, laptop computer, desktop computer, smart speaker, smart watch, smart car, etc., but is not limited to this. The computer device 102 mentioned above can be a server, which can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms. Figure 1 The system architecture of the video processing system is merely exemplified and is not intended to be limiting. Figure 1 The computer device 102 can be deployed as a node in the blockchain network, or the computer device 102 can be connected to the blockchain network, so that the computer device 102 can upload internal data to the blockchain network for storage to prevent the internal data from being tampered with, thereby ensuring data security.

[0034] In a specific implementation, the video processing solution proposed in the embodiment of the present application is implemented by the above-mentioned video processing system, and the general process is as follows: the computer device 102 processes the video to be processed to obtain a video frame sequence to be processed, and then performs scene segmentation processing on the video frame sequence, removes scene switching video frames, and obtains N video frame sets, each video frame set corresponds to a scene, and then determines the reference object of the target video frame set in the N video frame sets, and obtains a cropped image set based on the image area where the reference object is located in the target video frame included in the target video frame set, and finally generates a video clip of the video to be processed based on the cropped image set.

[0035] Based on the above description, the video processing method provided in the embodiment of the present application crops the scene transition frames in the video frame sequence of the video to be processed, making the video connection more natural, and then determines the final video segment through the reference object, so that the converted video can be concentrated on the reference object, thereby improving the quality of the video. At the same time, since no human participation is required in the video processing process, the labor is reduced and the video processing efficiency can also be improved.

[0036] In a specific implementation scenario, for some music playing software, each piece of music has a corresponding music video (Music Video, MV). Therefore, when users use the mobile terminal to play music, they will also watch the MV synchronously. Most MVs require users to click on the screen and flip the mobile terminal to watch them more conveniently. Therefore, in order to facilitate users, the MV needs to be properly processed, such as directly converting it into a vertical video, so that users can watch the MV without flipping the mobile terminal, which can give users a better experience. This solution is to convert the MV to a vertical version, which can improve the quality of the converted video and the efficiency of the conversion process, thereby improving the user's viewing experience. For developers, it saves manual editing operation costs and has a higher video conversion efficiency.

[0037] Based on the above video processing system, the present application embodiment discloses a video processing method, see Figure 2 , Figure 2 : is a flow chart of a video processing method disclosed in an embodiment of the present application. The method can be executed by a computer device, and the computer device can be an independent server. Specifically, the method flow can include steps S201-S205:

[0038] S201: Obtain a video frame sequence of a video to be processed.

[0039] In the specific implementation process, the computer device first obtains the video to be processed. The video to be processed can be a video sent to the computer device when a user on the client needs a video clip. The user here generally refers to a person who manages the video; or, the video to be processed can be automatically obtained from a database by the computer device when the computer device receives a user search for a video clip. The user here refers to a user who uses video software to play the video.

[0040] In a possible implementation, after the computer device obtains the video to be processed, the video to be processed is preprocessed. The preprocessing can be specifically performed by frame processing on the video to be processed to obtain each video frame corresponding to the video to be processed, thereby obtaining a video frame sequence, and the video frame sequence is sorted according to the playback order of the video to be processed. Among them, the frame processing of the video to be processed can adopt opencv, which is a classic special library in computer vision, which supports multiple languages, cross-platform and powerful functions; other software can also be used to process the video to be processed to obtain a video frame sequence, which will not be listed here one by one. Before executing step S202, the computer device can also perform edge elimination based on pixel value calculation on the video frame sequence to eliminate useless elements of the video frame, such as solid color borders, to ensure better subsequent processing of the video frame.

[0041] The above-mentioned video to be processed can be a horizontal video. Horizontal video and vertical video are two different video playback modes. The most intuitive form is that the sizes of the two on the device screen are different. The ratio of horizontal video is generally 16:9, and the ratio of vertical video is generally 9:16. The purpose of this application is to convert horizontal video into vertical video, so that users can have a better video viewing experience without flipping the device.

[0042] S202: Perform scene segmentation processing on the video frame sequence to obtain N video frame sets.

[0043] In a possible implementation, after the computer device processes the video to be processed to obtain a video frame sequence, the method implements scene cutting by predicting the scene switching frame. Since the scene cutting task only needs to focus on the global features of the image, the specific steps for implementing the cutting include: greatly reducing each video frame image in the video frame sequence so that the pixel value of each video frame image is small, which is conducive to improving the processing speed; then inputting the reduced video frame sequence into a lightweight CNN neural network, connecting 2D convolution and 1D convolution in series to force the network to learn the image space and time features respectively; inputting the features into the Sigmoid function to obtain a score value of whether each video frame image belongs to a scene switching frame. Specifically, if a reference video frame with a score value greater than a specified threshold is included in the video frame sequence, wherein the reference video frame is any video frame in the video frame sequence, then the reference video frame is determined as a scene switching frame, and the scene switching frame is deleted from the video frame sequence, and the video frame with a score value less than or equal to the specified threshold is retained, and then the video frame sequence after deleting the reference video frame is sorted according to the position of each reference video frame in the video sequence to obtain a set of N video frames, wherein N is an integer greater than 1. It can be understood that a video frame set corresponds to a video clip, each video frame set corresponds to a video scene, and the video scenes can be the same or different. For example, after deleting the scene switching frames of the video frame sequence of the video to be processed, 5 video frame sets are obtained, which can be understood as 5 video clips, where each video clip corresponds to a video scene, and the 5 video clips can have the same video scene, such as the first video scene can be the same as or similar to the fourth video scene.

[0044] It should be noted that, in the process of arranging the video frame sequence after deleting the reference video frame (scene switching frame), in order to ensure the viewing quality of the final film, only the segments with a time length greater than 1 second may be retained for subsequent processing.

[0045] S203: Determine a reference object of a target video frame set among the N video frame sets.

[0046] In one possible implementation, after determining N video frame sets, the computer device determines a reference object of a target video frame set among the N video frame sets based on saliency detection and target detection methods, wherein the target video frame set is any video frame set among the N video frame sets, that is, it is necessary to determine a reference object in each video frame set.

[0047] The specific implementation process includes: in the embodiment of the present application, a single-order target detection network yolov3 (a target detection network) can be used to process the target video frame set frame by frame, detect and locate the objects appearing in the target video frame set, where the objects generally refer to people, and multiple objects can appear in one video frame at the same time. In this process, the video frames where no objects appear can also be directly discarded. If the object is a person, the video frames where no people appear are directly discarded. This operation is repeated for each video frame set in the N video frame sets.

[0048] It should be noted that not every video frame set can detect an object. If no person is detected in a video frame set, it is considered that there is no object in the video frame set. In some cases, the video frame set can be directly discarded, for example: 1. There is no person in the video frame set, such as a black screen; 2. The number of frames in the video frame set is too small (the video clip is 1 second long); 3. The text area in the frame is too large or too central, and it cannot be avoided. If the horizontal bottle is changed to the vertical bottle, it will affect the viewing experience if subtitles are edited in.

[0049] Among them, the Yolo series network is a classic model for target detection tasks, and the yolov3 model achieves a good balance between speed and prediction accuracy. Therefore, it is better to use yolov3 in this application. This application uses yolov3 for character detection, and specifically uses the target detection general data set COCO (COCO data set is a large, rich object detection, segmentation and subtitle data set, the image includes 91 types of targets, 328,000 images and 2,500,000 tags, which is the largest data set with semantic segmentation so far, and the categories provided are 80 categories, with more than 330,000 pictures, of which 200,000 are marked, and the number of individuals in the entire data set exceeds 1.5 million) The images of the characters contained in the pre-trained model of yolov3 are fine-tuned (finetune), so that the model can reach a higher level of accuracy in character detection. In addition to characters, the model is also suitable for the detection of almost any target of interest, such as pets, cars, etc., and if these elements are also added to the collection generated by editing in the future, it will inevitably make the film more ornamental.

[0050] If multiple objects are detected in the target video frame set, a reference object is determined from the multiple objects. This application jointly determines the reference object in the target video frame set based on person detection and saliency detection. The specific process includes: since there are multiple objects in a video frame, a tracking id is marked for each object. This application deals more with music clips, and multiple objects can be people. Therefore, face recognition technology is used to match multiple people with stars in the star library to determine the priority order. The priority order is based on the object type, such as singers, ordinary people, etc. The priority order from high to low can be singers, ordinary people, and people whose faces are not shown (in most cases, people with their backs to the camera). The person with the highest priority is selected from the tracking id as the reference person, which is the reference object in this application. Please refer to Figure 3a , which is a schematic diagram of a priority order disclosed in an embodiment of the present application. When there is only one object at the highest priority level, the object can be directly determined as the reference object. When there is more than one object at the highest priority level, the reference object is determined in combination with the heat map.

[0051] In some feasible embodiments, there may be multiple tracking IDs in a single level, that is, when the highest priority objects determined from multiple objects include at least two, such as singers, in this case, the tracking ID of the lens focus in the video frame can be determined in combination with saliency detection. Specifically, saliency detection is to generate a heat map for the target video frame, the target video frame is any video frame in the video frame sequence, the heat map represents the value of the object appearing in the video frame image, the value of the heat map is the score of each pixel of the picture by the saliency detection network, and the score range is 0 to 1, thereby obtaining a saliency heat map of a picture, and the higher the score area represents that the model predicts that the pixel is more likely to be the focus of human eye attention. The heat map is vertically summed to obtain the attention curve of the target video frame in the horizontal direction, the horizontal coordinate of the attention curve is the horizontal coordinate of the target video frame, and the vertical coordinate is the visual attention value. Calculate the average thermal value covered by each tracking id in the same level in the target video frame set, obtain the average thermal value of each tracking id, and select the tracking id with the highest average thermal value as the reference object of the target video frame set. The specific calculation method is: each tracking id corresponds to a 2D target detection result box, marked with (x1, y1, x2, y2), (x1, y1) is the coordinate of the upper left corner of the box, (x2, y2) is the coordinate of the lower right corner of the box, and since each pixel has a corresponding attention value, the attention average of the tracking id is the average of the attention values ​​corresponding to all pixels in the detection box.

[0052] See also Figure 3b , which is a schematic diagram of a reference object confirmation process disclosed in an embodiment of the present application, wherein 310 represents an attention curve diagram, 320 represents a video frame image, and each dotted box in 330 represents a target area, that is, multiple object areas, wherein, after the video frame image 320 is input into the neural network, a heat map is obtained, and the values ​​in the heat map are vertically summed to obtain the attention curve diagram 310, and then the average value of the heat value of each dotted box in the video frame image 330 is calculated through the attention curve diagram 310 to determine the final reference object. Among them, the horizontal axis represents the horizontal axis of the video frame image, and the vertical axis represents the visual attention value. The higher the vertical axis, the higher the visual attention of the corresponding video frame horizontal axis position. This operation is performed for each video frame set in the N video frame sets to obtain the reference object corresponding to each video frame set.

[0053] S204 . For the target video frame set, crop the image region where the reference object is located in the target video frame to obtain a cropped image set corresponding to the target video frame set.

[0054] The target video frame is a video frame in the target video frame set that includes a reference object.

[0055] In a possible implementation, the computer device can crop the image area where the reference object is located in the target video frame set, and obtain a cropped image set corresponding to the target video frame set. In order to convert the horizontal video into the vertical video, the original video frame needs to be cropped. On the one hand, the area including the reference object is cropped, and on the other hand, the subtitle area in the video frame is cropped to avoid the subtitle area from blocking the reference object in the video frame image, so that the converted video presents a high-quality state. Therefore, in the specific implementation process, the computer device first determines the subtitle area in the target video frame included in the target video frame set, that is, performs text detection on each video frame, clusters the text frames of the previous and next video frames, and obtains the position where the subtitles appear in the target video frame included in the target video frame set, and then crops the area where the subtitles appear. For the subtitles that appear in the cropped area, the method can adaptively select the largest effective area of ​​the original cropped area to retain, that is, on the basis of retaining the reference object, crop the largest subtitle area, and then obtain a cropped image set. The cropped image set includes: a cropped image obtained by cropping the image area of ​​the reference object in the target video frame, and the target video frame is a video frame that includes the reference object in the target video frame set. One video frame set corresponds to one cropped image set, and N video frame sets have N cropped image sets. Correspondingly, the cropping method can be found in. Figure 3c, which is a schematic diagram of a cropping method disclosed in an embodiment of the present application. We can see that the dotted box 301 is a clipping box that retains the reference object, and 302 is the maximum effective area retained after cropping the subtitles.

[0056] S205: Generate a video clip of the video to be processed according to the cropped image set.

[0057] In a possible implementation process, before generating a video clip of the video to be processed according to the cropped image set, the computer device can also resize the cropped image set, that is, convert the original horizontal size into a vertical size. Specifically, the images in the cropped image set can be converted to vertical screens according to a preset size, and then the video clips of the video to be processed can be generated according to the images after the vertical screen conversion. The ffmpeg tool can be used to splice the single-frame images of the images in the cropped image set that have been processed by super-resolution into a 1080p high-definition vertical video. Among them, it should be noted that since the video clips are mainly played on mobile terminals, in order to adapt to the popular large-screen high-definition smart mobile terminal screen, the cropped single-frame images will be processed by super-resolution through the GAN network (Generative Adversarial Networks) to fill pixels and improve clarity to ensure the quality of the played video.

[0058] In the implementation of this application, it is mainly described that a computer device processes a video to be processed to obtain a sequence of video frames to be processed, and then performs scene cutting processing on the video frame sequence, removes scene switching video frames, and obtains N video frame sets, each video frame set corresponds to a scene, and then determines the reference object of the target video frame set in the N video frame sets, and obtains a cropped image set according to the image area where the reference object is located in the target video frame included in the target video frame set, and finally generates a video clip of the video to be processed according to the cropped image set. This method cuts the video scene, and then automatically crops the video frame to obtain a video clip according to the reference object in each scene and the image area including the reference object, thereby improving the processing efficiency of the video.

[0059] According to the above video processing system and video processing method, the present application embodiment discloses another video processing method, see Figure 4 , which is a flow chart of another video processing method disclosed in an embodiment of the present application, the corresponding execution subject may be a computer device, and the computer device may be an independently deployed server. Specifically, the video processing method may include steps S401-S408:

[0060] S401: Obtain a video frame sequence of a video to be processed.

[0061] S402: Perform scene segmentation processing on the video frame sequence to obtain N video frame sets.

[0062] S403: Determine a reference object of a target video frame set among the N video frame sets.

[0063] Among them, steps S401-S403 are Figure 2 It has been explained in the related embodiments shown and will not be described again here.

[0064] S404: Determine a target motion trajectory of a reference object in a target video frame set.

[0065] In one possible implementation, any one of the N video frame sets is used as the target video frame set. In this case, the computer device determines the target motion trajectory of the reference object in the target video frame set. The embodiment of the present application can use VIoU-tracker to track the motion trajectory of the reference object. The specific steps include: after determining the reference object, mark the reference object with a tracking id. The VIou-tracker tracking algorithm is used to track the tracking id of the reference object, and finally a target motion trajectory of the reference object is obtained. Among them, the target motion trajectory is used to describe the position change of the reference object in different video frames of the target video frame set, the horizontal coordinate of the target motion trajectory is used to indicate the video frame number where the reference object is located, and the vertical coordinate of the target motion trajectory is used to indicate the horizontal coordinate in the video frame where the reference object is located. The main working principle of VIoU-tracker is: first, all valid detection frames of two adjacent frames are obtained, and at the same time, the trajectory of the target detected in the previous frame is tracked, the position of the target in the next frame is predicted, and the IoU between the target predicted in the next frame and the target detected in the next frame is calculated (IoU is an indicator for calculating the overlap rate of two detection frames). The target with the largest IoU is considered to be the same target and matched. At the same time, for targets that are not matched, the method will continue to track the target. If it matches the detection result within the specified maximum prediction time range, it will be successful. This method can fill the blank range that cannot be matched. Corresponding to the embodiment of the present application, based on the image position of the reference object in the first video frame in the target video frame set, the tracking algorithm is used to predict the image position of the reference object in the next video frame after the first video frame, and the target motion trajectory of the reference object is determined according to the image position of the first video frame and the image position of the next video frame. A video frame set often includes multiple video frames, which may be dozens. The image position of the reference object in each video frame is determined in turn according to the tracking algorithm, and then the target motion trajectory of the reference object is determined according to the image position of the reference object in each video frame.

[0066] In another feasible embodiment, after obtaining the image positions of the reference objects in each video frame of the target video frame set, it is necessary to optimize these image positions to obtain the final target motion trajectory. Figure 5 , which is a schematic diagram of a target motion trajectory disclosed in an embodiment of the present application, Figure 5 The horizontal axis is the frame number of each video frame in a video frame set, and the vertical axis is the horizontal coordinate position of the reference object of the video frame ( Figure 5 ). In the embodiment of the present application, the horizontal version to vertical version mainly focuses on the motion trajectory of the horizontal coordinate of the reference object. In order to make the video lens movement have a better sense of motion, the motion trajectory needs to be further optimized. There are three main aspects that need to be optimized: a. Regression of trajectory points with large deviations; b. The movement of the lens needs to be smoothed when the reference object moves over a large range of the picture; c. The lens can be fixed when the reference object moves slightly.

[0067] Accordingly, the main steps of optimization include: the first step is to determine the subject motion trajectory points corresponding to the target video frame set according to the image position of the first video frame and the image position of the next video frame, and determine the target position points in the subject motion trajectory points that meet the deviation conditions, see Figure 5 , the hollow point is the target position point that meets the deviation condition; the corresponding deviation condition can be calculated Figure 5 The vertical coordinate mean μ and standard deviation σ of all discrete points are determined as target position points that meet the deviation conditions when the vertical coordinate value of the reference object is greater than μ+1.5×σ, or the vertical coordinate value of the reference object is less than μ-1.5×σ. These target position points are then deleted. The second step is to use polynomial regression fitting to predict the position points. The predicted position points are the position points corresponding to the deleted target position points. The third step is to use a one-dimensional Gaussian function to smooth the entire trajectory after the noise points are repositioned. For trajectories with small fluctuations, a horizontal line (i.e., the fixed lens mentioned above) can be used instead.

[0068] S405 : Determine, according to the target motion trajectory, an image region where a reference object is located in each target video frame included in the target video frame set.

[0069] The image region where the reference object is located in each target video frame included in the target video frame set is determined according to the target motion trajectory, that is, the image region is based on the target motion trajectory, that is, each image region contains the reference object.

[0070] S406: Determine a subtitle area in the image area where the reference object is located in each target video frame.

[0071] S407: Crop the image area according to the largest subtitle area to obtain a cropped image set.

[0072] S408: Generate a video clip of the video to be processed according to the cropped image set.

[0073] Correspondingly, steps S405-S408 have been Figure 2 The video processing method provided in the embodiment of the present application can be used to Figure 6 To give a general description, Figure 6 It is a flowchart of another video processing method disclosed in an embodiment of the present application. The general process is to obtain a video to be processed, perform frame processing and scene cutting on the video to be processed to obtain multiple video frame sets, use any one of the multiple video frame sets as a target video frame set, determine a reference object based on target detection and saliency detection, and determine the target motion trajectory of the reference object according to a tracking algorithm. At the same time, the processing process includes performing no pure color border detection and subtitle detection on the video frames in the target video frame set. Finally, the video frames corresponding to the target motion trajectory are cropped according to the no pure color border detection results and the subtitle detection results to obtain a video clip.

[0074] In this application, combined with the characteristics of music videos, a video processing method for vertical conversion is proposed with the following advantages: First, this application uses a deep learning method to detect the scene switching frames related to the rapid transition and gradual transition of the video, which can ensure that the scene is segmented and the transition frames are eliminated. In addition, in order to efficiently track the reference objects appearing in the video frame set, this application integrates character detection, saliency detection and face recognition to jointly locate the reference objects in the video frame set, and uses a tracker to track the inter-frame trajectory of the reference object in the entire video frame set. In order to prevent the jump or rapid movement of the trajectory from causing discomfort to the viewing experience, this application optimizes the obtained trajectory lines. Finally, this application uses the reference object in each video frame set as the center to perform horizontal to vertical cropping, which can well avoid the interference of subtitles in the picture. In general, in this application, the cropping of the video to be processed includes the cropping of the timing and the picture. In terms of timing, the complete video is cropped into a collection of clips; in terms of the picture, the image area where the reference object is located in each frame is cropped out.

[0075] Based on the above method embodiment, the present application embodiment also provides a structural diagram of a video processing device. Figure 7 , is a structural diagram of a video processing device provided in an embodiment of the present application. Figure 7 The video processing device 700 shown can run the following units:

[0076] An acquisition unit 701 is used to acquire a video frame sequence of a video to be processed;

[0077] The processing unit 702 is used to perform scene segmentation processing on the video frame sequence to obtain N video frame sets, each video frame set corresponds to a scene, and N is an integer greater than 1;

[0078] A determining unit 703 is configured to determine a reference object of a target video frame set among the N video frame sets, where the target video frame set is any one of the N video frame sets;

[0079] The processing unit 702 is further configured to crop an image region where a reference object is located in a target video frame included in the target video frame set to obtain a cropped image set corresponding to the target video frame set, wherein the target video frame is a video frame including the reference object in the target video frame set;

[0080] The processing unit 702 is further configured to generate a video segment of the video to be processed according to the cropped image set.

[0081] In a possible implementation, the processing unit 702 crops an image region where a reference object is located in the target video frame set to obtain a cropped image set corresponding to the target video frame set, specifically for:

[0082] Determine a target motion trajectory of the reference object in the target video frame set, wherein the target motion trajectory is used to describe a position change of the reference object in different target video frames, the abscissa of the target motion trajectory is used to indicate the target video frame number where the reference object is located, and the ordinate of the target motion trajectory is used to indicate the abscissa of the reference object in the target video frame where the reference object is located;

[0083] According to the target motion trajectory, determining an image region where the reference object is located in the target video frames included in the target video frame set;

[0084] The image region is cropped to obtain a cropped image set.

[0085] In a possible implementation, the processing unit 702 crops an image region where a reference object is located in the target video frame set to obtain a cropped image set corresponding to the target video frame set, specifically for:

[0086] Determining a subtitle area in each of the image areas;

[0087] The image area is cropped according to the largest subtitle area to obtain a cropped image set.

[0088] In a possible implementation, the processing unit 702 performs scene segmentation processing on the video frame sequence to obtain N video frame sets, which is specifically used for:

[0089] If the video frame sequence includes a scene switching frame, deleting the scene switching frame;

[0090] According to the position of the scene switching frame in the video sequence, the video frame sequence after deleting the scene switching frame is sorted to obtain N video frame sets.

[0091] In a possible implementation manner, the determining unit 703 determines the reference object of the target video frame set in the N video frame sets, specifically for:

[0092] For a target video frame set in the N video frame sets, determining an object corresponding to each video frame included in the target video frame set;

[0093] If the target video frame set corresponds to multiple objects, the object with the highest priority among the multiple objects of the target video frame set is determined as the reference object of the target video frame set according to a preset priority order, and the priority order is divided based on object type.

[0094] In a possible implementation, the determining unit 703 determines, according to a preset priority order, an object with the highest priority among multiple objects in the target video frame set as a reference object of the target video frame set, specifically for:

[0095] If the objects with the highest priority determined from the multiple objects according to the preset priority order include at least two, determining heat maps of the at least two objects in the target video frame set;

[0096] Calculating an average value of the thermal value of each of the at least two objects according to the thermal map;

[0097] The object corresponding to the highest average value of the thermal value is determined as the reference object of the target video frame set.

[0098] In a possible implementation manner, the determining unit 703 determines the target motion trajectory of the reference object in the target video frame set, specifically for:

[0099] Based on the image position of the reference object in the first video frame in the target video frame set, determine the image position of the reference object in the next video frame after the first video frame by using a tracking algorithm;

[0100] The target motion trajectory of the reference object is determined at least according to the image position of the first video frame and the image position of the next video frame.

[0101] In a possible implementation, the determining unit 703 determines the target motion trajectory of the reference object at least according to the image position of the first video frame and the image position of the next video frame, and is specifically configured to:

[0102] Determine the subject motion trajectory point corresponding to the target video frame set according to the image position of the first video frame and the image position of the next video frame;

[0103] Determine a target position point that satisfies a deviation condition among the subject motion trajectory points, and delete the target position point;

[0104] Using a polynomial regression method to fit a predicted position point, the predicted position point corresponds to the deleted target position point;

[0105] The subject motion trajectory points are optimized according to the predicted position points to obtain a target motion trajectory.

[0106] In a possible implementation, the processing unit 702 generates a video segment of the video to be processed according to the cropped image set, specifically for:

[0107] Converting the images in the cropped image set to vertical screen according to a preset size;

[0108] Generate a video clip of the video to be processed according to the image converted from vertical screen to vertical screen.

[0109] According to one embodiment of the present application, Figure 2 as well as Figure 4 The various steps involved in the video processing method shown can be Figure 7 The video processing device shown in the figure is executed by each unit. For example, Figure 2 In the video processing method shown in FIG. 1 , step S201 can be performed by Figure 7 The acquisition unit 701 in the video processing device shown in FIG. 1 is used to perform the steps S202 and S204. Figure 7 The processing unit 702 in the video processing device shown in FIG. 1 is used to perform step S203. Figure 7 The determination unit 703 in the video processing device shown in the figure is executed; for example, Figure 4 In the video processing method shown in FIG. 1 , step S401 can be performed by Figure 7 The acquisition unit 701 in the video processing device shown in FIG. 1 is used to perform the steps S402 and S407-S408. Figure 7The processing unit 702 in the video processing device shown in FIG. 1 is used to perform the steps S403-S406. Figure 7 The determination unit 703 in the video processing device shown is executed.

[0110] According to another embodiment of the present application, Figure 7 The various units in the video processing device shown can be respectively or completely merged into one or several other units to constitute, or some of the units (some) can also be split into multiple smaller units in function to constitute, which can achieve the same operation without affecting the realization of the technical effects of the embodiments of the present application. The above-mentioned units are divided based on logical functions. In practical applications, the functions of one unit can also be realized by multiple units, or the functions of multiple units can be realized by one unit. In other embodiments of the present application, other units can also be included based on the video processing device. In practical applications, these functions can also be implemented with the assistance of other units, and can be implemented by the collaboration of multiple units.

[0111] According to another embodiment of the present application, the program can be executed by running on a general computing device such as a computer including a central processing unit (CPU), a random access memory medium (RAM), a read-only memory medium (ROM), and other processing elements and storage elements. Figure 2 as well as Figure 4 A computer program (including program code) for each step involved in the corresponding method shown in FIG. Figure 7 The video processing device shown in and the video processing method of the embodiment of the present application are implemented. The computer program can be recorded on, for example, a computer-readable storage medium, and loaded into the above-mentioned computer device through the computer-readable storage medium, and run therein.

[0112] In the embodiment of the present application, the acquisition unit 701 acquires the video frame sequence of the processed video, and then the processing unit 702 performs scene cutting processing on the video frame sequence, removes the scene switching video frame, and obtains N video frame sets, each video frame set corresponds to a scene, and then the determination unit 703 determines the reference object of the target video frame set in the N video frame sets, and obtains the cropped image set according to the image area where the reference object is located in the target video frame included in the target video frame set, and finally the processing unit 702 generates a video clip of the video to be processed according to the cropped image set. Through this method, by cutting the video scene, and then according to the reference object in each scene and the image area including the reference object, the video frame can be automatically cropped to obtain a video clip, thereby improving the processing efficiency of the video.

[0113] Based on the above method and device embodiments, the present application embodiment provides a computer device. Figure 8, is a structural diagram of a computer device provided in an embodiment of the present application. Figure 8 The computer device 800 shown includes at least a processor 801, an input interface 802, an output interface 803, a computer storage medium 804, and a memory 805. The processor 801, the input interface 802, the output interface 803, the computer storage medium 804, and the memory 805 may be connected via a bus or other means.

[0114] The computer storage medium 804 may be stored in the memory 805 of the computer device 800, the computer storage medium 804 is used to store a computer program, the computer program includes program instructions, and the processor 801 is used to execute the program instructions stored in the computer storage medium 804. The processor 801 (or CPU (Central Processing Unit)) is the computing core and control core of the computer device 800, which is suitable for implementing one or more instructions, and is specifically suitable for loading and executing one or more computer instructions to implement the corresponding method flow or corresponding function.

[0115] The embodiment of the present application also provides a computer storage medium (Memory), which is a memory device in the computer device 800 for storing programs and data. It is understandable that the computer storage medium here can include both the built-in storage medium in the computer device 800 and the extended storage medium supported by the computer device 800. The computer storage medium provides a storage space, which stores the operating system of the computer device 800. In addition, one or more instructions suitable for being loaded and executed by the processor 801 are also stored in the storage space, and these instructions can be one or more computer programs (including program codes). It should be noted that the computer storage medium here can be a high-speed RAM memory, or a non-volatile memory (non-volatile memory), such as at least one disk storage; optionally, it can also be at least one computer storage medium located away from the aforementioned processor.

[0116] In one embodiment, the computer storage medium can be loaded by the processor 801 and execute one or more instructions stored in the computer storage medium to implement the above-mentioned Figure 2 , Figure 4 as well as Figure 6 In a specific implementation, one or more instructions in the computer storage medium are loaded by the processor 801 and execute the following steps:

[0117] Get the video frame sequence of the video to be processed;

[0118] Performing scene segmentation processing on the video frame sequence to obtain N video frame sets, each video frame set corresponds to a scene, and N is an integer greater than 1;

[0119] Determine a reference object of a target video frame set among the N video frame sets, where the target video frame set is any one of the N video frame sets;

[0120] For the target video frame set, cropping an image region where a reference object is located in the target video frame included in the target video frame set to obtain a cropped image set corresponding to the target video frame set, wherein the target video frame is a video frame in the target video frame set that includes the reference object;

[0121] A video segment of the video to be processed is generated according to the cropped image set.

[0122] In a possible implementation, the processor 801 crops an image region where a reference object is located in the target video frame set to obtain a cropped image set corresponding to the target video frame set, specifically for:

[0123] Determine a target motion trajectory of the reference object in the target video frame set, wherein the target motion trajectory is used to describe a position change of the reference object in different target video frames, the abscissa of the target motion trajectory is used to indicate the target video frame number where the reference object is located, and the ordinate of the target motion trajectory is used to indicate the abscissa of the reference object in the target video frame where the reference object is located;

[0124] According to the target motion trajectory, determining an image region where the reference object is located in the target video frames included in the target video frame set;

[0125] The image region is cropped to obtain a cropped image set.

[0126] In a possible implementation, the processor 801 crops an image region where a reference object is located in the target video frame set to obtain a cropped image set corresponding to the target video frame set, specifically for:

[0127] Determining a subtitle area in each of the image areas;

[0128] The image area is cropped according to the largest subtitle area to obtain a cropped image set.

[0129] In a possible implementation, the processor 801 performs scene segmentation processing on the video frame sequence to obtain N video frame sets, which is specifically used for:

[0130] If the video frame sequence includes a scene switching frame, deleting the scene switching frame;

[0131] According to the position of the scene switching frame in the video sequence, the video frame sequence after deleting the scene switching frame is sorted to obtain N video frame sets.

[0132] In a possible implementation manner, the processor 801 determines a reference object of a target video frame set in the N video frame sets, specifically for:

[0133] For a target video frame set in the N video frame sets, determining an object corresponding to each video frame included in the target video frame set;

[0134] If the target video frame set corresponds to multiple objects, the object with the highest priority among the multiple objects of the target video frame set is determined as the reference object of the target video frame set according to a preset priority order, and the priority order is divided based on object type.

[0135] In a possible implementation manner, the processor 801 determines, according to a preset priority order, an object with the highest priority among multiple objects in the target video frame set as a reference object of the target video frame set, specifically for:

[0136] If the objects with the highest priority determined from the multiple objects according to the preset priority order include at least two, determining heat maps of the at least two objects in the target video frame set;

[0137] Calculating an average value of the thermal value of each of the at least two objects according to the thermal map;

[0138] The object corresponding to the highest average value of the thermal value is determined as the reference object of the target video frame set.

[0139] In a possible implementation manner, the processor 801 determines the target motion trajectory of the reference object in the target video frame set, specifically for:

[0140] Based on the image position of the reference object in the first video frame in the target video frame set, determine the image position of the reference object in the next video frame after the first video frame by using a tracking algorithm;

[0141] The target motion trajectory of the reference object is determined at least according to the image position of the first video frame and the image position of the next video frame.

[0142] In a possible implementation manner, the processor 801 determines the target motion trajectory of the reference object at least according to the image position of the first video frame and the image position of the next video frame, specifically for:

[0143] Determine the subject motion trajectory point corresponding to the target video frame set according to the image position of the first video frame and the image position of the next video frame;

[0144] Determine a target position point that satisfies a deviation condition among the subject motion trajectory points, and delete the target position point;

[0145] Using a polynomial regression method to fit a predicted position point, the predicted position point corresponds to the deleted target position point;

[0146] The subject motion trajectory points are optimized according to the predicted position points to obtain a target motion trajectory.

[0147] In a possible implementation manner, the processor 801 generates a video segment of the video to be processed according to the cropped image set, specifically for:

[0148] Converting the images in the cropped image set to vertical screen according to a preset size;

[0149] Generate a video clip of the video to be processed according to the image converted from vertical screen to vertical screen.

[0150] In the embodiment of the present application, the processor 801 included in the computer device 800 obtains a video frame sequence of a processed video, and then performs scene cutting processing on the video frame sequence, removes the scene switching video frame, and obtains N video frame sets, each video frame set corresponds to a scene, and then determines the reference object of the target video frame set in the N video frame sets, and obtains a cropped image set according to the image area where the reference object is located in the target video frame included in the target video frame set, and finally generates a video clip of the video to be processed according to the cropped image set. Through this method, by cutting the video scene, and then according to the reference object in each scene and the image area including the reference object, the video frame can be automatically cropped to obtain a video clip, thereby improving the processing efficiency of the video.

[0151] According to one aspect of the present application, the embodiment of the present application further provides a computer product or a computer program, the computer product or the computer program includes a computer instruction, the computer instruction is stored in a computer-readable storage medium. The processor 801 reads the computer instruction from the computer-readable storage medium, and the processor 801 executes the computer instruction, so that the computer device 800 executes Figure 2 , Figure 4 as well as Figure 6The video processing method shown.

[0152] It should be noted that, for the above-mentioned method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the order of the actions described, because according to the present application, some steps may be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present application.

[0153] In the several embodiments provided in this application, it should be understood that the disclosed device can be implemented in other ways. For example, the device embodiments described above are only schematic, and the division of the modules described above is only a logical function division. There may be other division methods in actual implementation, such as multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed.

[0154] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A video processing method, characterized in that: The method comprises: Obtaining a video frame sequence of a video to be processed, wherein the video frame sequence is sorted according to the playback order of the video to be processed; Performing scene segmentation processing on the video frame sequence to obtain N video frame sets, each video frame set corresponds to a scene, and N is an integer greater than 1; Determine a reference object of a target video frame set in the N video frame sets, the target video frame set being any one of the N video frame sets, and the determining of the reference object of the target video frame set in the N video frame sets comprises: for the target video frame set in the N video frame sets, determine an object corresponding to each video frame included in the target video frame set; if there are multiple objects corresponding to the target video frame set, if the objects with the highest priority determined from the multiple objects according to a preset priority order include at least two, determine a thermal map of the at least two objects in the target video frame set; calculate an average value of the thermal value of each object in the at least two objects according to the thermal map; and determine the object corresponding to the highest average value of the thermal value as the reference object of the target video frame set; For the target video frame set, cropping an image region where a reference object is located in the target video frame included in the target video frame set to obtain a cropped image set corresponding to the target video frame set, wherein the target video frame is a video frame in the target video frame set that includes the reference object; A video segment of the video to be processed is generated according to the cropped image set.

2. The method according to claim 1, characterized in that The step of cropping an image region where a reference object is located in the target video frame set to obtain a cropped image set corresponding to the target video frame set includes: Determine a target motion trajectory of the reference object in the target video frame set, wherein the target motion trajectory is used to describe a position change of the reference object in different target video frames, the abscissa of the target motion trajectory is used to indicate the target video frame number where the reference object is located, and the ordinate of the target motion trajectory is used to indicate the abscissa of the reference object in the target video frame where the reference object is located; According to the target motion trajectory, determining an image region where the reference object is located in the target video frames included in the target video frame set; The image region is cropped to obtain a cropped image set.

3. The method according to claim 2, characterized in that The step of cropping an image region where a reference object is located in the target video frame set to obtain a cropped image set corresponding to the target video frame set includes: Determining a subtitle area in each of the image areas; The image area is cropped according to the largest subtitle area to obtain a cropped image set.

4. The method according to claim 1, characterized in that: The performing scene segmentation processing on the video frame sequence to obtain N video frame sets includes: If the video frame sequence includes a scene switching frame, deleting the scene switching frame; According to the position of the scene switching frame in the video sequence, the video frame sequence after deleting the scene switching frame is sorted to obtain N video frame sets.

5. The method according to claim 2, wherein determining the target motion trajectory of the reference object in the target video frame set comprises: Based on the image position of the reference object in the first video frame in the target video frame set, determine the image position of the reference object in the next video frame after the first video frame by using a tracking algorithm; The target motion trajectory of the reference object is determined at least according to the image position of the first video frame and the image position of the next video frame.

6. The method according to claim 5, characterized in that The step of determining the target motion trajectory of the reference object at least according to the image position of the first video frame and the image position of the next video frame includes: Determine the subject motion trajectory point corresponding to the target video frame set according to the image position of the first video frame and the image position of the next video frame; Determine a target position point that satisfies a deviation condition among the subject motion trajectory points, and delete the target position point; Using a polynomial regression method to fit a predicted position point, the predicted position point corresponds to the deleted target position point; The subject motion trajectory points are optimized according to the predicted position points to obtain a target motion trajectory.

7. The method according to claim 1, characterized in that The step of generating a video segment of the video to be processed according to the cropped image set includes: Converting the images in the cropped image set to vertical screen according to a preset size; Generate a video clip of the video to be processed according to the image converted from vertical screen to vertical screen.

8. A computer device, characterized in that: The computer device comprises: a processor adapted to implement one or more computer programs; and, A computer storage medium storing one or more computer programs, wherein the one or more computer programs are suitable for being loaded by the processor and executing the video processing method according to any one of claims 1 to 7.

9. A computer-readable storage medium, characterized in that: The computer storage medium stores one or more computer programs, and the one or more computer programs are suitable for being loaded by a processor and executing the video processing method according to any one of claims 1 to 7.