Video cutting method, device and storage medium

Abstract

The present application discloses a video cropping method, device and storage medium, characterized in that it includes: obtaining N image frames in a first video, and performing salient area detection on the N image frames respectively to obtain a salient area heat map of each of the N image frames; summing the pixel values ​​in each column of image data in the salient area heat map of each of the image frames to obtain a longitudinal projection map of each of the image frames; determining the center point of the area to be cropped of each of the image frames according to the weighted sum of the areas of one or more peaks of the longitudinal projection map of each of the image frames; determining the cropping area of ​​each of the N image frames according to the center point of the area to be cropped of each of the image frames and a preset cropping size. By adopting the present application, the amount of calculation required for cropping can be reduced while optimizing the display effect of the target object in the interface after video cropping.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a video data processing method, device and storage medium. Background Art

[0002] Video intelligent cropping technology has a wide range of applications on the product side. With the advent of the 5th generation mobile networks (5G) and the development of mobile Internet platforms, the demand for short videos is increasing. The method of generating short videos using video intelligent cropping technology increases the content diversity of video platforms on the one hand, and increases the distribution volume of the platform on the other hand. At the same time, it can also change the interactive experience on the mobile terminal, which can greatly improve the user experience of the product. Many professionally generated content (PGC) video platforms such as Tencent Video and TV stations use vertical screen videos for mobile video playback, which is more in line with the human body's habitual viewing posture. Video intelligent cropping technology can be used to generate vertical screen videos. When the amount of data on the mobile terminal or platform is relatively large, it is very necessary to accelerate the time-consuming performance of cropping.

[0003] At present, video intelligent cropping technology usually crops the spatial domain of the video frame by frame. First, the size and direction of the cropping sliding window are determined according to the cropping ratio. Based on the cropping sliding window and the heat map of the salient area, the sliding window contains multiple largest salient areas in a certain cropping position. In this way, the cropping position of each frame is determined. At the same time, in order to avoid the shaking of the cropped lens, local smoothing and scene switching detection are performed on the final cropping position.

[0004] Among them, the video intelligent cropping technology mainly includes the time domain frame extraction module, the region of interest (ROI) area detection module including black and white edge detection and subtitle watermark detection, the salient area detection module, the post-processing module based on the saliency heat map, and the video re-encoding module. After performance analysis, the time loss is mainly concentrated in the time domain frame extraction module and the post-processing module based on the saliency heat map.

[0005] For example, the current method uses a sliding window combined with a heat map of a salient area to calculate the area for spatial cropping, which is very time-consuming. In addition, in order to achieve a balance with the human eye, smoothing the final result is essentially an operation that eliminates position accuracy. This frame-by-frame detection method has many redundant operations. Another point is that there are certain limitations when determining the ROI area. The fact that the area of ​​multiple salient areas in the sliding window is the largest does not mean that the main body of the salient area is in the center of the cropped image, resulting in an inaccurate ROI area and affecting the visual effect. Summary of the invention

[0006] The embodiments of the present application provide a video cropping method, device and storage medium, which can reduce the amount of calculation required for cropping while optimizing the display effect of the target object in the interface after video cropping.

[0007] On the one hand, an embodiment of the present application provides a video cropping method, the method comprising:

[0008] Obtaining N image frames in the first video, and performing salient region detection on the N image frames respectively, to obtain a salient region heat map of each of the N image frames, where N is a positive integer; wherein the salient region heat map of each of the image frames is used to indicate a salient degree of at least one salient region in the corresponding image frame;

[0009] The pixel values ​​in each column of the image data in the heat map of the salient region of each of the image frames are summed to obtain a longitudinal projection map of each of the image frames; the abscissa of the longitudinal projection map is the lateral position of each column of the image data in the corresponding image frame, and the ordinate is the sum of the pixel values ​​in each column of the image data;

[0010] Determine the center point of the to-be-cropped region of each of the image frames according to a weighted sum of the areas of one or more peaks of the longitudinal projection image of each of the image frames;

[0011] The cropping area of ​​each of the N image frames is determined according to the center point of the to-be-cropped area of ​​each of the image frames and a preset cropping size.

[0012] The method further includes: acquiring a first video, the first video including M image frames, where M is a positive integer; and sparsely sampling the M image frames to obtain the N image frames, where N is a positive integer less than M.

[0013] The method further includes: performing non-significant area detection on the N image frames to obtain non-significant area features of each of the image frames, wherein the non-significant area features include one or more of upper and lower or left and right black and white borders, subtitles, and watermarks in the image frames; and removing the non-significant area features in each of the image frames to obtain a pure image frame of each of the image frames.

[0014] Wherein, the salient area detection is performed on the N image frames respectively to obtain a salient area heat map of each of the N image frames; including: performing salient area detection on the pure image frame of each of the image frames respectively to obtain a salient area heat map of each of the N image frames.

[0015] The method further includes: performing video scene segmentation detection on the M image frames to obtain scene switching frames of the M image frames; and interpolating the N image frames according to the scene switching frames to obtain M restored image frames.

[0016] Among them, the audio of the first video is a first audio clip; the method also includes: cropping the first video to obtain a second video according to the central cropping area of ​​the N image frames and the M restored image frames; and re-encoding the video clip of the second video and the first audio clip to obtain a cropped complete video corresponding to the first video.

[0017] On the one hand, an embodiment of the present application provides a video cropping device, the device comprising:

[0018] A detection module, used to obtain N image frames in the first video, and perform salient region detection on the N image frames respectively, to obtain a salient region heat map of each of the N image frames, where N is a positive integer; wherein the salient region heat map of each of the image frames is used to indicate a salient degree of at least one salient region in the corresponding image frame;

[0019] A longitudinal projection module, used for summing the pixel values ​​in each column of image data in the heat map of the salient region of each image frame to obtain a longitudinal projection map of each image frame; the abscissa of the longitudinal projection map is the lateral position of each column of image data in the corresponding image frame, and the ordinate is the sum of the pixel values ​​in each column of image data;

[0020] A center point determination module, used to determine the center point of the to-be-cropped area of ​​each of the image frames according to a weighted sum of the areas of one or more peaks of the longitudinal projection image of each of the image frames;

[0021] The cropping region determining module is used to determine the cropping region of each of the N image frames according to the center point of the to-be-cropped region of each of the image frames and a preset cropping size.

[0022] Wherein, the device further comprises:

[0023] The acquisition module is used to acquire a first video, wherein the first video includes M image frames, where M is a positive integer; the sampling module is used to perform sparse sampling on the M image frames to obtain the N image frames, where N is a positive integer less than M. The embodiment of the present application mainly reduces the amount of calculation by replacing dense sampling with sparse sampling.

[0024] Wherein, the device further comprises:

[0025] The non-significant area detection module is used to perform non-significant area detection on the N image frames to obtain the non-significant area features of each of the image frames, wherein the non-significant area features include one or more of the upper and lower or left and right black and white borders, subtitles and watermarks in the image frame; the non-significant area features in each of the image frames are removed to obtain a pure image frame of each of the image frames. The embodiment of the present application mainly detects the text area in the picture through the OCR-based character recognition technology and performs keyword matching and position statistics of the watermark in the text area. Finally, these factors can be removed to achieve a better subjective effect.

[0026] The detection module is further used to perform salient area detection on the clean image frame of each of the image frames to obtain a salient area heat map of each of the N image frames. In the embodiment of the present application, when the clean version of the image frame is detected, the obtained cropped area can be made more consistent with the perception of the human eye.

[0027] Wherein, the device further comprises:

[0028] The restoration module is used to perform video scene segmentation detection on the M image frames to obtain scene switching frames of the M image frames; according to the scene switching frames, the N image frames are interpolated to obtain M restored image frames. In the embodiment of the present application, a sparse sampling frame extraction strategy is designed by utilizing the visual persistence effect of the human eye and the fact that the ROI area in the same scene will not undergo positional mutations. The ROI position of the remaining frames is restored by combining interpolation with video scene switching detection, thereby achieving the same time domain detection result as dense frame-by-frame detection.

[0029] Wherein, the device further comprises:

[0030] The integration module is used to crop the first video according to the central cropping area of ​​the N image frames and the M restored image frames to obtain a second video; and re-encode the video segment of the second video and the first audio segment to obtain a cropped complete video corresponding to the first video. In this embodiment of the application, cropping is performed according to the determined central cropping area to finally obtain a cropped video.

[0031] On the one hand, an embodiment of the present application provides a computer device, the computer device comprising: a processor and a memory;

[0032] The processor is connected to the memory, wherein the memory is used to store a computer program, and the processor is used to call the computer program so that the computer device executes the method in any aspect of the embodiments of the present application.

[0033] On one hand, an embodiment of the present application provides a computer-readable storage medium, in which a computer program is stored. The computer program is suitable for being loaded and executed by a processor so that a computer device having a processor executes the method in any aspect of the embodiment of the present application.

[0034] In one aspect, the present application provides a computer program product or a computer program, which includes a computer instruction stored in a computer-readable storage medium. A processor of a computer device reads the computer instruction from the computer-readable storage medium, and the processor executes the computer instruction, so that the computer device executes the method in any aspect of the present application.

[0035] In an embodiment of the present application, by performing salient area detection on N image frames in a video to be cropped, a salient area heat map of each image frame in the N image frames is obtained, where the salient area heat map of each image frame is used to indicate the salient degree of at least one salient area in the corresponding image frame, that is, the area of ​​interest to the human eye in each frame of the image can be identified; further, the pixel values ​​in each column of image data in the salient area heat map of each image frame are summed to obtain a longitudinal projection map of each of the image frames, where the longitudinal projection map is a coordinate map, and the abscissa is a coordinate map indicating the salient area of ​​each image frame. The horizontal position of each column of image data in the domain thermal map in the corresponding image frame, and the vertical coordinate represents the sum of the pixel values ​​in each column of image data, that is, the longitudinal projection diagram is a curve diagram with peaks and troughs; further, according to the weighted sum of the areas of one or more peaks of the longitudinal projection diagram of each image frame, the center point of the area to be cropped of each image frame is determined, wherein the central cropping area of ​​each frame is determined based on the weighted sum of the areas of the peaks in the longitudinal projection diagram of each frame; finally, the cropping area of ​​each of the N image frames can be determined based on the center point of the area to be cropped of each image frame and the preset cropping size. The above scheme in the present application only needs to calculate the heat map of the salient area of ​​the image frame and its longitudinal projection map to obtain the central cropping area, so the amount of calculation for subsequent processing can be reduced, and the problem of large amount of calculation caused by the sliding window in the sliding of the cropping frame to obtain the cropping area in the prior art is avoided; in addition, since the embodiment of the present application adopts the method of first determining the center point and then confirming the central cropping area, the center point of the salient area can be cropped, so that the main body of the salient area is located in the center of the cropping frame to optimize the final display effect, avoiding the problem of inaccurate display effect caused by sliding the sliding window in the cropping frame in the prior art. Optionally, the N image frames can be obtained by sparse sampling of the M image frames in the video to be cropped, and the use of sparse sampling here can further reduce the time consumed by subsequent post-processing. It can be seen that by implementing the embodiment of the present application, the cropping area of ​​each image frame in the N image frames in the video to be cropped will be more in line with the visual range of the human eye, and the amount of calculation required for video cropping can be greatly reduced. That is, the embodiment of the present application provides a solution that can reduce the amount of calculation required for cropping while optimizing the display effect of the target object in the interface after video cropping. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0037] Figure 1 It is a structural diagram of a network architecture provided by an embodiment of the present application;

[0038] Figure 2a This is a schematic diagram of a terminal device playing a video in a horizontal screen mode provided in an embodiment of the present application;

[0039] Figure 2b It is a schematic diagram of a terminal device playing a video and switching from a horizontal screen to a vertical screen provided by an embodiment of the present application;

[0040] Figure 2c This is a horizontal screen schematic diagram of a video with a watermark played by a terminal device provided in an embodiment of the present application;

[0041] Figure 2d This is a vertical screen schematic diagram of removing a watermark from a video played by a terminal device provided in an embodiment of the present application;

[0042] Figure 3 It is a flowchart of a video intelligent cropping method provided in an embodiment of the present application;

[0043] Figure 4 It is a cropping position prediction comparison diagram of cropping an image frame in different ways provided by an embodiment of the present application;

[0044] Figure 5 is a schematic diagram of salient area detection provided in an embodiment of the present application;

[0045] Figure 6a This is a schematic diagram of a video played by a terminal device before watermark removal provided in an embodiment of the present application;

[0046] Figure 6b This is a schematic diagram of a video played by a terminal device after watermark removal provided in an embodiment of the present application;

[0047] Figure 6c It is a video played by a terminal device provided in an embodiment of the present application and includes a schematic diagram of black and white edges;

[0048] Figure 7 It is an original image before the image frame is cropped in a first video provided in an embodiment of the present application;

[0049] Figure 8A heat map of a salient region of an original image before image frame cropping in a first video provided in an embodiment of the present application;

[0050] Fig. 9 It is a longitudinal projection diagram of a heat map of a salient area of ​​an original image before cropping provided in an embodiment of the present application;

[0051] Fig.10 This is a comparison diagram of the score distribution before and after the video cropping optimization process provided by an embodiment of the present application;

[0052] Fig.11 is a flow chart of a video cropping method provided in an embodiment of the present application;

[0053] Fig.12a It is a continuous frame image of a video provided by an embodiment of the present application;

[0054] Figure 12b It is another continuous frame image of a video provided in an embodiment of the present application;

[0055] Fig.12c It is another continuous frame image of a video provided in an embodiment of the present application;

[0056] Fig.13 is a structural schematic diagram of a video cropping device provided in an embodiment of the present application;

[0057] Fig.14 is a structural schematic diagram of a computer device provided in an embodiment of the present application;

[0058] Fig.15 It is a structural diagram of a video cropping system provided in an embodiment of the present application. DETAILED DESCRIPTION

[0059] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0060] For further information, see Figure 1 , Figure 1 Schematic diagram of a network architecture provided by an embodiment of the present application. Figure 1 As shown, the network architecture can be applied to a video processing system. The video processing system may specifically include Figure 1The server 100, the terminal device cluster and the audience cluster are shown. The terminal device cluster may include one or more terminals, and the number of terminals in the terminal device cluster is not limited. Figure 1 As shown, the multiple terminal devices may specifically include terminal device 110, terminal device 120, terminal device 130, etc.; each terminal device may be connected to the server 100 through a network, so that each terminal device may exchange data with the server 100 through the network connection. Any video cropping method in the present application may be applied to the above Figure 1 On the server or terminal device in the architecture, for example, when any video cropping method in the present application is applied to the server 100, any video cropping method in the present application can be executed on the server 100 side, so that the video resource can be pre-cropped on the server 100 to obtain the cropped video resource, and the user can directly obtain the pre-cropped video stream on the server 100 side through the terminal device 110, the terminal device 120, or the terminal device 130 to enjoy the video playback experience and service. For another example, when any video cropping method in the present application is applied to the terminal device, any video cropping method in the present application can be executed on the terminal device side, so that after the terminal device obtains the original video resource on the server 100, the video is cropped locally through the back end (such as a related processor) to obtain the cropped video resource, and finally, the cropped video resource is presented through the front end (such as a display screen) of the terminal device 110, the terminal device 120, or the terminal device 130 to enjoy the video playback experience and service.

[0061] For specific application scenarios of this application, please refer to Figure 2a and Figure 2b , Figure 2a is a schematic diagram of a terminal device playing a video in a horizontal screen mode provided in an embodiment of the present application, Figure 2b is a schematic diagram of a terminal device playing a video and switching from a horizontal screen to a vertical screen provided by an embodiment of the present application. The main application scenarios of the embodiment of the present application may include Figure 2a The horizontal screen is cropped into Figure 2b Because many PGC video platforms such as Tencent Video, TV stations and other mobile terminals use vertical screen videos for video playback, which is more in line with the human body's habitual viewing posture, video intelligent cropping technology can be used to generate vertical screen videos. When the amount of data on the mobile terminal or platform is relatively large, it is very necessary to accelerate the time-consuming performance of cropping. Therefore, the main application scenario of the embodiment of the present application is to crop the video playback of mobile terminals such as PGC video platforms such as Tencent Video, TV stations, etc. into vertical screen videos that are more in line with the human body's habitual viewing posture.

[0062] Optionally, the application scenario of this application can also be applied to the original video with some watermarks and letters that are not interesting to the human eye. Figure 2c and Figure 2d , Figure 2c It is a horizontal screen schematic diagram of a video with a watermark played on a terminal device provided in an embodiment of the present application. Figure 2d is a vertical screen schematic diagram of removing watermarks from a video played by a terminal device provided in an embodiment of the present application. Similarly, the main application scenarios of the embodiment of the present application may include Figure 2c The horizontal screen with watermark is cropped into Figure 2d Vertical screen without watermark. Because many PGC video platforms such as Tencent Video, TV stations and other mobile terminals use vertical screen videos for video playback, which is more in line with the human body's habitual viewing posture, video intelligent cropping technology can be used to generate vertical screen videos. When the amount of data on the mobile terminal or platform is relatively large, the time-consuming performance acceleration of cropping is very necessary. Therefore, the main application scenario of the embodiment of the present application is to crop the video playback of mobile terminals such as PGC video platforms such as Tencent Video, TV stations, etc. into vertical screen videos that are more in line with the human body's habitual viewing posture.

[0063] For further understanding, please refer to Figure 3 , Figure 3 1 is a flow chart of a video intelligent cropping method provided by an embodiment of the present application. Figure 3 As shown in the figure, the video intelligent cropping method can mainly include a time domain frame extraction module, a non-ROI area detection module including black and white edge detection and subtitle watermark detection, a salient area detection module, a post-processing module based on a saliency heat map, and a video re-encoding module. After performance analysis, the time loss is mainly concentrated in the time domain frame extraction module and the post-processing module based on a saliency heat map.

[0064] Among them, the embodiments of the present application mainly optimize the frame extraction module in the time domain and the post-processing module based on the saliency heat map.

[0065] Specifically, Figure 4 As shown, Figure 4 : is a comparison diagram of cropping position predictions of image frames by different methods provided by the embodiment of the present application. The dotted line is the original dense sampling prediction position, the black dots are the sparse sampling detection points, and the solid line is the prediction position interpolated by the sparse detection points. Figure 4It can be seen that through the sparse sampling frame extraction strategy, combined with interpolation and video scene switching detection to restore the ROI position of the remaining frames, the same time domain detection result as dense frame-by-frame detection is achieved. Because of the optimization of the time domain frame extraction module, for video intelligent cropping technology, the difficulty of ROI-based cropping is that the ROI area changes with the change of frames. In order to more accurately detect the ROI area, the current dense detection method is used, that is, each frame performs significant area detection, but this dense detection brings huge performance loss. The embodiment of the present application uses the visual persistence effect of the human eye combined with the two characteristics that the ROI area will not undergo positional mutations in the same scene, and designs a sparse sampling frame extraction strategy, and then combines interpolation and video scene switching detection to restore the ROI position of the remaining frames, achieving the same time domain detection result as dense frame-by-frame detection.

[0066] In addition, specific Figure 3 In the non-ROI area detection module shown in the figure, in the spatial detection process of the video, there are some factors that seriously affect the subjective effect of the video. Some videos have black and white borders on the top and bottom or left and right, and some videos contain subtitles or even watermarks. For these parts that users do not want in the cropped video screen, the embodiment of the present application can detect the text area in the screen through character recognition technology based on optical character recognition (Optical Character Recognition, OCR) and perform keyword matching and position statistics of the watermark in the text area. Finally, these factors can be removed to achieve a better subjective effect.

[0067] Specifically, Figure 5 As shown, Figure 5 is a schematic diagram of salient area detection provided by an embodiment of the present application. Figure 3An example diagram of a salient region monitoring module in . In the module based on salient region detection, the embodiment of the present application adopts a video salient region detection based on a spatiotemporal dual-stream network to detect the salient region of the video. The architecture includes a temporal stream network and a spatial stream network. In terms of extracting visual features, the temporal stream network and the spatial stream network share parameters. In the modeling of the spatial stream network, the open source software development website (SourceForge.net, SF-Net) is a backbone network based on the Visual Geometry Group Network (VGG-16) model, and a multi-level feature fusion method is designed to extract basic features related to saliency. Then, the embodiment of the present application designs a service provider network (ServiceProvider A SP-Network is used to integrate spatial features and Gaussian prior features of different scales. In the modeling of the temporal stream network, the embodiment of the present application performs spatial modeling based on a network combining three-dimensional (3D) convolution and two-dimensional (2D) convolution. First, based on SF-Net, feature extraction is performed from input frames of multiple time dimensions under the condition of shared parameters to form significant features of the time domain dimension. Then, the feature extraction is sent to a shallow 3D convolution to obtain a time domain significant feature sequence, thereby realizing a spatiotemporal dual-stream model for significant region detection.

[0068] Optionally, before performing the salient area detection, the method further includes performing non-significant area detection on the N image frames to obtain non-significant area features of each image frame, wherein the non-significant area features include one or more of upper and lower or left and right black and white borders, subtitles, and watermarks in the image frame; and removing the non-significant area features in each image frame to obtain a pure image frame of each image frame.

[0069] Specifically, Figure 6a , Figure 6a 1 is a schematic diagram of a video played by a terminal device before watermark removal provided by an embodiment of the present application. It can be clearly seen that the video has watermarks and letters. The watermarks and letters of the picture can be removed by the present application. Figure 6b The picture, Figure 6b is a schematic diagram of a video played by a terminal device after watermark removal provided by an embodiment of the present application. At the same time, the picture may contain some black and white borders, such as Figure 6c , Figure 6cThis is a schematic diagram of a terminal device playing a video and including black and white borders provided by an embodiment of the present application. Because in the process of spatial detection of the video, there are some factors that seriously affect the subjective effect of the video. Some videos have black and white borders on the top and bottom or left and right, and some videos contain subtitles or even watermarks. For these parts, users do not want to see them in the cropped video screen. The embodiment of the present application can detect the text area in the screen through OCR-based character recognition technology and perform keyword matching and position statistics of the watermark in the text area. Finally, these factors can be removed to achieve a better subjective effect.

[0070] Specifically, Figure 7-Figure 9 Shown is Figure 3 Optimization process diagram of the post-processing module based on the saliency heat map in . Figure 7 It is an original image before the image frame is cropped in a first video provided in an embodiment of the present application. Figure 8 It is a heat map of the salient areas of the original image before the image frame is cropped in the first video provided in an embodiment of the present application. Fig. 9 It is a longitudinal projection diagram of a heat map of a salient region of an original image before image frame cropping in a first video provided by an embodiment of the present application. Specifically, Figure 7 The picture shows a whale in the sea in the center of the picture, with a string of letters in the lower right corner. Figure 5 The detection of the salient area of ​​the picture is obtained Figure 8 The heat map of the significant regions, from Figure 8 It can be clearly seen that there are two obvious white areas, which can be seen as the main body of the whale and the letters in the lower right corner. That is to say, the salient area obtained by salient area detection has two parts, namely the whale and the letters (that is, the area that the human eye may usually be more interested in). Then the heat map of this salient area is Figure 8 Perform longitudinal projection to obtain Fig. 9 , Fig. 9 is a longitudinal projection of a heat map of a salient region of an original image before cropping provided in an embodiment of the present application, corresponding to the two salient regions of the heat map above, Fig. 9There are also two peaks. The first peak with a larger area corresponds to the main body of the whale, and the smaller peak on the right is the letter in the lower right corner. Finally, according to this longitudinal projection, the areas of the two peaks are weighted and summed to get the center point of the area to be cropped. In the original version, the cropping sliding window is first determined, and then the included area of ​​each position on the cropping sliding window based on the heat map of the significant area is calculated, and finally the positions of multiple maximum areas including the significant area are obtained. The calculation amount of this process is particularly large, resulting in a very long time. In the embodiment of the present application, a method of first determining multiple maximum significant areas can be adopted, and the significant heat map is vertically projected and summed to obtain the significant statistical value of each horizontal position to form a heat curve at the horizontal position, and then the center point is determined by finding multiple maximum values. When determining the center point, the embodiment of the present application can perform weighted summation through the area of ​​the significant area, and perform post-processing in combination with the cropping length. The cropping frame is determined based on the center point of the significant area, thereby avoiding the sliding window in the cropping frame to calculate the area of ​​each position, which greatly reduces the calculation amount of post-processing. In addition, cropping with the center of the significant area can make the main body of the significant area located in the center of the cropping frame, so that the subjective experience is better.

[0071] Optionally, after determining the cropping area of ​​each of the N image frames, the method further includes performing video scene segmentation detection on the M image frames to obtain scene switching frames of the M image frames; and based on the scene switching frames, interpolating the N image frames to obtain M restored image frames.

[0072] Specifically, before cropping, the N image frames obtained by sparse sampling need to be restored to M restored image frames. This process is restored by using an interpolation function in the embodiment of the present application. In order to ensure a more accurate restoration, the embodiment of the present application first detects the scene switching frame of the first video before interpolation processing, and then uses the scene switching frame as a turning point. The image frame before the scene switching frame is restored using the image frame before the scene switching frame obtained by sparse sampling as a template, and the image frame after the scene switching frame is restored using the image frame after the scene switching frame obtained by sparse sampling as a template. Such processing can effectively reduce the gap between the restored image frame and the original image frame.

[0073] For details, see Fig.12a , Fig.12aIt is a continuous frame image of a video provided by an embodiment of the present application. Among them, the 6th frame and the 8th frame are scene switching frames. When the 6th frame is identified as a scene switching frame by the video cropping method in the present application, assuming that the 1st frame, the 3rd frame and the 10th frame are obtained by sparse sampling, the embodiment of the present application can restore the image frames between the 1st frame and the 6th frame according to the cropping area of ​​the 3rd frame when performing interpolation processing, and restore the image frames between the 8th frame and the 12th frame according to the cropping area of ​​the 10th frame. In this way, the central cropping area finally restored by the embodiment of the present application is more accurate.

[0074] For example, Figure 12b , Figure 12b It is another continuous frame image of a video provided by an embodiment of the present application. Among them, the 4th frame and the 9th frame are scene switching frames. When the 4th frame and the 9th frame are identified as scene switching frames by the video cropping method in the present application, assuming that the 1st frame, the 3rd frame and the 10th frame are obtained by sparse sampling, the embodiment of the present application can restore the image frames between the 1st frame and the 4th frame according to the cropping area of ​​the 3rd frame during interpolation processing, and restore the image frames between the 4th frame to the 9th frame according to the cropping area of ​​the 3rd frame. Frames 9 to 17 are restored according to the cropping area of ​​Frame 10. In this way, the central cropping area finally restored by the embodiment of the present application is more accurate.

[0075] For example, Fig.12c , Fig.12c It is another continuous frame image of a video provided by an embodiment of the present application. Among them, the 10th frame and the 12th frame are scene switching frames. When the 10th frame and the 12th frame are identified as scene switching frames by the video cropping method in the present application, assuming that the 1st frame, the 11th frame and the 15th frame are obtained by sparse sampling, the embodiment of the present application can restore the image frames between the 1st frame and the 10th frame according to the cropping area of ​​the 1st frame during interpolation processing, and restore the image frames between the 10th frame and the 12th frame according to the cropping area of ​​the 11th frame. Frames 12 to 24 are restored according to the cropping area of ​​Frame 15. In this way, the central cropping area finally restored by the embodiment of the present application is more accurate.

[0076] Specifically, Figure 3 The video re-encoding module shown in is designed to adapt to the different encoding requirements of the business party and merge the video clips and audio clips into a complete video. The embodiment of the present application uses a multimedia video processing tool (Fast Forward Mpeg, FFmpeg) to re-encode the video and audio. The business party can specify commonly used audio and video encoding parameters such as bit rate, frame rate, encoder type and file format.

[0077] Specifically, FFmpeg is a set of open source computer programs that can be used to record, convert digital audio and video, and convert them into streams. It provides a complete solution for recording, converting and streaming audio and video. It contains very advanced audio / video codecs. In order to ensure high portability and codec quality, many codes in the codec are developed from scratch.

[0078] FFmpeg was developed on the Linux platform, but it can also be compiled and run on other operating systems. The project was first initiated by Fabrice Bellard and maintained by Michael Niedermayer from 2004 to 2015. Many FFmpeg developers came from the MPlayer project, and FFmpeg is currently placed on the MPlayer project team's server. The name of the project comes from the video coding standard, and the "FF" in front stands for "Fast Forward".

[0079] The multimedia video processing tool FFmpeg has very powerful functions including video acquisition, video format conversion, video capture, video watermarking, etc. Video acquisition function: ffmpeg video acquisition function is very powerful. It can not only capture images from video capture cards, but also perform screen recording and support live broadcast applications.

[0080] Video format conversion function: ffmpeg video conversion function. Video format conversion, for example, can convert multiple video formats into other formats, but it is not a video signal conversion. ffmpeg can also easily realize the conversion between multiple video formats, for example, it can convert the recorded video into the format used by the video website. Video screenshot function For the selected video, capture the thumbnail of the specified time.

[0081] The above is the main process of the embodiment of the present application. The beneficial effects produced by the embodiment of the present application are described in detail below. The performance comparison before and after the optimization process is shown in Table 1 below. Table 1 is a performance comparison table before and after the optimization process. The first column represents the resolution of the video, the second column is the frame rate of the video, the third column is the duration of the video, the fourth column is the speed of cropping the video in the existing solution, that is, the solution that has not been optimized, the fifth column is the speed of the cropped video after optimization in the embodiment of the present application, and the sixth column is the multiple of the improvement after optimization compared to before optimization. Among them, the processing speed = processing time / video duration. The smaller the value, the faster the processing speed and the better the performance:

[0082] Table 1

[0083]

[0084] Among them, in order to verify the impact of video intelligent cropping acceleration on subjective effects, the embodiment of the present application collected 50 videos for subjective testing and evaluated them one by one by 30 subjective testers. The relevant evaluation results are shown in Table 2 below. Table 2 is the subjective feeling evaluation results. The first column represents the version, that is, before and after optimization, the second column is the mean, and the third column is the variance.

[0085] Table 2

[0086] Version Mean variance Before optimization 6.4529 1.1094 After optimization 6.5473 1.0618 Difference 0.0618 0.0476

[0087] Specifically, Fig.10 As shown, Fig.10 This is a comparison chart of the score distribution before and after the video cropping optimization processing provided by an embodiment of the present application. The horizontal axis represents the mean, and the vertical axis represents the probability. The highest points are 6.4529 and 6.5473 respectively. Assuming that the subjective evaluation scores conform to the normal distribution, the distribution area overlap reaches 93.72%; in summary, the embodiment of the present application has increased the speed by 8-10 times while the subjective effect difference is less than 1%.

[0088] The embodiments of the present application optimize the existing video intelligent cropping method and mainly solve the following problems:

[0089] 1. Optimize the dense frame sampling method of the existing intelligent cropping capability, and propose a sparse sampling combined with interpolation method to predict the cropping position of all frames;

[0090] 2. In terms of determining the salient area, the existing method of determining the salient area based on sliding window area calculation adopts vertical projection combined with multiple peak-finding algorithms to determine the central cropping area, which greatly accelerates the current post-processing part.

[0091] In summary, the embodiments of the present application accelerate and optimize the current video cropping method through a time domain detection algorithm based on sparse sampling combined with interpolation and a post-processing algorithm based on a peak finding algorithm combined with weighted determination of the center point, thereby greatly expanding the application prospects of this technology. Spatial video cropping is performed based on dual-stream video saliency area detection, and the subjective effect of intelligent video cropping is further improved by black and white edge detection and OCR-based subtitle watermark detection. The video re-encoding is adapted to the business party's video encoding requirements, which has huge practical application value.

[0092] For further information, see Fig.11 , Fig.11 : is a flow chart of a video cropping method provided by an embodiment of the present application. Fig.11 As shown, the method may at least include the following steps S301-S304:

[0093] Step S301, obtaining N image frames in a first video, and performing salient region detection on the N image frames respectively to obtain a salient region heat map of each of the N image frames;

[0094] Specifically, the process of step S301 can be specifically referred to above. Figure 5 The relevant process of salient region detection described in , and this process can correspond to Figure 3 The functions performed by the salient region monitoring module in the salient region detection module. For example, in the salient region detection module, the salient region of the video is detected based on the spatiotemporal dual-stream network, and the architecture of the salient region detection module includes a temporal stream network and a spatial stream network, and the temporal stream network and the spatial stream network share parameters in terms of extracting visual features; in the modeling of the spatial stream network, SF-Net is a backbone network based on the VGG-16 model, and a multi-level feature fusion method is designed to extract basic features related to saliency, and then the embodiment of the present application designs SP-Net for integrating spatial features and Gaussian prior features of different scales; in the modeling of the temporal stream network, the embodiment of the present application performs spatial modeling based on Te-Net combined with 3D convolution and 2D convolution, firstly based on SF-Net, feature extraction is performed from input frames of multiple time dimensions under the condition of shared parameters to form salient features of the time domain dimension, and then the shallow 3D convolution is used to extract features to obtain a time domain salient feature sequence, thereby realizing the spatiotemporal dual-stream model for salient region detection.

[0095] It can be understood that N here is a positive integer; the salient region heat map of each image frame is used to represent the salient degree of at least one salient region in the corresponding image frame.

[0096] For further understanding, please refer to Figure 7-Figure 9 ,in Figure 7 It is an original image before cropping provided in an embodiment of the present application. Figure 8 is a heat map of the salient regions of the original image before cropping provided in an embodiment of the present application, Fig. 9 It is a longitudinal projection diagram of a heat map of a salient area of ​​an original image before cropping provided in an embodiment of the present application. Figure 7 There is a whale in the sea in the center of the picture, with a string of letters in the lower right corner. Figure 5 The detection of the salient area of ​​the picture is obtained Figure 8 The heat map of the significant regions, from Figure 8 It can be clearly seen that there are two obvious white areas, which can be seen as the main body of the whale and the letters in the lower right corner. That is to say, the salient area obtained by salient area detection has two parts, namely the whale and the letters. Then the heat map of this salient area is Figure 8Perform longitudinal projection to obtain Fig. 9 . Fig. 9 It is a longitudinal projection of a heat map of a salient region of an original image before cropping provided in an embodiment of the present application. Corresponding to the two salient regions of the heat map above, Fig. 9 There are also two peaks. The first peak with a larger area corresponds to the main body of the whale, and the smaller peak on the right is the letter in the lower right corner. Finally, based on this longitudinal projection, the areas of the two peaks are weighted and summed to get the center point of the area to be cropped. In the original version, the cropping sliding window is first determined, and then the included area of ​​each position on the cropping sliding window based on the heat map of the significant area is calculated, and finally the positions of multiple maximum areas including the significant area are obtained. The calculation amount of this process is particularly large, resulting in a very long time. In the embodiment of the present application, a method of first determining multiple maximum significant areas is adopted, and the significant heat map is vertically projected and summed to obtain the significant statistical value of each horizontal position to form a heat curve at the horizontal position, and then the center point is determined by finding multiple maximum values. When determining the center point, the embodiment of the present application performs weighted summation through the area of ​​the significant area, and performs post-processing in combination with the cropping length. The cropping frame is determined based on the center point of the significant area, which avoids the sliding of the sliding window in the cropping frame to calculate the area of ​​each position, which greatly reduces the calculation amount of post-processing. In addition, cropping with the center of the significant area can make the main body of the significant area located in the center of the cropping frame, so that the subjective experience is better.

[0097] Optionally, before performing salient area detection, the method further includes acquiring a first video, wherein the first video includes M image frames, where M is a positive integer; and sparsely sampling the M image frames to obtain the N image frames, where N is a positive integer less than M.

[0098] Specifically, for video intelligent cropping technology, the difficulty of ROI-based cropping is that the ROI area changes with the change of frames. In order to detect the ROI area more accurately, the current dense detection method is used, that is, each frame performs significant area detection, but this dense detection brings huge performance loss. The embodiment of the present application uses the visual persistence effect of the human eye combined with the fact that the ROI area in the same scene will not undergo sudden changes in position. A sparse sampling frame extraction strategy is designed, and then interpolation and video scene switching detection are combined to restore the ROI position of the remaining frames, achieving the same time domain detection results as dense frame-by-frame detection. Some prediction results are shown below. Figure 4 As shown, Figure 4 It is a cropping position prediction comparison diagram provided in an embodiment of the present application, in which the dotted line is the original densely sampled predicted position, the black dots are sparsely sampled detection points, and the solid line is the predicted position interpolated through the sparse detection points.

[0099] Optionally, before performing the salient area detection, the method further includes performing non-significant area detection on the N image frames to obtain non-significant area features of each image frame, wherein the non-significant area features include one or more of upper and lower or left and right black and white borders, subtitles, and watermarks in the image frame; and removing the non-significant area features in each image frame to obtain a pure image frame of each image frame.

[0100] Specifically, before performing the salient area detection, the method further includes performing non-significant area detection on the N image frames to obtain non-significant area features of each of the image frames, wherein the non-significant area features include one or more of upper and lower or left and right black and white borders, subtitles, and watermarks in the image frames; and removing the non-significant area features in each of the image frames to obtain a pure image frame of each of the image frames.

[0101] Specifically, Figure 6a , Figure 6a This is a schematic diagram before watermark removal provided by the embodiment of the present application. It can be clearly seen that there are watermarks and letters in the picture. After the embodiment of the present application, the watermarks and letters of the picture can be removed. Figure 6b The picture, Figure 6b is a schematic diagram of a watermark removed according to an embodiment of the present application. At the same time, the picture may contain some black and white borders, such as Figure 6c , Figure 6c This is a schematic diagram of a black and white border provided in an embodiment of the present application. Because in the process of spatial detection of the video, there are some factors that seriously affect the subjective effect of the video. Some videos have black and white borders on the top and bottom or left and right, and some videos contain subtitles or even watermarks. For these parts that users do not want to be in the cropped video screen, the embodiment of the present application detects the text area in the screen through OCR-based character recognition technology and performs keyword matching and position statistics of the watermark in the text area. Finally, these factors can be removed to achieve a better subjective effect.

[0102] Optionally, performing salient area detection on the N image frames respectively to obtain a salient area heat map of each of the N image frames; includes: performing salient area detection on the pure image frame of each of the image frames respectively to obtain a salient area heat map of each of the N image frames.

[0103] Specifically, in the process of spatial detection of videos, there are some factors that seriously affect the subjective effect of videos. Some videos have black and white borders on the top and bottom or left and right, and some videos contain subtitles or even watermarks. For these parts that users do not want to be in the cropped video screen, the embodiment of the present application detects the text area in the screen through OCR-based character recognition technology and performs keyword matching and position statistics of the watermark in the text area. Finally, these factors can be removed to achieve a better subjective effect. Then, the video with the non-significant area features removed is subjected to significant area detection, so that the final cropped video will be more in line with the perception of the human eye.

[0104] Step S302, summing the pixel values ​​in each column of image data in the heat map of the salient region of each image frame to obtain a longitudinal projection map of each image frame;

[0105] Specifically, the process of step S302 can be specifically referred to above. Figure 7 The process for determining the preset cropping area described in Figure 3 The function performed by the post-processing module based on the saliency heat map in the vertical projection image. The horizontal coordinate of the vertical projection image is the horizontal position of each column of image data in the corresponding image frame, and the vertical coordinate is the sum of the pixel values ​​in each column of image data.

[0106] For further understanding, please refer to Figure 7-Figure 9 ,in Figure 7 It is an original image before cropping provided in an embodiment of the present application. Figure 8 is a heat map of the salient regions of the original image before cropping provided in an embodiment of the present application, Fig. 9 It is a longitudinal projection diagram of a heat map of a salient area of ​​an original image before cropping provided in an embodiment of the present application. Figure 7 There is a whale in the sea in the center of the picture, with a string of letters in the lower right corner. Figure 5 The detection of the salient area of ​​the picture is obtained Figure 8 The heat map of the significant regions, from Figure 8 It can be clearly seen that there are two obvious white areas, which can be seen as the main body of the whale and the letters in the lower right corner. That is to say, the salient area obtained by salient area detection has two parts, namely the whale and the letters. Figure 8 The black area represents a pixel value of 0, and the white area represents a pixel value. In the embodiment of the present application, the pixel values ​​of each column are summed to obtain a longitudinal projection map based on the heat map of the salient region. Fig. 9 It is a longitudinal projection of a heat map of a salient region of an original image before cropping provided in an embodiment of the present application. Corresponding to the two salient regions of the heat map above, Fig. 9There are two peaks. The first peak with a larger area corresponds to the main body of the whale, and the smaller peak on the right is the letter in the lower right corner. Fig. 9 The horizontal axis represents the horizontal position of each column of image data in the corresponding image frame, and the vertical axis represents the normalized sum of the pixel values ​​in each column of image data in the salient region heat map. Finally, the weighted sum of the areas of the two peaks is performed according to this longitudinal projection map to obtain the center point of the area to be cropped. In the original version, the cropping sliding window is first determined, and then the included area of ​​each position on the cropping sliding window based on the heat map of the significant area is calculated, and finally the positions of multiple maximum areas including the significant area are obtained. The calculation amount of this process is particularly large, resulting in a very long time. In the embodiment of the present application, a method of first determining multiple maximum significant areas is adopted, and the significant heat map is vertically projected and summed to obtain the significant statistical value of each horizontal position to form a heat curve at the horizontal position, and then the center point is determined by finding multiple maximum values. When determining the center point, the embodiment of the present application performs weighted summation through the area of ​​the significant area, and performs post-processing in combination with the cropping length. The cropping frame is determined based on the center point of the significant area, which avoids the sliding of the sliding window in the cropping frame to calculate the area of ​​each position, which greatly reduces the calculation amount of post-processing. In addition, cropping with the center of the significant area can make the main body of the significant area located in the center of the cropping frame, so that the subjective experience is better.

[0107] Step S303, determining the center point of the to-be-cropped region of each of the image frames according to the weighted sum of the areas of one or more peaks of the longitudinal projection graph of each of the image frames;

[0108] Specifically, the process of step S303 can be specifically referred to above. Figure 8 The process for determining the preset cropping area described in Figure 3 The functions performed by the post-processing module based on the saliency heat map in the application. For example, in the application, a method of first determining multiple maximum saliency regions is adopted, and the saliency heat map is vertically projected and summed to obtain the saliency statistics of each horizontal position to form a heat curve at the horizontal position, and then the center point is determined by finding multiple maximum values. When determining the center point, the embodiment of the application performs weighted summation by the area of ​​the saliency region.

[0109] For details, see Figure 7-Figure 9 ,in Figure 7 It is an original image before cropping provided in an embodiment of the present application. Figure 8 is a heat map of the salient regions of the original image before cropping provided in an embodiment of the present application, Fig. 9It is a longitudinal projection diagram of a heat map of a salient area of ​​an original image before cropping provided in an embodiment of the present application. Figure 7 There is a whale in the sea in the center of the picture, with a string of letters in the lower right corner. Figure 5 The detection of the salient area of ​​the picture is obtained Figure 8 The heat map of the significant regions, from Figure 8 It can be clearly seen that there are two obvious white areas, which can be seen as the main body of the whale and the letters in the lower right corner. That is to say, the salient area obtained by salient area detection has two parts, namely the whale and the letters. Figure 8 The black area represents a pixel value of 0, and the white area represents a pixel value. In the embodiment of the present application, the pixel values ​​of each column are summed to obtain a longitudinal projection map based on the heat map of the salient region. Fig. 9 It is a longitudinal projection of a heat map of a salient region of an original image before cropping provided in an embodiment of the present application. Corresponding to the two salient regions of the heat map above, Fig. 9 There are two peaks. The first peak with a larger area corresponds to the main body of the whale, and the smaller peak on the right is the letter in the lower right corner. Fig. 9 The horizontal axis represents the horizontal position of each column of image data in the corresponding image frame, and the vertical axis represents the normalized sum of the pixel values ​​in each column of image data in the salient region heat map. Finally, the weighted sum of the areas of the two peaks is performed according to this longitudinal projection map to obtain the center point of the area to be cropped. In the original version, the cropping sliding window is first determined, and then the included area of ​​each position on the cropping sliding window based on the heat map of the significant area is calculated, and finally the positions of multiple maximum areas including the significant area are obtained. The calculation amount of this process is particularly large, resulting in a very long time. In the embodiment of the present application, a method of first determining multiple maximum significant areas is adopted, and the significant heat map is vertically projected and summed to obtain the significant statistical value of each horizontal position to form a heat curve at the horizontal position, and then the center point is determined by finding multiple maximum values. When determining the center point, the embodiment of the present application performs weighted summation through the area of ​​the significant area, and performs post-processing in combination with the cropping length. The cropping frame is determined based on the center point of the significant area, which avoids the sliding of the sliding window in the cropping frame to calculate the area of ​​each position, which greatly reduces the calculation amount of post-processing. In addition, cropping with the center of the significant area can make the main body of the significant area located in the center of the cropping frame, so that the subjective experience is better.

[0110] Step S304, determining a cropping area of ​​each of the N image frames according to a center point of the to-be-cropped area of ​​each of the image frames and a preset cropping size;

[0111] Specifically, the process of step S304 can be specifically referred to above. Fig. 9 The process for determining the preset cropping area described in Figure 3 Functions performed by the post-processing module based on the saliency heat map in. For example, in the embodiment of the present application, a method of first determining multiple maximum saliency regions is adopted, and the saliency heat map is vertically projected and summed to obtain the saliency statistical value of each horizontal position to form a thermal curve at the horizontal position, and then the center point is determined by finding multiple maximum values. When determining the center point, the embodiment of the present application performs weighted summation by the area of ​​the saliency region, and performs post-processing in combination with the cropping length. The cropping frame is determined based on the center point of the saliency region, which avoids the sliding window sliding in the cropping frame to calculate the area of ​​each position, which greatly reduces the amount of calculation for post-processing. In addition, cropping with the center of the saliency region can make the main body of the saliency region located in the center of the cropping frame, so that the subjective experience is better.

[0112] Optionally, after determining the cropping area of ​​each of the N image frames, the method further includes performing video scene segmentation detection on the M image frames to obtain scene switching frames of the M image frames; and based on the scene switching frames, interpolating the N image frames to obtain M restored image frames.

[0113] Specifically, before cropping, the N image frames obtained by sparse sampling need to be restored to M restored image frames. This process is restored by using an interpolation function in the embodiment of the present application. In order to ensure a more accurate restoration, the embodiment of the present application first detects the scene switching frame of the first video before interpolation processing, and then uses the scene switching frame as a turning point. The image frame before the scene switching frame is restored using the image frame before the scene switching frame obtained by sparse sampling as a template, and the image frame after the scene switching frame is restored using the image frame after the scene switching frame obtained by sparse sampling as a template. Such processing can effectively reduce the gap between the restored image frame and the original image frame.

[0114] For details, see Fig.12a , Fig.12a It is a continuous frame image of a video provided by an embodiment of the present application. Among them, the 6th frame and the 8th frame are scene switching frames. When the video cropping method in the present application recognizes that the 6th frame is a scene switching frame, assuming that the 1st frame, the 3rd frame and the 10th frame are obtained by sparse sampling, the embodiment of the present application can restore the image frames between the 1st frame and the 6th frame according to the cropping area of ​​the 3rd frame when performing interpolation processing, and restore the image frames between the 8th frame and the 12th frame according to the cropping area of ​​the 10th frame. In this way, the central cropping area finally restored by the embodiment of the present application is more accurate.

[0115] For example, Figure 12b , Figure 12b It is another continuous frame image of a video provided by an embodiment of the present application. Among them, the 4th frame and the 9th frame are scene switching frames. When the video cropping method in the present application recognizes that the 4th frame and the 9th frame are scene switching frames, assuming that the 1st frame, the 3rd frame and the 10th frame are obtained by sparse sampling, the embodiment of the present application can restore the image frames between the 1st frame and the 4th frame according to the cropping area of ​​the 3rd frame during interpolation processing, and restore the image frames between the 4th frame to the 9th frame according to the cropping area of ​​the 3rd frame. Frames 9 to 17 are restored according to the cropping area of ​​Frame 10. In this way, the central cropping area finally restored by the embodiment of the present application is more accurate.

[0116] For example, Fig.12c , Fig.12c It is another continuous frame image of a video provided by an embodiment of the present application. Among them, the 10th frame and the 12th frame are scene switching frames. When the video cropping method in the present application recognizes that the 10th frame and the 12th frame are scene switching frames, assuming that the 1st frame, the 11th frame and the 15th frame are obtained by sparse sampling, the embodiment of the present application can restore the image frames between the 1st frame and the 10th frame according to the cropping area of ​​the 1st frame during interpolation processing, and restore the image frames between the 10th frame and the 12th frame according to the cropping area of ​​the 11th frame. Frames 12 to 24 are restored according to the cropping area of ​​Frame 15. In this way, the central cropping area finally restored by the embodiment of the present application is more accurate.

[0117] Optionally, after determining the cropping area of ​​each of the N image frames, the method further includes: cropping the first video to obtain a second video according to the central cropping area of ​​the N image frames and the M restored image frames; and re-encoding the video segment of the second video and the first audio segment to obtain a cropped complete video corresponding to the first video. The audio of the first video is the first audio segment.

[0118] Specifically, in order to adapt to the different encoding requirements of the business party and merge the video clips and audio clips into a complete video, the embodiment of the present application can use FFmpeg to re-encode the video and audio, and the business party can specify commonly used audio and video encoding parameters such as bit rate, frame rate, encoder type and file format.

[0119] Specifically, FFmpeg is an open source computer program that can be used to record, convert digital audio and video, and convert them into streams. It provides a complete solution for recording, converting, and streaming audio and video. For example, FFmpeg can perform video capture, video format conversion, video capture, and video watermarking.

[0120] The embodiments of the present application optimize the existing video intelligent cropping method and mainly solve the following problems:

[0121] 1. Optimize the dense frame sampling method of the existing intelligent cropping capability, and propose a sparse sampling combined with interpolation method to predict the cropping position of all frames;

[0122] 2. In terms of determining the salient area, the existing method of determining the salient area based on sliding window area calculation adopts vertical projection combined with multiple peak-finding algorithms to determine the central cropping area, which greatly accelerates the current post-processing part.

[0123] In summary, the embodiments of the present application accelerate and optimize the current video cropping method through a time domain detection algorithm based on sparse sampling combined with interpolation and a post-processing algorithm based on a peak finding algorithm combined with weighted determination of the center point, thereby greatly expanding the application prospects of this technology. Spatial video cropping is performed based on dual-stream video saliency area detection, and the subjective effect of intelligent video cropping is further improved by black and white edge detection and OCR-based subtitle watermark detection. The video re-encoding is adapted to the business party's video encoding requirements, which has huge practical application value.

[0124] In summary, the above scheme in the present application only needs to calculate the heat map of the salient area of ​​the image frame and its longitudinal projection map to obtain the central cropping area, so the amount of calculation for subsequent processing can be reduced; it avoids the problem of large amount of calculation caused by the sliding window in the sliding of the cropping frame to obtain the cropping area in the prior art; in addition, since the embodiment of the present application adopts the method of first determining the center point and then confirming the central cropping area, it can be cropped with the center point of the salient area, so that the main body of the salient area is located in the center of the cropping frame to optimize the final display effect, avoiding the inaccurate display effect caused by the sliding of the sliding window in the cropping frame in the prior art. Optionally, the N image frames can be obtained by sparse sampling of the M image frames in the video to be cropped, and the use of sparse sampling here can further reduce the time spent on subsequent post-processing. It can be seen that by implementing the embodiment of the present application, the cropping area of ​​each image frame in the N image frames in the video to be cropped will be more in line with the visual range of the human eye, and the amount of calculation required for video cropping can be greatly reduced. That is, the embodiment of the present application provides a solution that can reduce the amount of calculation required for cropping while optimizing the display effect of the target object in the interface after video cropping.

[0125] For further information, see Fig.13 , Fig.131 is a schematic diagram of the structure of a video cropping device provided in an embodiment of the present application. The video cropping device 1 may include: a detection module 10, a longitudinal projection module 20, a center point determination module 30, and a cropping determination module 40.

[0126] The detection module 10 is used to obtain N image frames in the first video, and perform salient region detection on the N image frames respectively to obtain a salient region heat map of each of the N image frames, where N is a positive integer; wherein the salient region heat map of each of the image frames is used to indicate the salient degree of at least one salient region in the corresponding image frame;

[0127] A longitudinal projection module 20 is used to sum the pixel values ​​in each column of image data in the heat map of the salient region of each image frame to obtain a longitudinal projection map of each image frame; the abscissa of the longitudinal projection map is the lateral position of each column of image data in the corresponding image frame, and the ordinate is the sum of the pixel values ​​in each column of image data;

[0128] A center point determination module 30, configured to determine the center point of the to-be-cropped area of ​​each of the image frames according to a weighted sum of the areas of one or more peaks of the longitudinal projection image of each of the image frames;

[0129] The cropping region determining module 40 is used to determine the cropping region of each of the N image frames according to the center point of the to-be-cropped region of each of the image frames and a preset cropping size.

[0130] Specifically, the device further includes:

[0131] The acquisition module is used to acquire a first video, wherein the first video includes M image frames, where M is a positive integer; the sampling module is used to perform sparse sampling on the M image frames to obtain the N image frames, where N is a positive integer less than M. The embodiment of the present application mainly reduces the amount of calculation by replacing dense sampling with sparse sampling.

[0132] Wherein, the device further comprises:

[0133] The non-significant area detection module is used to perform non-significant area detection on the N image frames to obtain the non-significant area features of each of the image frames, wherein the non-significant area features include one or more of the upper and lower or left and right black and white borders, subtitles and watermarks in the image frame; the non-significant area features in each of the image frames are removed to obtain a pure image frame of each of the image frames. The embodiment of the present application mainly detects the text area in the picture through the OCR-based character recognition technology and performs keyword matching and position statistics of the watermark in the text area. Finally, these factors can be removed to achieve a better subjective effect.

[0134] The detection module is further used to perform salient area detection on the clean image frame of each of the image frames to obtain a salient area heat map of each of the N image frames. In the embodiment of the present application, when the clean version of the image frame is detected, the obtained cropped area can be made more consistent with the perception of the human eye.

[0135] Wherein, the device further comprises:

[0136] The restoration module is used to perform video scene segmentation detection on the M image frames to obtain scene switching frames of the M image frames; according to the scene switching frames, the N image frames are interpolated to obtain M restored image frames. In the embodiment of the present application, a sparse sampling frame extraction strategy is designed by utilizing the visual persistence effect of the human eye and the fact that the ROI area in the same scene will not undergo positional mutations. The ROI position of the remaining frames is restored by combining interpolation with video scene switching detection, thereby achieving the same time domain detection result as dense frame-by-frame detection.

[0137] Wherein, the device further comprises:

[0138] The integration module is used to crop the first video according to the central cropping area of ​​the N image frames and the M restored image frames to obtain a second video; and re-encode the video segment of the second video and the first audio segment to obtain a cropped complete video corresponding to the first video. In this embodiment of the application, cropping is performed according to the determined central cropping area to finally obtain a cropped video.

[0139] The specific implementation of the detection module 10, the longitudinal projection module 20, the center point determination module 30, and the cropping area determination module 40 can be found in the above Fig. 9 The description of step S301 and step S304 in the corresponding embodiment will not be repeated here. In addition, the description of the beneficial effects of the same method will not be repeated.

[0140] For further information, see Fig.14 , Fig.14Schematic diagram of the structure of a computer device provided in an embodiment of the present application. Fig.14 As shown, the computer device 1000 may be a terminal device, and the terminal device may be the above Figure 1 The terminal device 120 in the corresponding embodiment, optionally, the computer device 1000 may also be a service server, and the service server may be the above-mentioned Figure 1 The server 100 in the corresponding embodiment. At this time. The computer device 1000 may include: a processor 1001, a network interface 1004 and a memory 1005. In addition, the computer device 1000 may also include: a user interface 1003, and at least one communication bus 1002. Among them, the communication bus 1002 is used to realize the connection and communication between these components. Among them, the user interface 1003 may include a display screen (Display), a keyboard (Keyboard), and the optional user interface 1003 may also include a standard wired interface and a wireless interface. The network interface 1004 may optionally include a standard wired interface and a wireless interface (such as a WI-FI interface). The memory 1005 may be a high-speed RAM memory, or it may be a non-volatile memory (non-volatile memory), such as at least one disk storage. The memory 1005 may optionally also be at least one storage device located away from the aforementioned processor 1001. As Fig.11 As shown, the memory 1005 as a computer storage medium may include an operating system, a network communication module, a user interface module, and a device control application program.

[0141] The network interface 1004 in the computer device 1000 can also provide a network communication function, and the optional user interface 1003 can also include a display screen (Display) and a keyboard (Keyboard). Fig.14 In the computer device 1000 shown in the figure, the network interface 1004 can provide a network communication function; the user interface 1003 is mainly used to provide an input interface for the user; and the processor 1001 can be used to call the device control application stored in the memory 1005 to implement the above Fig. 9 , Figure 3 or Figure 4 The description of the video cropping method in the corresponding embodiment can also be performed as described above. Fig.10 The description of the video cropping device 1 in the corresponding embodiment will not be repeated here. In addition, the description of the beneficial effects of the same method will not be repeated here either.

[0142] In addition, it should be pointed out that: the embodiment of the present application also provides a computer storage medium, and the computer storage medium stores a computer program executed by the video data processing device 1 mentioned above, and the computer program includes program instructions. When the processor executes the program instructions, it can execute the above-mentioned Fig. 9 , Figure 3 or Figure 4 The description of the video cropping method in the corresponding embodiment will not be repeated here. In addition, the description of the beneficial effects of the same method will not be repeated. For technical details not disclosed in the computer storage medium embodiment involved in this application, please refer to the description of the method embodiment of this application.

[0143] It is understandable that the embodiment of the present application further provides a computer program product or a computer program, the computer program product or the computer program includes computer instructions, the computer instructions are stored in a computer-readable storage medium. The processor of the computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the above Fig. 9 , Figure 3 or Figure 4 The description of the video cropping method in the corresponding embodiment will not be repeated here. In addition, the description of the beneficial effects of the same method will not be repeated. For technical details not disclosed in the computer storage medium embodiment involved in this application, please refer to the description of the method embodiment of this application.

[0144] For further information, see Fig.15 , Fig.15 1 is a schematic diagram of the structure of a video cropping system provided in an embodiment of the present application. The video cropping system 2 may include a video cropping device 1. The video cropping device 1 may be the above-mentioned Fig.13 The video cropping device 1 in the corresponding embodiment can be understood that the video cropping device 1 can be integrated in the above Figure 1 The terminal device 120 in the corresponding embodiment will not be described in detail here. In addition, the description of the beneficial effects of the same method will not be described in detail. For technical details not disclosed in the video data processing system embodiment involved in this application, please refer to the description of the method embodiment of this application.

[0145] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program, and the program can be stored in a computer-readable storage medium, and when the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the storage medium can be a disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM), etc.

[0146] The above disclosure is only the preferred embodiment of the present application, which certainly cannot be used to limit the scope of rights of the present application. Therefore, equivalent changes made according to the claims of the present application are still within the scope covered by the present application.

Claims

1. A video cropping method, characterized in that: include: Acquire a first video, where the first video includes M image frames, where M is a positive integer; Sparsely sampling the M image frames to obtain N image frames, where N is a positive integer less than M; Acquire the N image frames in the first video, and perform salient region detection on the N image frames respectively to obtain a salient region heat map of each of the N image frames, where N is a positive integer; wherein the salient region heat map of each of the image frames is used to indicate the salient degree of at least one salient region in the corresponding image frame; The pixel values ​​in each column of the image data in the heat map of the salient region of each of the image frames are summed to obtain a longitudinal projection map of each of the image frames; the abscissa of the longitudinal projection map is the lateral position of each column of the image data in the corresponding image frame, and the ordinate is the sum of the pixel values ​​in each column of the image data; Determine the center point of the to-be-cropped region of each of the image frames according to a weighted sum of the areas of one or more peaks of the longitudinal projection image of each of the image frames; Determining a cropping area of ​​each of the N image frames according to a center point of the to-be-cropped area of ​​each of the image frames and a preset cropping size; Performing video scene segmentation detection on the M image frames to obtain scene switching frames of the M image frames; According to the scene switching frame, the N image frames are interpolated to obtain M restored image frames, including: the image frame before the scene switching frame is restored using the image frame before the scene switching frame obtained by the sparse sampling as a template, and the image frame after the scene switching frame is restored using the image frame after the scene switching frame obtained by the sparse sampling as a template.

2. The method according to claim 1, characterized in that The method further comprises: Performing non-significant area detection on the N image frames to obtain non-significant area features of each of the image frames, wherein the non-significant area features include one or more of upper and lower or left and right black and white borders, subtitles, and watermarks in the image frames; The non-significant region features in each of the image frames are removed to obtain a pure image frame of each of the image frames.

3. The method according to claim 2, characterized in that The performing salient region detection on the N image frames respectively to obtain a salient region heat map of each of the N image frames comprises: The salient region detection is performed on the clean image frame of each of the image frames to obtain a salient region heat map of each of the N image frames.

4. The method according to any one of claims 1 to 3, characterized in that: The audio of the first video is a first audio segment; the method further includes: Cropping the first video according to the central cropping area of ​​the N image frames and the M restored image frames to obtain a second video; The video segment of the second video and the first audio segment are re-encoded to obtain a cropped complete video corresponding to the first video.

5. A video cutting device, characterized in that: include: An acquisition module, configured to acquire a first video, wherein the first video includes M image frames, where M is a positive integer; A sampling module, used for performing sparse sampling on the M image frames to obtain N image frames, where N is a positive integer less than M; a detection module, configured to obtain the N image frames in the first video, and perform salient region detection on the N image frames respectively, to obtain a salient region heat map of each of the N image frames, where N is a positive integer; wherein the salient region heat map of each of the image frames is used to indicate a salient degree of at least one salient region in the corresponding image frame; A longitudinal projection module, used for summing the pixel values ​​in each column of image data in the heat map of the salient region of each image frame to obtain a longitudinal projection map of each image frame; the abscissa of the longitudinal projection map is the lateral position of each column of image data in the corresponding image frame, and the ordinate is the sum of the pixel values ​​in each column of image data; A center point determination module, used to determine the center point of the to-be-cropped area of ​​each of the image frames according to a weighted sum of the areas of one or more peaks of the longitudinal projection image of each of the image frames; A cropping region determining module, configured to determine a cropping region of each of the N image frames according to a center point of the to-be-cropped region of each of the image frames and a preset cropping size; The restoration module is used to perform video scene segmentation detection on the M image frames to obtain scene switching frames of the M image frames; according to the scene switching frames, the N image frames are interpolated to obtain M restored image frames, including: the image frames before the scene switching frames are restored using the image frames before the scene switching frames obtained by the sparse sampling as templates, and the image frames after the scene switching frames are restored using the image frames after the scene switching frames obtained by the sparse sampling as templates.

6. A computer device, characterized in that: include: Processor and memory; The processor is connected to a memory, wherein the memory is used to store a computer program, and the processor is used to call the computer program so that the computer device executes the method according to any one of claims 1 to 4.

7. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and the computer program is suitable for being loaded and executed by a processor, so that a computer device having the processor executes the method according to any one of claims 1 to 4.

8. A computer program product, characterized in that The computer program product comprises computer instructions, which are stored in a computer-readable storage medium. The computer instructions are suitable for being read and executed by a processor, so that a computer device having the processor executes the method according to any one of claims 1 to 4.

Images