Video Processing Method, Device, Electronic Device and Storage Medium

Abstract

The present disclosure provides a video processing method, apparatus, electronic device, and storage medium, belonging to the field of video technology. In the present disclosure, a first cropping parameter sequence of a target image in a video to be cropped is obtained, as well as a second cropping parameter sequence of the previous frame image of the target image. If a first difference between the first cropping parameter sequence and the second cropping parameter sequence exceeds a preset difference range, at least one frame image in the video to be cropped and located after the target image is determined as a transition image of the target image. According to the first difference, parameter smoothing is performed on the first cropping parameter sequence to determine a third cropping parameter sequence; wherein, a second difference between the parameters in each third cropping parameter sequence and the parameters in the second cropping parameter sequence is different, and the largest second difference is less than or equal to the first difference. Based on the third cropping parameter sequence, object cropping is performed on the target image and the transition image. In this way, the problem of object cropping jitter can be avoided to a certain extent, ensuring the cropping effect.

Description

Technical Field

[0001] The present disclosure belongs to the field of video technology, and particularly relates to a video processing method, apparatus, electronic device, and storage medium. Background Art

[0002] Currently, in the field of video processing, it is often necessary to crop the objects included in the images in the video. For example, it is necessary to crop the faces included in each frame image included in the video.

[0003] In related technologies, usually, a sequence of cropping parameters for each frame image is first calculated, and then each frame image is directly cropped based on the calculated sequence of cropping parameters. In this way, affected by factors such as environmental illumination, image quality, and the robustness of the calculation method of the sequence of cropping parameters itself, the object cropping result of the video will be jittery and the cropping effect will be poor. Summary of the Invention

[0004] To overcome the problems existing in related technologies, the present disclosure provides a video processing method, apparatus, electronic device, and storage medium.

[0005] According to a first aspect of the present disclosure, a video processing method is provided. The method includes:

[0006] Obtaining a first sequence of cropping parameters for a target image in a video to be cropped, and a second sequence of cropping parameters for the previous frame image of the target image;

[0007] If a first difference between the first sequence of cropping parameters and the second sequence of cropping parameters exceeds a preset difference range, determining at least one frame image after the target image in the video to be cropped as a transition image of the target image;

[0008] Performing parameter smoothing on the first sequence of cropping parameters according to the first difference to determine a third sequence of cropping parameters; wherein, a second difference between the parameters in each third sequence of cropping parameters and the parameters in the second sequence of cropping parameters is different, and the largest second difference is less than or equal to the first difference;

[0009] Performing object cropping on the target image and the transition image based on the third sequence of cropping parameters.

[0010] Optionally, the method further includes:

[0011] Calculating a third difference between each parameter in the first sequence of cropping parameters and each parameter in the second sequence of cropping parameters;

[0012] If the absolute value of each third difference of each parameter is greater than a preset absolute value threshold corresponding to each parameter, determining that the first difference exceeds the preset difference range.

[0013] Optionally, the object clipping of the target image and the transition image based on the third clipping parameter sequence includes:

[0014] Performing object clipping on the target image based on the third clipping parameter sequence corresponding to the target image, and performing object clipping on the transition image based on the third clipping parameter sequence corresponding to the transition image.

[0015] Optionally, before performing object clipping on the transition image based on the third clipping parameter sequence corresponding to the transition image, the method further includes:

[0016] Detecting whether the difference between the third clipping parameter sequence corresponding to the transition image and the third clipping parameter sequence of the previous frame image of the transition image exceeds the preset difference range;

[0017] If not, performing the operation of object clipping on the transition image based on the third clipping parameter sequence corresponding to the transition image;

[0018] If it exceeds, clearing the third clipping parameter sequence of the transition image that has not been clipped yet, and determining the transition image as the new target image.

[0019] Optionally, the method further includes:

[0020] If the first difference does not exceed the preset difference range, obtaining the second clipping parameter sequence of the previous frame image of the target image to set as the third clipping parameter sequence of the target image; the second clipping parameter sequence is the clipping parameter sequence used when performing object clipping on the previous frame image.

[0021] Optionally, if the first difference does not exceed the preset difference range, obtaining the second clipping parameter sequence of the previous frame image of the target image to set as the third clipping parameter sequence of the target image includes:

[0022] If the first difference does not exceed the preset difference range and the target image does not belong to the transition image of the previous target image, obtaining the second clipping parameter sequence of the previous frame image of the target image to set as the third clipping parameter sequence of the target image.

[0023] Optionally, obtaining the second clipping parameter sequence of the previous frame image of the target image to set as the third clipping parameter sequence of the target image includes:

[0024] Reading the clipping parameter sequence stored in the current tail element of the preset parameter queue; the preset parameter queue stores the clipping parameter sequences used when clipping each frame image in the video to be clipped;

[0025] Insert the read sequence of cropping parameters as a new tail element into the preset parameter queue to set the third cropping parameter sequence of the target image.

[0026] Optionally, the method further includes:

[0027] If the mode indicated by the current value of the preset mode identifier is the first mode, determine that the target image is not a transitional image of the previous target image; if the mode indicated by the current value of the preset mode identifier is the second mode, determine that the target image is a transitional image of the previous target image.

[0028] After performing parameter smoothing operation according to the first difference and the first cropping parameter sequence, the method further includes: setting the current value to an identifier value for indicating the second mode.

[0029] After obtaining the second cropping parameter sequence of the previous frame image of the target image and setting it as the third cropping parameter sequence of the target image, the method further includes: setting the current value to an identifier value for indicating the first mode.

[0030] Optionally, the number of the transitional images is M - 1; the performing parameter smoothing operation according to the first difference and the first cropping parameter sequence to determine a third cropping parameter sequence includes:

[0031] Calculate a ratio between the first difference and a preset interpolation number M.

[0032] On the basis of the second cropping parameter sequence, generate M groups of sequentially increasing smoothed cropping parameter sequences according to the ratio to obtain the third cropping parameter sequence.

[0033] According to a second aspect of the present disclosure, there is provided a video processing apparatus, the apparatus includes:

[0034] An obtaining module, configured to obtain a first cropping parameter sequence of a target image in a video to be cropped, and a second cropping parameter sequence of the previous frame image of the target image;

[0035] A first determining module, configured to, if a first difference between the first cropping parameter sequence and the second cropping parameter sequence exceeds a preset difference range, determine at least one frame image after the target image in the video to be cropped as a transitional image of the target image;

[0036] A smoothing module, configured to smooth the first sequence of cropping parameters according to the first difference to determine a third sequence of cropping parameters; wherein, the second difference between the parameters in each third sequence of cropping parameters and the parameters in the second sequence of cropping parameters is different, and the maximum second difference is less than or equal to the first difference;

[0037] A cropping module, configured to perform object cropping on the target image and the transition image based on the third sequence of cropping parameters.

[0038] Optionally, the apparatus further includes:

[0039] A calculation module, configured to calculate a third difference between each parameter in the first sequence of cropping parameters and each parameter in the second sequence of cropping parameters;

[0040] A second determination module, configured to determine that the first difference exceeds the preset difference range if the absolute value of the third difference of each parameter is greater than the preset absolute value threshold corresponding to each parameter.

[0041] Optionally, the cropping module is specifically configured to:

[0042] Perform object cropping on the target image based on the third sequence of cropping parameters corresponding to the target image, and perform object cropping on the transition image based on the third sequence of cropping parameters corresponding to the transition image.

[0043] Optionally, the apparatus further includes:

[0044] A detection module, configured to detect whether the difference between the third sequence of cropping parameters corresponding to the transition image and the third sequence of cropping parameters of the previous frame image of the transition image exceeds the preset difference range;

[0045] An execution module, configured to perform the operation of performing object cropping on the transition image based on the third sequence of cropping parameters corresponding to the transition image if it does not exceed;

[0046] An emptying module, configured to empty the third sequence of cropping parameters of the transition image that has not been cropped yet and determine the transition image as a new target image if it exceeds.

[0047] Optionally, the apparatus further includes:

[0048] A first setting module, configured to obtain the second sequence of cropping parameters of the previous frame image of the target image and set it as the third sequence of cropping parameters of the target image if the first difference does not exceed the preset difference range; the second sequence of cropping parameters is the sequence of cropping parameters used when performing object cropping on the previous frame image.

[0049] Optionally, the first setting module is specifically configured to:

[0050] If the first difference does not exceed the preset difference range and the target image does not belong to the transition image of the previous target image, obtain the second cropping parameter sequence of the previous frame image of the target image, and set it as the third cropping parameter sequence of the target image.

[0051] Optionally, the first setting module is further specifically configured to:

[0052] Read the cropping parameter sequence stored in the current tail element of the preset parameter queue; the preset parameter queue stores the cropping parameter sequences used when cropping each frame image in the video to be cropped;

[0053] Insert the read cropping parameter sequence as a new tail element into the preset parameter queue to implement setting the third cropping parameter sequence of the target image.

[0054] Optionally, the device further includes:

[0055] A third determination module, configured to determine that the target image does not belong to the transition image of the previous target image if the mode indicated by the current value of the preset mode identifier is the first mode; and determine that the target image belongs to the transition image of the previous target image if the mode indicated by the current value of the preset mode identifier is the second mode;

[0056] A second setting module, configured to set the current value to the identifier value indicating the second mode after performing parameter smoothing according to the first difference and the first cropping parameter sequence;

[0057] A third setting module, configured to set the current value to the identifier value indicating the first mode after obtaining the second cropping parameter sequence of the previous frame image of the target image and setting it as the third cropping parameter sequence of the target image.

[0058] Optionally, the number of the transition images is M - 1; the smoothing module is specifically configured to:

[0059] Calculate the ratio between the first difference and the preset interpolation number M;

[0060] On the basis of the second cropping parameter sequence, generate M groups of sequentially increasing smoothed cropping parameter sequences according to the ratio to obtain the third cropping parameter sequence.

[0061] According to a third aspect of the present disclosure, there is provided an electronic device, including:

[0062] A processor;

[0063] A memory for storing the processor-executable instructions;

[0064] Wherein, the processor is configured to execute the instructions to implement the method according to any one of the first aspect.

[0065] According to a fourth aspect of the present disclosure, there is provided a storage medium, when the instructions in the storage medium are executed by a processor of an electronic device, the electronic device is caused to execute the method according to any one of the first aspect.

[0066] According to a fifth aspect of the present disclosure, there is provided a computer program product, the computer program product includes readable program instructions, when the readable program instructions are executed by a processor of an electronic device, the electronic device is caused to execute the method according to any one of the first aspect

[0067] Compared with the related art, the present disclosure has the following advantages and positive effects:

[0068] The video processing method provided by the embodiment of the present disclosure obtains a first cropping parameter sequence of a target image in a video to be cropped, and a second cropping parameter sequence of the previous frame image of the target image. If a first difference between the first cropping parameter sequence and the second cropping parameter sequence exceeds a preset difference range, at least one frame image in the video to be cropped and after the target image is determined as a transition image of the target image. According to the first difference, parameter smoothing is performed on the first cropping parameter sequence to determine a third cropping parameter sequence; wherein, a second difference between parameters in each third cropping parameter sequence and parameters in the second cropping parameter sequence is different, and the largest second difference is less than or equal to the first difference. Based on the third cropping parameter sequence, object cropping is performed on the target image and the transition image. In the present disclosure, when the change amplitude between the object in the target image and the object in the previous frame image is too large, that is, the first difference exceeds the preset difference range, parameter smoothing is performed according to the first difference, and the change between the object in the target image and the object in the previous frame image is evenly distributed between the previous frame image and the last frame of the transition image. To a certain extent, it can ensure that the change between the object regions obtained by object cropping of the target image / transition image based on the third cropping parameter sequence can be smoothly transitioned, and thus to a certain extent, it can avoid the problem of object cropping jitter and ensure the cropping effect.

[0069] The above description is only an overview of the technical solution of the present disclosure. In order to be able to understand the technical means of the present disclosure more clearly, it can be implemented according to the content of the specification. And in order to make the above and other purposes, features and advantages of the present disclosure more obvious and understandable, the following specifically describes the specific embodiments of the present disclosure. Description of the Drawings

[0070] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the following detailed description of the preferred embodiments. The drawings are only for the purpose of showing the preferred embodiments and are not considered to be a limitation of the present disclosure. Also, throughout the drawings, the same reference numerals are used to denote the same components. In the drawings:

[0071] Figure 1 is a flowchart of the steps of a video processing method provided by an embodiment of the present disclosure;

[0072] Figure 2 is a flowchart of a cropping process provided by an embodiment of the present disclosure;

[0073] Figure 3 is a flowchart of a smoothing process provided by an embodiment of the present disclosure;

[0074] Figure 4 is a block diagram of a video processing apparatus provided by an embodiment of the present disclosure;

[0075] Figure 5 is a block diagram of an apparatus for video processing shown according to an exemplary embodiment;

[0076] Figure 6 is a block diagram of another apparatus for video processing shown according to an exemplary embodiment. Detailed Embodiments

[0077] The exemplary embodiments of the present disclosure will be described in more detail below with reference to the drawings. Although the exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided so that the present disclosure can be more thoroughly understood and the scope of the present disclosure can be fully conveyed to those skilled in the art.

[0078] Figure 1 is a flowchart of the steps of a video processing method provided by an embodiment of the present disclosure, as Figure 1 shown, the method may include:

[0079] Step 101, obtain a first cropping parameter sequence of a target image in a video to be cropped, and a second cropping parameter sequence of the previous frame image of the target image.

[0080] Step 102, if a first difference between the first cropping parameter sequence and the second cropping parameter sequence exceeds a preset difference range, determine at least one frame image after the target image in the video to be cropped as a transition image of the target image.

[0081] In the embodiments of the present disclosure, the video to be cropped can be selected according to actual needs. The target image can be an image other than the first frame image in the video to be cropped. For example, the target image can be any image other than the first frame image, or a partial image. The present disclosure does not limit this. The first cropping parameter sequence of the target image can be determined in advance based on the object key point detection result, that is, the first cropping parameter sequence of the target image can be the original cropping parameter sequence. The second cropping parameter sequence of the previous frame image can be the cropping parameter sequence used when cropping the previous frame image. For example, it can be the original cropping parameter sequence of the previous frame image, or the cropping parameter sequence obtained after parameter smoothing. The present disclosure does not limit this. The cropping object corresponding to the first cropping parameter sequence of the target image is the same as the object to be cropped corresponding to the second cropping parameter sequence of the previous frame image. Among them, the object can be the face area in the image.

[0082] Further, if the difference between the cropping parameter sequences of the object in two adjacent frame images is too large, it often leads to a relatively abrupt change in the object area of the adjacent frame images cropped based on the cropping parameter sequence, and then results in jitter in the object cropping result of the video. Therefore, in this step, it can be first detected whether the first difference between the first cropping parameter sequence of the target image and the second cropping parameter sequence of the previous frame image of the target image exceeds the preset difference range. If the first difference exceeds the preset difference range, it can be determined that the object in the target image is not currently within the static range. Correspondingly, at least one frame image after the target image can be determined as the transition image of the target image to perform parameter smoothing on the target image, so as to reduce the occurrence probability of the object jitter problem.

[0083] Step 103: Perform parameter smoothing according to the first difference and the first cropping parameter sequence to determine the third cropping parameter sequence; the second differences between the parameters in each third cropping parameter sequence and the parameters in the second cropping parameter sequence are different, and the largest second difference is not greater than the first difference.

[0084] In the embodiments of the present disclosure, the cropping parameter sequence can include at least one cropping parameter. For example, the cropping parameter sequence can be a single cropping parameter or a set of multiple cropping parameters. The determined third cropping parameter sequence can include the third cropping parameter sequence corresponding to the target image and the third cropping parameter sequences corresponding to each transition image. For the same parameter, the second differences between the parameter in each third cropping parameter sequence and the parameter in the second cropping parameter sequence can be different. For example, they can change in sequence.

[0085] The first difference can characterize the change amplitude between the object in the target image and the object in the previous frame image. Correspondingly, if the first difference is larger, the change amplitude between the object in the target image and the object in the previous frame image is larger. On the contrary, if the first difference is smaller, the change amplitude between the object in the target image and the object in the previous frame image is smaller. Since the current first difference is large, that is, the change amplitude between the object in the target image and the object in the previous frame image is too large, it may cause object jitter in the object cropping result. Therefore, the first cropping parameter sequence can be smoothed according to the first difference to determine the third cropping parameter sequence of the target image and the transition image. Since the second differences between the parameters in each third cropping parameter sequence and the parameters in the second cropping parameter sequence change sequentially, and the largest second difference is not greater than the first difference, that is, the gap between the cropping parameters of two adjacent frames, namely the target image and the previous frame image, is evenly distributed among multiple frames composed of the target image and the transition images. Correspondingly, the change between the object in the target image and the object in the previous frame image can be evenly distributed between the previous frame image and the last transition image, so as to ensure to a certain extent that the changes between the object regions obtained after object cropping of the target image / transition image based on the third cropping parameter sequence can be smoothly transitioned.

[0086] Step 104: Perform object cropping on the target image and the transition image based on the third cropping parameter sequence of the target image.

[0087] In the embodiments of the present disclosure, the third cropping parameter sequence can be determined in sequence according to the time sequence of each frame image in the video to be cropped, and object cropping can be performed in sequence according to the time sequence. For example, object cropping can be performed on the target image based on the third cropping parameter sequence corresponding to the target image, and object cropping can be performed on the transition image based on the third cropping parameter sequence corresponding to the transition image. Specifically, the object cropping area in the target image can be located first based on the third cropping parameter sequence of the target image, and then the object cropping area can be extracted to achieve object cropping. Then, for the transition image located after the target image, the object cropping area in the transition image is located based on the third cropping parameter sequence of the transition image, and then the object cropping area is extracted to achieve object cropping.

[0088] In summary, the video processing method provided by the embodiments of the present disclosure obtains a first sequence of cropping parameters for a target image in a video to be cropped, and a second sequence of cropping parameters for the previous frame image of the target image. If the first difference between the first sequence of cropping parameters and the second sequence of cropping parameters exceeds a preset difference range, at least one frame image in the video to be cropped and located after the target image is determined as a transition image of the target image. According to the first difference, parameter smoothing is performed on the first sequence of cropping parameters to determine a third sequence of cropping parameters; wherein, the second difference between the parameters in each third sequence of cropping parameters and the parameters in the second sequence of cropping parameters is different, and the largest second difference is less than or equal to the first difference. Based on the third sequence of cropping parameters, object cropping is performed on the target image and the transition image. In the present disclosure, when the change amplitude between the object in the target image and the object in the previous frame image is too large, that is, the first difference exceeds the preset difference range, parameter smoothing is performed according to the first difference, and the change between the object in the target image and the object in the previous frame image is evenly distributed between the previous frame image and the last frame of the transition image. To a certain extent, it can ensure that the change between the object regions obtained after object cropping of the target image / transition image based on the third sequence of cropping parameters can be smoothly transitioned, and thus to a certain extent, it can avoid the problem of object cropping jitter and ensure the cropping effect.

[0089] Optionally, the following steps may also be performed in the embodiments of the present disclosure:

[0090] Step S21: Calculate a third difference between each parameter in the first sequence of cropping parameters and each parameter in the second sequence of cropping parameters.

[0091] In the embodiments of the present disclosure, the sequence of cropping parameters of a frame image may include multiple parameters. For example, in a scenario where the cropping parameters are determined based on object key points, object key points can be detected and calculated, and then the center point coordinates of the object in the original image, the cropping length and width, and the angle between the positive direction of the object and the upward direction can be calculated according to the object key points. The object key points refer to a series of coordinate points with special semantic information located on a two-dimensional object image. Taking face cropping as an example, the object key points can describe the position information of facial features such as the face contour, eyebrows, eyes, mouth, and nose. As the basic description information of the face, face key points are widely used in tasks such as face detection, face three-dimensional reconstruction, and face beautification.

[0092] Accordingly, the multiple parameters included in the cropping parameter sequence of the frame image can specifically be: the center point coordinates of the object in the image, the cropping length, the cropping width, and the angle between the positive direction of the object and the upward direction of the image. Accordingly, in this step, the third difference between the center point coordinates, the third difference between the cropping lengths, the third difference between the cropping widths, and the third difference between the angles can be calculated respectively. The aforementioned first difference is composed of the third differences of all parameters. Of course, the cropping parameter sequence can also include other parameters, and the cropping parameter sequence can also be determined by using a cropping method that does not depend on the key point detection result, so as to reduce the probability of the cropping parameter sequence being inaccurate due to inaccurate object key point detection, and further cause object jitter. Further, in the case of determining the cropping parameter sequence based on the key point detection result, a key point detection method with higher robustness can be selected to ensure the accuracy of the calculated original cropping parameter sequence as much as possible.

[0093] Step S22: If the absolute value of the third difference of each parameter is greater than the preset absolute value threshold corresponding to each parameter, it is determined that the first difference exceeds the preset difference range.

[0094] In the embodiments of the present disclosure, the preset absolute value threshold can be set according to actual needs. In this way, by flexibly configuring the preset absolute value threshold, the detection standard in different scenarios can be flexibly adapted, thereby increasing the flexibility of the scheme implementation. Further, the third difference may be positive or negative. In the case of the same absolute value, the positive third difference and the negative third difference represent the same change amplitude but different change directions. Accordingly, the above preset difference range can include the difference range corresponding to each parameter, and the minimum end value and the maximum end value of this difference range can be the negative preset absolute value threshold and the positive preset absolute value threshold corresponding to this parameter. Accordingly, if the absolute value of the third difference is greater than the preset absolute value threshold, it can be determined that this third difference exceeds the difference range corresponding to this item. Accordingly, if the absolute value of the third difference of all parameters is greater than the preset absolute value threshold, it can be determined that the third difference of all parameters exceeds the difference range corresponding to this item, and further determine that the first difference exceeds the preset difference range. Of course, if the absolute value of the third difference of at least one parameter is not greater than the preset absolute value threshold corresponding to this parameter, it can be determined that the first difference does not exceed the preset difference range.

[0095] Further, if the absolute values of the third differences of all parameters are greater than the preset absolute value thresholds corresponding to the respective parameters, it can be determined that the change amplitude between the object in the target image and the object in the previous frame image is too large, and the object in the target image is not currently within the stationary range. If object cropping is directly performed based on the first cropping parameter sequence, object jitter may occur in the object cropping result. Accordingly, it can be determined that the first difference exceeds the preset difference range, so as to perform parameter smoothing on the target image subsequently, determine the third cropping parameter sequences of the target image and the transition image, and thus distribute the object mutation between the previous frame image and the target image among multiple frame images, thereby reducing the occurrence probability of object jitter.

[0096] In the embodiments of the present disclosure, by calculating the third differences between the parameters in the first cropping parameter sequence and the parameters in the second cropping parameter sequence, it is determined that the first difference exceeds the preset difference range only when the absolute values of the third differences of all parameters are greater than the preset absolute value thresholds corresponding to the respective parameters. Since the parameter smoothing operation is performed on the cropping parameter sequence of the images in the video to be cropped, accordingly, detecting whether the first difference exceeds the preset difference range based on the differences between the cropping parameter sequences can ensure the adaptability of the detection result to the current application scenario to a certain extent, and thus determine the relative accuracy of the detection result. At the same time, only by comparing the third difference with the preset absolute value threshold, it is possible to detect whether the first difference exceeds the preset difference range, which can ensure the detection efficiency to a certain extent.

[0097] Optionally, in the embodiments of the present disclosure, the third cropping parameter sequences of each frame image in the video to be cropped are sequentially stored in a preset parameter queue. The preset parameter queue can be created in the initialization stage in advance. The preset parameter queue can be used to store the cropping parameter sequences used for object cropping of each frame image, that is, the third cropping parameter sequences. Accordingly, when performing the object cropping operation, the third cropping parameter sequence can be read from the preset parameter queue, and the frame image corresponding to the third cropping parameter sequence can be cropped using the third cropping parameter sequence. For example, a set of third cropping parameter sequences can be popped from the queue head, and the frame image corresponding to the third cropping parameter sequence can be cropped based on the set of third cropping parameter sequences.

[0098] In one implementation, in the embodiments of the present disclosure, the following steps may also be performed before object cropping is performed on the transition image based on the third cropping parameter sequence corresponding to the transition image:

[0099] Step S31: Detect whether the difference between the third cropping parameter sequence corresponding to the transition image and the third cropping parameter sequence of the previous frame image of the transition image exceeds the preset difference range.

[0100] For example, the difference between each parameter in the third cropping parameter sequence corresponding to the transition image and each parameter in the third cropping parameter sequence of the previous frame image of the transition image may be calculated, and if the absolute value of the difference between each parameter is greater than the preset absolute value threshold corresponding to each parameter, it may be determined that the difference between the third cropping parameter sequence corresponding to the transition image and the third cropping parameter sequence of the previous frame image of the transition image exceeds the preset difference range. Otherwise, it may be determined that it does not exceed.

[0101] Step S32: If it does not exceed, executing the operation of performing object cropping on the transition image based on the third cropping parameter sequence corresponding to the transition image.

[0102] If it does not exceed the limit, it can be determined that when cropping with the third cropping parameter sequence corresponding to the transition image, the cropped object can smoothly transition with the object cropped in the previous frame image. Therefore, the object cropping operation of the transition image can be performed based on the third cropping parameter sequence corresponding to the transition image.

[0103] Step S33: If exceeded, clear the third cropping parameter sequence of the transition image that has not been cropped yet, and determine the transition image as a new target image.

[0104] If it exceeds, it can be determined that when cropping with the third cropping parameter sequence corresponding to the transition image, there may still be jitter between the candidate cropped objects. Therefore, the third cropping parameter sequence of the transition image that has not been cropped can be cleared, and the current frame of transition image can be determined as the new target image to perform a new round of parameter smoothing. In the embodiment of the present disclosure, if it does not exceed, the transition image is cropped based on the third cropping parameter sequence corresponding to the transition image, so as to avoid the problem of object cropping jitter as much as possible.

[0105] Alternatively, in another implementation of the embodiment of the present disclosure, it is also possible to detect whether the third cropping parameter sequence of the previous frame image of the target image is the same as the specified cropping parameter sequence; the specified cropping parameter sequence is the third cropping parameter sequence of the previous frame image of the previous frame image of the target image.

[0106] In an embodiment of the present disclosure, if the third cropping parameter sequence of the previous frame image of the target image is the same as the third cropping parameter sequence of the previous frame image of the previous frame image, it can be indicated that, relative to its previous frame image, the object is within the stationary range for the previous frame image, and the previous frame image uses the third cropping parameter sequence of its previous frame image, and the current is in the "hold" mode. Conversely, if they are different, it can be determined that, relative to its previous frame image, the object is not within the stationary range for the previous frame image, and the previous frame image does not use the third cropping parameter sequence of its previous frame image, and the finally determined cropping parameter sequence for the previous frame image is obtained by interpolation, that is, the current is in the "interpolation" mode. Wherein, the current mode is used to indicate whether the finally adopted cropping parameter sequence of the previous frame image of the current target image uses the third cropping parameter sequence of its previous frame image.

[0107] If the third cropping parameter sequence of the previous frame image of the target image is different from the specified cropping parameter sequence, detect whether the target image belongs to a transition image of the previous target image.

[0108] If the third cropping parameter sequence of the previous frame image of the target image is different from the specified cropping parameter sequence, that is, the current is in the "interpolation" mode, the current target image may belong to a transition image of the previous target image, and the third cropping parameter sequence of the current target image has been determined when performing parameter smoothing for the previous target image. Therefore, in an embodiment of the present disclosure, it can be further detected whether the target image belongs to a transition image of the previous target image. Specifically, it can be checked whether all groups of the third cropping parameter sequences calculated last time (that is, the third cropping parameter sequences of the previous target image and the transition images of the previous target image) have been applied. In an embodiment of the present disclosure, after the third cropping parameter sequence of the transition image is applied, that is, after using the third cropping parameter sequence of the transition image to crop the object in the transition image, a preset identifier can be set for the cropping of the third cropping parameter sequence of the transition image. Correspondingly, it can be detected whether there is a third cropping parameter sequence without the preset identifier set in the third cropping parameter sequences of the transition images of the previous target image. If so, it can be determined that the target image belongs to the previous target image, and vice versa, it can be determined that the target image does not belong to the previous target image. Alternatively, after the third cropping parameter sequence of the transition image is applied, the third cropping parameter sequence of the transition image can be cleared from the preset parameter queue. Correspondingly, it can be detected whether the preset parameter queue still stores the third cropping parameter sequences of the transition images of the previous target image. If it exists, it can be determined that the target image belongs to the previous target image, and vice versa, it can be determined that the target image does not belong to the previous target image.

[0109] Of course, if the third cropping parameter sequence of the previous frame image is the same as the specified cropping parameter sequence, that is, the current is in the "hold" mode, the step of smoothing the parameters of the first cropping parameter sequence according to the first difference to determine the third cropping parameter sequence can be directly executed.

[0110] If the target image belongs to the transition image of the previous target image, clear the third cropping parameter sequence corresponding to the transition image of the previous target image from the preset parameter queue.

[0111] If the target image belongs to the previous target image, it can be determined that the third cropping parameter sequence of the transition image of the previous target image has not been fully applied, and there is still an unapplied third cropping parameter sequence of the transition image of the previous target image in the preset parameter queue. However, since the object of the current target image changes greatly from the previous frame image, if the previously determined third cropping parameter sequence of the transition image is directly applied, it cannot cope with the object change between the current target image and the previous frame image. Therefore, the third cropping parameter sequences of the current target image and the transition image can be recalculated to further reduce the probability of object jitter in the cropping result. Correspondingly, the third cropping parameter sequence corresponding to the transition image of the previous target image can be cleared from the preset parameter queue. In this way, by further clearing these unapplied third cropping parameter sequences, space can be saved, and the problem that the terminal performs unnecessary cropping operations due to the existence of unrequired third cropping parameter sequences in the preset parameter queue can be avoided, thereby ensuring that the cropping operation can proceed orderly.

[0112] In the embodiments of the present disclosure, when it is detected that the third cropping parameter sequence of the previous frame image is different from the specified cropping parameter sequence, that is, the current target image may belong to the transition image of the previous target image, it is further detected whether the target image belongs to the transition image of the previous target image. If it does not belong, the third cropping parameter sequence corresponding to the transition image of the previous target image is correspondingly cleared, thereby saving space and ensuring the orderly progress of the cropping operation.

[0113] Optionally, in the embodiments of the present disclosure, the following steps may also be executed:

[0114] Step S41: If the first difference does not exceed the preset difference range, obtain the second cropping parameter sequence of the previous frame image of the target image to be set as the third cropping parameter sequence of the target image; the second cropping parameter sequence is the parameter used for object cropping of the previous frame image.

[0115] In the embodiments of the present disclosure, if the first difference does not exceed the preset difference range, that is, the object in the target image is within the static range, the second cropping parameter sequence of the previous frame image can be directly obtained as the third cropping parameter sequence of the target image to achieve smooth processing, thereby saving processing resources. If there are multiple consecutive frames in the video to be cropped and the first differences between these consecutive frames do not exceed the preset difference range, then the third cropping parameter sequences used for object cropping of these consecutive frames will be the same. That is, if in the "hold" mode, the parameter is the fixed parameter during the current "hold" mode, the object cropping parameter sequence remains static, and the cropped object image also remains unchanged. Correspondingly, if in the "interpolation" mode, the parameter is a set of interpolation parameters (i.e., the third object cropping parameter sequence obtained by smooth processing) that are sequentially applied during the current "interpolation" mode, that is, the cropped object image is in the process of transitioning from one position on the target image to another position.

[0116] In the embodiments of the present disclosure, when the first difference does not exceed the preset difference range, the second cropping parameter sequence used for object cropping of the previous frame image is determined as the third cropping parameter sequence of the target image. In this way, when the object in the frame image remains static, that is, within the static range, the consistency of the parameters used for object cropping can be maintained, thereby ensuring the absolute stability of object cropping and further avoiding the probability of object jitter in the cropping result.

[0117] Optionally, in the embodiments of the present disclosure, further, the operation of obtaining the second cropping parameter sequence of the previous frame image of the target image and setting it as the third cropping parameter sequence of the target image when the first difference does not exceed the preset difference range may specifically be: if the first difference does not exceed the preset difference range and the target image does not belong to the transition image of the previous target image, then obtain the second cropping parameter sequence of the previous frame image of the target image and set it as the third cropping parameter sequence of the target image. In this way, it can be avoided that when the target image belongs to the transition image of the previous target image and the third cropping parameter sequence determined by the previous smooth processing can be applied, the second cropping parameter sequence of the previous frame image of the target image is mistakenly set as the third cropping parameter sequence of the target image, thereby avoiding unnecessary operations and saving processing resources.

[0118] Further, the operation of obtaining the second cropping parameter sequence of the previous frame image of the target image and setting it as the third cropping parameter sequence of the target image may specifically include: reading the parameters stored in the current tail element of the preset parameter queue; the preset parameter queue stores the cropping parameter sequences used for cropping each frame image in the video to be cropped; inserting the read cropping parameter sequence as the new tail element into the preset parameter queue to achieve setting the third cropping parameter sequence of the target image. Wherein, one element in the preset parameter queue can store the cropping parameter sequence used for cropping a frame image. Wherein, the third cropping parameter sequence can be understood as the cropping parameter sequence used for cropping. Of course, the cropping parameter sequence used for cropping can also be named in other ways, and the embodiments of the present disclosure do not limit this. Specifically, the parameters stored in the current tail element of the preset parameter queue can be directly inserted into the queue tail again to increase the third cropping parameter sequence of the target image in the preset parameter queue, thereby completing the setting.

[0119] In the embodiments of the present disclosure, by pre-creating a preset parameter queue, only the operation of inserting the parameters stored in the current tail element as the new tail element into the preset parameter queue again needs to be executed to achieve the setting, and thus the acquisition efficiency can be ensured to a certain extent.

[0120] That is to say, if the third cropping parameter sequence of the previous frame image of the target image is the same as the specified cropping parameter sequence, that is, the current is in the "hold" mode, the step of obtaining the second cropping parameter sequence of the previous frame image of the target image and setting it as the third cropping parameter sequence of the target image can be executed. That is, in the embodiments of the present disclosure, when the current is in the "hold" mode and the object in the target image is within the static range, the current tail element of the preset parameter queue can be written into the preset parameter queue again as the new tail element.

[0121] If the third cropping parameter sequence of the previous frame image of the target image is different from the specified cropping parameter sequence, that is, currently in the "interpolation" mode, the current target image may be a transitional image of the previous target image. The third cropping parameter sequence of the current target image has been determined when smoothing the parameters for the previous target image. Further, if the target image does not belong to the transitional image of the previous target image, that is, the third cropping parameter sequence of the transitional image of the previous target image has been applied, then perform the step of obtaining the third cropping parameter sequence of the previous frame image and setting it as the third cropping parameter sequence of the target image. That is, in the embodiments of the present disclosure, when currently in the "interpolation" mode, the object in the target image is within the static range, and all the calculated third cropping parameter sequences of all groups in the previous calculation have been applied, the current tail element of the preset parameter queue can be written into the preset parameter queue again as the new tail element. In the embodiments of the present disclosure, different smoothing methods are used for the static and non-static states of the object, which can maintain the absolute stability of the object cropping when it is static. When it is non-static, the interpolation method is used to achieve a smoother transition and weaken the sudden change of the effect.

[0122] If the target image belongs to the transitional image of the previous target image, that is, the third cropping parameter sequence of the transitional image of the previous target image has not been fully applied, then the third cropping parameter sequence corresponding to the transitional image of the previous target image can be read. For example, the first unapplied third cropping parameter sequence in the third cropping parameter sequence of the transitional image of the previous target image can be determined as the third cropping parameter sequence of the target image, so as to ensure that the third cropping parameter sequence determined for the transitional image of the previous target image can be used normally. In this way, it can be ensured that the determined transitional image can be accurately applied to the object cropping, and the transitional image can play a smoothing role, thereby reducing the probability of object jitter in the cropping result.

[0123] Optionally, the embodiments of the present disclosure can also perform the following operations to determine whether the target image belongs to the transitional image of the previous target image: Step S51, if the mode indicated by the current value of the preset mode flag is the first mode, it is determined that the target image does not belong to the transitional image of the previous target image; if the mode indicated by the current value of the preset mode flag is the second mode, it is determined that the target image belongs to the transitional image of the previous target image.

[0124] Among them, the first mode can be the "hold" mode. If the mode indicated by the current value of the preset mode identifier is the first mode, it can be determined that the previous frame image of the target image has adopted the third cropping parameter sequence of its previous frame image. That is, currently in the "hold" mode, no parameter smoothing operation is performed, and the current target image does not belong to the transition image of the previous target image. The second mode can be the "interpolation" mode. If the mode indicated by the current value of the preset mode identifier is the second mode, it can be determined that the previous frame image of the target image has not adopted the third cropping parameter sequence of its previous frame image, and the third cropping parameter sequence of the previous frame image is obtained through parameter smoothing calculation. That is, currently in the "interpolation" mode, the current target image belongs to the transition image of the previous target image. The identification value for indicating the first mode and the identification value for indicating the second mode can be set according to actual needs. For example, the identification value for indicating the first mode can be "1", and the identification value for indicating the second mode can be "0". The embodiments of the present disclosure do not limit this. The current value of the preset mode identifier can be set after determining the third cropping parameter sequence of the previous frame image of the current target image in the previous round. In the embodiments of the present disclosure, only by detecting the mode indicated by the current value of the preset mode identifier, it is possible to conveniently determine whether the target image belongs to the transition image of the previous target image, thereby improving the overall processing efficiency.

[0125] Correspondingly, in the embodiments of the present disclosure, the following operations may also be performed when performing parameter smoothing according to the first difference and the first cropping parameter sequence: Step S61, set the current value to the identification value for indicating the second mode.

[0126] If in this round of processing, parameter smoothing is performed according to the first difference and the first cropping parameter sequence to determine the second cropping parameter sequence, it means that the third cropping parameter sequence of the previous frame image of the target image in this round is not adopted. Correspondingly, the current value can be set to the identification value for indicating the second mode to ensure that in the next round of processing, to a certain extent, the operation of detecting whether the third cropping parameter sequence of the previous frame image is the same as the specified cropping parameter sequence can be accurately realized based on the current value. Specifically, if the mode indicated by the current value is currently the second mode, the current value can be kept to achieve the setting. Or, if the mode indicated by the current value is currently the first mode, the current value can be modified, for example, changing the current value from "1" to "0", to achieve the setting.

[0127] After obtaining the second cropping parameter sequence of the previous frame image of the target image and setting it as the third cropping parameter sequence of the target image, step S62 can be executed: set the current value to the identification value for indicating the first mode.

[0128] If in this round of processing, the second cropping parameter sequence of the previous frame image of the target image is obtained to set as the third cropping parameter sequence of the target image, it means that the target image of this round continues to use the second cropping parameter sequence of the previous frame image of the target image. Accordingly, the current value can be set to an identification value for indicating the first mode to ensure that in the next round of processing, the operation of accurately detecting whether the target image belongs to the transition image of the previous target image can be accurately implemented based on the current value. Specifically, if the mode indicated by the current value is the first mode, the current value can be maintained to achieve the setting. Alternatively, if the mode indicated by the current value is the second mode, the current value can be modified, for example, the current value can be modified from "0" to "1" to achieve the setting.

[0129] In the disclosed embodiment, two parameter smoothing modes are set up, namely, "maintain" and "interpolate" modes, and the parameter smoothing modes are flexibly updated, and different smoothing strategies are adopted for different modes. By updating and maintaining the value of the preset mode identifier, subsequent processing only needs to detect the mode represented by the preset detection mode identifier to realize the operation of detecting whether the target image is a transition image of the previous target image, thereby ensuring the detection efficiency to a certain extent.

[0130] Optionally, in the embodiment of the present disclosure, the number of transition images may be M-1. Accordingly, the operation of performing parameter smoothing operation according to the first difference and the first cropping parameter sequence to determine the third cropping parameter sequence may specifically include:

[0131] Step S71, calculating the ratio between the first difference and a preset interpolation number M.

[0132] In the embodiment of the present disclosure, the preset interpolation number M can be set according to actual needs, and M can be an integer not less than 1. M can be used to represent the number of transition frames used to smoothly transition an object from a static position to a current frame position. The larger M is, the longer the smoothing process is, and the better the smoothing effect is. Specifically, it can be the ratio between any parameter in the first difference and the preset interpolation number M.

[0133] Step S72: Based on the second cropping parameter sequence and according to the ratio, M groups of smoothing cropping parameter sequences that increase in sequence are generated to obtain the third cropping parameter sequence.

[0134] For example, for any parameter, the ratio of the parameter + X * the parameter in the second clipping parameter sequence can be calculated, where X takes values of 1, 2, …, M. Then, each parameter in M groups of smoothed clipping parameter sequences can be obtained, and thus M third clipping parameter sequences can be obtained. Further, the smoothed clipping parameter sequence calculated when X takes the value of 1 can be determined as the third clipping parameter sequence of the target image, the smoothed clipping parameter sequence calculated when X takes the value of 2 can be determined as the third clipping parameter sequence of the transition image with the earliest time sequence, …, and the smoothed clipping parameter sequence calculated when X takes the value of M can be determined as the third clipping parameter sequence of the transition image with the latest time sequence.

[0135] In the embodiments of the present disclosure, the ratio between the first difference and the preset interpolation number M is calculated, and based on the second clipping parameter sequence, M groups of sequentially increasing smoothed clipping parameter sequences are generated according to the ratio, so that the third clipping parameter sequence can be obtained, and to a certain extent, the efficiency of interpolation smoothing can be ensured.

[0136] Optionally, in the embodiments of the present disclosure, the target image can also be removed when the absolute value of the first difference is greater than a specified threshold. Specifically, filtering methods such as Kalman filtering can be used to determine the original clipping parameter sequence with excessive jitter (the absolute value of the first difference is greater than the specified threshold) in the entire frame image sequence as an outlier and filter out the corresponding one, so as to achieve a smoothing effect. Or, the default value can be directly set as the third clipping parameter sequence of the target image. The default value can be determined based on the third clipping parameter sequences of the previous N frames. For example, the default value can be the mean of the third clipping parameter sequences of the previous N frames, and thus a smoothing effect can be achieved to a certain extent.

[0137] Next, an application scenario related to the embodiments of the present disclosure will be described. Currently, in the preprocessing step in the field of portrait stylization based on Generative Adversarial Networks (GANs), a face image is often cropped, and the cropping of the face image often depends on the detection results of face key points. Among them, GAN is a method of unsupervised learning, which learns by making two neural networks play against each other. GAN consists of a generative network and a discriminative network. The two networks compete with each other and continuously adjust parameters. The ultimate goal is to make the discriminative network unable to determine whether the output result of the generative network is real, and it is often used to generate realistic-looking pictures. However, in the time sequence, the detection results of face key points are often not stable enough, which leads to jitter in the cropping results of face images based on the detection results of face key points, and further leads to jitter in the size, position, and angle of the cropped face images between frames, making the results of portrait stylization under the field of view unstable, with problems such as color flickering and deformation jitter.

[0138] Figure 2 is a flowchart of a cropping process provided by an embodiment of the present disclosure. As Figure 2 shown, face key point detection can be performed on the input original image. Then, face cropping parameter calculation is performed according to the face key point detection result to obtain the original cropping parameter sequence of the frame image (i.e., the first cropping parameter sequence). Next, an operation of smoothing the face cropping parameters can be performed. Specifically, the unsmoothed first cropping parameter sequence of the current frame can be input into the parameter smoothing module to obtain the smoothed object cropping parameters (i.e., the third cropping parameter sequence). Finally, face image cropping can be performed on the original image according to the smoothed third cropping parameters. For example, according to the center point coordinates, cropping length, and cropping width in the third cropping parameter sequence, the original image is cropped, and then the cropped image is rotated and scaled according to the included angle in the third cropping parameter sequence, so as to obtain a face image with the face centered, the proportion occupied being unified, the size being unified, and the positive direction being upward, thereby obtaining the cropped face image.

[0139] Furthermore, the parameter smoothing module can implement each of the above-mentioned steps. By way of example, Figure 3 is a flowchart of a smoothing process provided by an embodiment of the present disclosure. As Figure 3 shown, it can first be determined whether the current is in the "interpolation" mode according to the face cropping parameters of the current frame (the first cropping parameter sequence). If so (i.e., the mode indicated by the current value of the preset mode identifier is the second mode), it can be determined whether the face of the current frame is within the static range (i.e., it is determined whether the first difference exceeds the preset difference range). If so, it can be determined whether the previous interpolation parameter has been applied (i.e., it is detected whether the target image belongs to the transition image of the previous target image). If so, the parameter at the end of the preset parameter queue can be inserted again at the end of the parameter queue, and the current state is modified to the "hold" mode (the current value is set to the identifier value used to indicate the first mode). Otherwise, the third cropping parameter sequence corresponding to the transition image of the previous target image can be determined as the third cropping parameter sequence of the target image. Specifically, the third cropping parameter sequence corresponding to the transition image of the previous target image in the preset queue parameters can be popped to perform object cropping. And the parameter at the head of the queue is returned for convenient detection.

[0140] If the current mode is "interpolation" but the face in the current frame is within the static range, the interpolation between the tail parameter of the queue and p_now can be calculated (i.e., parameter smoothing is performed on the first cropping parameter sequence according to the first difference to determine the third cropping parameter sequence), then the current queue is cleared, and then all the interpolations are inserted into the tail of the queue in sequence (i.e., the third cropping parameter sequences of the target image and the transition image are inserted into the tail of the queue in sequence). If the current mode is not "interpolation" and the face in the current frame is not within the static range, the interpolation between the tail parameter of the queue and p_now can be calculated. Then all the interpolations are inserted into the tail of the queue in sequence, and the current state is modified to the "interpolation" mode (the current value is set to the identification value for indicating the second mode).

[0141] Furthermore, if the current mode is not "interpolation" and the face in the current frame is within the static range, the parameter at the tail of the preset parameter queue can be directly inserted into the tail of the parameter queue again.

[0142] In the embodiments of the present disclosure, face cropping based on temporal smoothing can crop stable and smooth face images in the temporal sequence in both the face static and face moving states, thereby weakening the influence of the jitter of the face key point detection result on face cropping, and improving the stability of cropping and the stability of the portrait stylization effect and the visual effect.

[0143] Figure 4 is a block diagram of a video processing device provided by the embodiments of the present disclosure. As Figure 4 shown, the device 20 may include:

[0144] An acquisition module 201, configured to acquire a first cropping parameter sequence of a target image in a video to be cropped, and a second cropping parameter sequence of the previous frame image of the target image;

[0145] A first determination module 202, configured to determine at least one frame image after the target image in the video to be cropped as the transition image of the target image if the first difference between the first cropping parameter sequence and the second cropping parameter sequence exceeds a preset difference range;

[0146] A smoothing module 203, configured to perform parameter smoothing on the first cropping parameter sequence according to the first difference to determine a third cropping parameter sequence; wherein, the second difference between the parameters in each third cropping parameter sequence and the parameters in the second cropping parameter sequence is different, and the largest second difference is less than or equal to the first difference;

[0147] A cropping module 204, configured to perform object cropping on the target image and the transition image based on the third cropping parameter sequence.

[0148] The video processing device provided by an embodiment of the present disclosure obtains a first sequence of cropping parameters of a target image in a video to be cropped, and a second sequence of cropping parameters of the previous frame image of the target image. If a first difference between the first sequence of cropping parameters and the second sequence of cropping parameters exceeds a preset difference range, at least one frame image in the video to be cropped and after the target image is determined as a transition image of the target image. According to the first difference, parameter smoothing is performed on the first sequence of cropping parameters to determine a third sequence of cropping parameters; wherein, a second difference between parameters in each third sequence of cropping parameters and parameters in the second sequence of cropping parameters is different, and the largest second difference is less than or equal to the first difference. Based on the third sequence of cropping parameters, object cropping is performed on the target image and the transition image. In the present disclosure, when the change amplitude between the object in the target image and the object in the previous frame image is too large, that is, the first difference exceeds the preset difference range, parameter smoothing is performed according to the first difference, and the change between the object in the target image and the object in the previous frame image is evenly distributed between the previous frame image and the last frame of the transition image. To a certain extent, it can ensure that the change between the object regions obtained by performing object cropping on the target image / transition image based on the third sequence of cropping parameters subsequently can be smoothly transitioned, and thus to a certain extent, it can avoid the problem of object cropping jitter and ensure the cropping effect.

[0149] Optionally, the device 20 further includes:

[0150] A calculation module, configured to calculate a third difference between each parameter in the first sequence of cropping parameters and each parameter in the second sequence of cropping parameters;

[0151] A second determination module, configured to determine that the first difference exceeds the preset difference range if the absolute value of the third difference of each parameter is greater than the preset absolute value threshold corresponding to each parameter.

[0152] Optionally, the cropping module 204 is specifically configured to:

[0153] Perform object cropping on the target image based on the third sequence of cropping parameters corresponding to the target image, and perform object cropping on the transition image based on the third sequence of cropping parameters corresponding to the transition image.

[0154] Optionally, the device 20 further includes:

[0155] A detection module, configured to detect whether a difference between the third sequence of cropping parameters corresponding to the transition image and the third sequence of cropping parameters of the previous frame image of the transition image exceeds the preset difference range;

[0156] An execution module, configured to perform the operation of performing object cropping on the transition image based on the third sequence of cropping parameters corresponding to the transition image if it does not exceed.

[0157] The clearing module is configured to, if it exceeds, clear the third cropping parameter sequence of the current uncropped transition image and determine the transition image as the new target image.

[0158] Optionally, the apparatus 20 further includes:

[0159] The first setting module is configured to, if the first difference does not exceed the preset difference range, obtain the second cropping parameter sequence of the previous frame image of the target image and set it as the third cropping parameter sequence of the target image; the second cropping parameter sequence is the cropping parameter sequence used when performing object cropping on the previous frame image.

[0160] Optionally, the first setting module is specifically configured to:

[0161] If the first difference does not exceed the preset difference range and the target image does not belong to the transition image of the previous target image, obtain the second cropping parameter sequence of the previous frame image of the target image and set it as the third cropping parameter sequence of the target image.

[0162] Optionally, the first setting module is further specifically configured to:

[0163] Read the cropping parameter sequence stored in the current tail element of the preset parameter queue; the preset parameter queue stores the cropping parameter sequences used when cropping each frame image in the video to be cropped;

[0164] Insert the read cropping parameter sequence as the new tail element into the preset parameter queue to implement setting the third cropping parameter sequence of the target image.

[0165] Optionally, the apparatus 20 further includes:

[0166] The third determination module is configured to, if the mode indicated by the current value of the preset mode identifier is the first mode, determine that the target image does not belong to the transition image of the previous target image; if the mode indicated by the current value of the preset mode identifier is the second mode, determine that the target image belongs to the transition image of the previous target image;

[0167] The second setting module is configured to set the current value to the identifier value indicating the second mode after performing parameter smoothing operation according to the first difference and the first cropping parameter sequence;

[0168] The third setting module is configured to set the current value to the identifier value indicating the first mode after obtaining the second cropping parameter sequence of the previous frame image of the target image and setting it as the third cropping parameter sequence of the target image.

[0169] Optionally, the number of the transition images is M - 1; the smoothing module 203 is specifically configured to:

[0170] Calculate a ratio between the first difference and a preset interpolation number M;

[0171] Based on the second clipping parameter sequence, generate M groups of sequentially increasing smoothing clipping parameter sequences according to the ratio to obtain the third clipping parameter sequence.

[0172] Regarding the device in the above embodiments, the specific manners in which each module performs operations have been described in detail in the embodiments related to the method, and will not be elaborated herein.

[0173] According to an embodiment of the present disclosure, there is provided an electronic device, including: a processor and a memory for storing processor-executable instructions, wherein the processor is configured to implement the steps in the video processing method in any of the above embodiments when executed.

[0174] According to an embodiment of the present disclosure, there is also provided a storage medium, when the instructions in the storage medium are executed by a processor of an electronic device, enabling the electronic device to execute the steps in the video processing method in any of the above embodiments.

[0175] According to an embodiment of the present disclosure, there is also provided a computer program product, which includes readable program instructions, and when the readable program instructions are executed by a processor of an electronic device, enabling the electronic device to execute the steps in the video processing method in any of the above embodiments.

[0176] Figure 5 It is a block diagram of a device for video processing shown according to an exemplary embodiment. For example, the device 700 may be a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, etc.

[0177] Referring to Figure 5 , the device 700 may include one or more of the following components: a processing component 702, a memory 704, a power component 706, a multimedia component 708, an audio component 710, an input / output (I / O) interface 712, a sensor component 714, and a communication component 716.

[0178] The processing component 702 generally controls the overall operation of the device 700, such as operations associated with display, telephone calls, data communication, camera operation, and recording operation. The processing component 702 may include one or more processors 720 to execute instructions to complete all or part of the steps of the above video processing method. In addition, the processing component 702 may include one or more modules to facilitate the interaction between the processing component 702 and other components. For example, the processing component 702 may include a multimedia module to facilitate the interaction between the multimedia component 708 and the processing component 702.

[0179] The memory 704 is configured to store various types of data to support the operation of the device 700. Examples of such data include instructions for any application or method operating on the device 700, contact data, phone book data, messages, pictures, videos, etc. The memory 704 may be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk, or optical disk.

[0180] The power component 706 provides power to various components of the device 700. The power component 706 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power for the device 700.

[0181] The multimedia component 708 includes a screen that provides an output interface between the device 700 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors may not only sense the boundaries of touch or swipe actions, but also detect the duration and pressure associated with the touch or swipe operation. In some embodiments, the multimedia component 708 includes a front camera and / or a rear camera. When the device 700 is in an operation mode, such as a shooting mode or a video mode, the front camera and / or the rear camera may receive external multimedia data. Each of the front camera and the rear camera may be a fixed optical lens system or have a focal length and optical zoom capabilities.

[0182] The audio component 710 is configured to output and / or input audio signals. For example, the audio component 710 includes a microphone (MIC) that is configured to receive external audio signals when the device 700 is in an operating mode, such as a call mode, a recording mode, and a voice recognition mode. The received audio signals can be further stored in the memory 704 or transmitted via the communication component 716. In some embodiments, the audio component 710 further includes a speaker for outputting audio signals.

[0183] The I / O interface 712 provides an interface between the processing component 702 and a peripheral interface module, and the peripheral interface module may be a keyboard, a click wheel, buttons, etc. These buttons may include, but are not limited to: a home button, a volume button, a power button, and a lock button.

[0184] The sensor component 714 includes one or more sensors for providing an assessment of various aspects of the state of the device 700. For example, the sensor component 714 can detect the open / closed state of the device 700, the relative positioning of components, such as the display and keypad of the device 700, the sensor component 714 can also detect a change in the position of the device 700 or a component of the device 700, the presence or absence of user contact with the device 700, the orientation or acceleration / deceleration of the device 700, and the temperature change of the device 700. The sensor component 714 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. The sensor component 714 may also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor component 714 may further include an acceleration sensor, a gyro sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0185] The communication component 716 is configured to facilitate communication between the device 700 and other devices in a wired or wireless manner. The device 700 can access a wireless network based on a communication standard, such as WiFi, a carrier network (such as 2G, 3G, 4G, or 5G), or a combination thereof. In an exemplary embodiment, the communication component 716 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 716 further includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0186] In an exemplary embodiment, the apparatus 700 may be implemented by one or more application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components for performing the above video processing method.

[0187] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 704 including instructions, and the above instructions can be executed by a processor 720 of the apparatus 700 to complete the above video processing method. For example, the non-transitory computer-readable storage medium may be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, and an optical data storage device, etc.

[0188] Figure 6 is a block diagram of another apparatus for video processing shown according to an exemplary embodiment. For example, the apparatus 800 may be provided as a server. Referring to Figure 6 , the apparatus 800 includes a processing component 822, which further includes one or more processors, and memory resources represented by a memory 832 for storing instructions executable by the processing component 822, such as application programs. The application programs stored in the memory 832 may include one or more modules each corresponding to a set of instructions. In addition, the processing component 822 is configured to execute instructions to perform the above video processing method.

[0189] The apparatus 800 may further include a power component 826 configured to perform power management of the apparatus 800, a wired or wireless network interface 850 configured to connect the apparatus 800 to a network, and an input / output (I / O) interface 858. The apparatus 800 may operate based on an operating system stored in the memory 832, such as Windows ServerTM, Mac OS XTM, UnixTM, LinuxTM, FreeBSDTM, or the like.

[0190] Those skilled in the art will readily conceive of other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. The present disclosure is intended to cover any variations, uses, or adaptations of the present disclosure, which follow the general principles of the present disclosure and include common general knowledge or conventional technical means in the technical field not disclosed by the present disclosure. The specification and embodiments are only to be considered as exemplary, and the true scope and spirit of the present disclosure are pointed out by the following claims.

[0191] It should be understood that the present disclosure is not limited to the exact structures described above and shown in the drawings, and various modifications and changes can be made without departing from its scope. The scope of the present disclosure is only limited by the appended claims.

Claims

1. A video processing method, characterized in that, the method includes: obtaining a first cropping parameter sequence of a target image in a video to be cropped, and a second cropping parameter sequence of the previous frame image of the target image; if a first difference between the first cropping parameter sequence and the second cropping parameter sequence exceeds a preset difference range, determining at least one frame image after the target image in the video to be cropped as a transition image of the target image; performing parameter smoothing on the first cropping parameter sequence according to the first difference to determine a third cropping parameter sequence; wherein, a second difference between parameters in each third cropping parameter sequence and parameters in the second cropping parameter sequence is different, and the largest second difference is less than or equal to the first difference; performing object cropping on the target image and the transition image based on the third cropping parameter sequence; calculating a third difference between each parameter in the first cropping parameter sequence and each parameter in the second cropping parameter sequence; if the absolute value of the third difference of each parameter is greater than a preset absolute value threshold corresponding to each parameter, determining that the first difference exceeds the preset difference range.

2. The method according to claim 1, characterized in that, the performing object cropping on the target image and the transition image based on the third cropping parameter sequence includes: performing object cropping on the target image based on the third cropping parameter sequence corresponding to the target image, and performing object cropping on the transition image based on the third cropping parameter sequence corresponding to the transition image.

3. The method according to claim 2, characterized in that, before performing object cropping on the transition image based on the third cropping parameter sequence corresponding to the transition image, the method further includes: detecting whether a difference between the third cropping parameter sequence corresponding to the transition image and the third cropping parameter sequence of the previous frame image of the transition image exceeds the preset difference range; if not, performing the operation of performing object cropping on the transition image based on the third cropping parameter sequence corresponding to the transition image; if it exceeds, clearing the third cropping parameter sequence of the current uncropped transition image and determining the transition image as a new target image.

4. The method according to claim 2, characterized in that, the method further includes: if the first difference does not exceed the preset difference range, obtaining the second cropping parameter sequence of the previous frame image of the target image to set as the third cropping parameter sequence of the target image; the second cropping parameter sequence is the cropping parameter sequence used when performing object cropping on the previous frame image.

5. The method according to claim 4, characterized in that, the if the first difference does not exceed the preset difference range, obtaining the second cropping parameter sequence of the previous frame image of the target image to set as the third cropping parameter sequence of the target image includes: If the first difference does not exceed the preset difference range and the target image does not belong to the transition image of the previous target image, obtain the second cropping parameter sequence of the previous frame image of the target image and set it as the third cropping parameter sequence of the target image.

6. The method according to claim 5, wherein, the obtaining the second cropping parameter sequence of the previous frame image of the target image and setting it as the third cropping parameter sequence of the target image includes: reading the cropping parameter sequence stored in the current tail element of the preset parameter queue; the preset parameter queue stores the cropping parameter sequences used when cropping each frame image in the video to be cropped; inserting the read cropping parameter sequence as a new tail element into the preset parameter queue to implement setting the third cropping parameter sequence of the target image.

7. The method according to claim 5 or 6, wherein, the method further includes: if the mode indicated by the current value of the preset mode identifier is the first mode, determine that the target image does not belong to the transition image of the previous target image; if the mode indicated by the current value of the preset mode identifier is the second mode, determine that the target image belongs to the transition image of the previous target image; after performing parameter smoothing operation according to the first difference and the first cropping parameter sequence, the method further includes: setting the current value to the identifier value for indicating the second mode; after obtaining the second cropping parameter sequence of the previous frame image of the target image and setting it as the third cropping parameter sequence of the target image, the method further includes: setting the current value to the identifier value for indicating the first mode.

8. The method according to claim 1, wherein, the number of the transition images is M - 1; the performing parameter smoothing according to the first difference and the first cropping parameter sequence to determine the third cropping parameter sequence includes: calculating the ratio between the first difference and the preset interpolation number M; generating M groups of smoothly increasing cropping parameter sequences based on the ratio on the basis of the second cropping parameter sequence to obtain the third cropping parameter sequence.

9. A video processing device, wherein, the device includes: an obtaining module, configured to obtain a first cropping parameter sequence of a target image in a video to be cropped, and a second cropping parameter sequence of the previous frame image of the target image; a first determining module, configured to, if a first difference between the first cropping parameter sequence and the second cropping parameter sequence exceeds a preset difference range, determine at least one frame image after the target image in the video to be cropped as the transition image of the target image; a smoothing module, configured to perform parameter smoothing on the first cropping parameter sequence according to the first difference to determine a third cropping parameter sequence; wherein, the second difference between the parameters in each third cropping parameter sequence and the parameters in the second cropping parameter sequence is different, and the maximum second difference is less than or equal to the first difference. A cropping module, configured to perform object cropping on the target image and the transition image based on the third cropping parameter sequence; A calculation module, configured to calculate a third difference between each parameter in the first cropping parameter sequence and each parameter in the second cropping parameter sequence; A second determination module, configured to determine that the first difference exceeds the preset difference range if the absolute value of the third difference of each parameter is greater than the preset absolute value threshold corresponding to each parameter.

10. The apparatus according to claim 9, wherein, the cropping module is specifically configured to: perform object cropping on the target image based on the third cropping parameter sequence corresponding to the target image, and perform object cropping on the transition image based on the third cropping parameter sequence corresponding to the transition image.

11. The apparatus according to claim 10, wherein, the apparatus further comprises: A detection module, configured to detect whether a difference between the third cropping parameter sequence corresponding to the transition image and the third cropping parameter sequence of the previous frame image of the transition image exceeds the preset difference range; An execution module, configured to, if not exceeding, perform the operation of performing object cropping on the transition image based on the third cropping parameter sequence corresponding to the transition image; An emptying module, configured to, if exceeding, empty the third cropping parameter sequence of the transition image that has not been cropped yet, and determine the transition image as a new target image.

12. The apparatus according to claim 10, wherein, the apparatus further comprises: A first setting module, configured to, if the first difference does not exceed the preset difference range, obtain the second cropping parameter sequence of the previous frame image of the target image to set as the third cropping parameter sequence of the target image; the second cropping parameter sequence is the cropping parameter sequence used when performing object cropping on the previous frame image.

13. The apparatus according to claim 12, wherein, the first setting module is specifically configured to: if the first difference does not exceed the preset difference range and the target image does not belong to the transition image of the previous target image, obtain the second cropping parameter sequence of the previous frame image of the target image to set as the third cropping parameter sequence of the target image.

14. The apparatus according to claim 13, wherein, the first setting module is further specifically configured to: read the cropping parameter sequence stored in the current tail element of the preset parameter queue; the preset parameter queue stores the cropping parameter sequences used when cropping each frame image in the video to be cropped; insert the read cropping parameter sequence as a new tail element into the preset parameter queue to implement setting the third cropping parameter sequence of the target image.

15. The apparatus according to claim 13 or 14, wherein, the apparatus further comprises: A third determination module, configured to determine that the target image does not belong to the transition image of the previous target image if the mode indicated by the current value of the preset mode identifier is the first mode; and determine that the target image belongs to the transition image of the previous target image if the mode indicated by the current value of the preset mode identifier is the second mode; A second setting module, configured to set the current value to an identifier value for indicating the second mode after performing parameter smoothing operation according to the first difference and the first sequence of cropping parameters; A third setting module, configured to set the current value to an identifier value for indicating the first mode after obtaining the second sequence of cropping parameters of the previous frame image of the target image and setting it as the third sequence of cropping parameters of the target image.

16. The apparatus according to claim 9, wherein, the number of the transition images is M-1; the smoothing module is specifically configured to: calculate a ratio between the first difference and a preset interpolation number M; generate M groups of sequentially increasing smoothed cropping parameter sequences based on the ratio on the basis of the second sequence of cropping parameters to obtain the third sequence of cropping parameters.

17. An electronic device, wherein, it includes: a processor; a memory for storing executable instructions of the processor; wherein, the processor is configured to execute the instructions to implement the method according to any one of claims 1 to 8.

18. A storage medium, wherein, when the instructions in the storage medium are executed by a processor of an electronic device, the electronic device is caused to execute the method according to any one of claims 1 to 8.

19. A computer program product, wherein, the computer program product includes readable program instructions, and when the readable program instructions are executed by a processor of an electronic device, the electronic device is caused to execute the method according to any one of claims 1 to 8.

Images