Video processing method and apparatus, device, and medium

Abstract

Embodiments of the present disclosure relate to a video processing method and apparatus, a device, and a medium. The method comprises: acquiring a first video to be processed; detecting a target object in original frame images of the first video; executing target operation on the original frame images on the basis of a detection result for the target object to obtain special effect frame images; and generating a second video on the basis of the special effect frame images.

Description

Video processing method, device, equipment and medium

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to Chinese patent application number 202410381638.8, filed on March 29, 2024, entitled “Video Processing Method, Device, Equipment and Medium”, the entire contents of which are incorporated herein by reference. Technical Field

[0003] The present disclosure relates to the field of video processing technology, and in particular to a video processing method, apparatus, device, and medium. Background Art

[0004] Currently, achieving video camera effects primarily relies on camera techniques, requiring specialized shooting techniques or camera-assisted tools. Consequently, many users struggle to capture videos with the desired camera effects. The inventors have discovered that most existing multimedia editing software offers special effects processing capabilities, such as adding special effects to user-uploaded videos to enhance their appeal and interest. Summary of the Invention

[0005] The present disclosure provides a video processing method, apparatus, device, and medium.

[0006] An embodiment of the present disclosure provides a video processing method, which includes: obtaining a first video to be processed; detecting a target object in an original frame image of the first video; performing a target operation on the original frame image based on the detection result of the target object to obtain a special effect frame image; and generating a second video based on the special effect frame image.

[0007] Optionally, the method of performing a target operation on the original frame image based on the detection result of the target object to obtain a special effect frame image includes: dividing the original frame image into an object frame image and an environment frame image based on the detection result of the target object; wherein the target object is included in the object frame image and the target object is not detected in the environment frame image; obtaining target key points in the object frame image, and performing a target operation on the object frame image based on the target key points to obtain a first frame image; obtaining a second frame image for replacing the environment frame image based on a preset transition video; and obtaining a special effect frame image based on the first frame image and the second frame image.

[0008] Optionally, the target key points include a first key point, a second key point and a third key point; and the target operation is performed on the object frame image based on the target key points to obtain a first frame image, including: cropping the object frame image based on the first key point to obtain a cropped frame image; rotating the cropped frame image based on the second key point to obtain a rotated frame image; and scaling the rotated frame image based on the third key point to obtain the first frame image.

[0009] Optionally, the cropping of the object frame image based on the first key point includes: obtaining the relative position relationship between the first key point and the center point of the object frame image; determining the target position information of the target object in the cropped frame image based on the relative position relationship; and cropping the object frame image based on the target position information.

[0010] Optionally, determining the target position information of the target object in the cropped frame image based on the relative position relationship includes: when the object frame image is not the first frame image, obtaining the first position information of the target object in the object frame image, and obtaining the second position information of the target object in the frame image that is located before the object frame image in the first video; based on the relative position relationship, the first position information and the second position information, determining the target position information of the target object in the cropped frame image using a preset smoothing algorithm.

[0011] Optionally, when the number of target objects is one, the first key point is the designated point of the target object; when the number of target objects is multiple, the first key point is the designated point of the target object with the largest proportion among the multiple target objects, or the first key point is the designated point of the minimum enclosing box of the multiple target objects.

[0012] Optionally, the number of the second key points is two, and the rotation processing of the cropped frame image based on the second key points includes: obtaining the deflection angle information of the target object in the cropped frame image based on the angle between the line connecting the two second key points and a specified direction; and rotating the cropped frame image based on the deflection angle information.

[0013] Optionally, the number of the third key points is two, and the scaling processing of the rotated frame image based on the third key points includes: obtaining a proportion value of the target object in the rotated frame image based on the distance between the two third key points and the target side length of the rotated frame image; and scaling the rotated frame image based on the proportion value.

[0014] Optionally, the scaling processing of the rotated frame image based on the proportion value includes: when the proportion value is greater than a preset proportion threshold, determining the reduction ratio of the rotated frame image based on the proportion value, and scaling down the rotated frame image based on the reduction ratio; when the proportion value is not greater than the proportion threshold, determining the magnification ratio of the rotated frame image based on the proportion value, and magnifying the rotated frame image based on the magnification ratio.

[0015] Optionally, obtaining a second frame image for replacing the environmental frame image based on a preset transition video includes: obtaining a target video segment to which the environmental frame image belongs; wherein the target video segment is a segment in the first video that does not contain a target object; determining a transition frame image in the preset transition video for replacing the environmental frame image based on the number of frame images contained in the target video segment and the position of the environmental frame image in the target video segment; and obtaining the second frame image based on the transition frame image.

[0016] Optionally, obtaining the second frame image based on the connecting frame image includes: when the environmental frame image belongs to the first N frames of the target video clip, obtaining first specified parameter information corresponding to the previous frame image in the first video located in the target video clip, and second specified parameter information corresponding to the first frame image in the connecting video; wherein N is a preset integer value; determining processing information of the connecting frame image based on the first specified parameter information, the second specified parameter information and the position of the connecting frame image in the connecting video; and processing the connecting frame image based on the processing information to obtain the second frame image.

[0017] The embodiment of the present disclosure also provides a video processing device, including: a video acquisition module, used to acquire a first video to be processed; an object detection module, used to detect a target object in an original frame image of the first video; an operation execution module, used to perform a target operation on the original frame image based on the detection result of the target object to obtain a special effect frame image; and a video generation module, used to generate a second video based on the special effect frame image.

[0018] An embodiment of the present disclosure further provides an electronic device, comprising: a processor; a memory for storing instructions executable by the processor; the processor for reading the executable instructions from the memory and executing the instructions to implement the video processing method provided by the embodiment of the present disclosure.

[0019] An embodiment of the present disclosure further provides a computer-readable storage medium, wherein the storage medium stores a computer program, and the computer program is used to execute the video processing method provided by the embodiment of the present disclosure.

[0020] The above-mentioned technical solution provided by the embodiment of the present disclosure can detect the target object in the original frame image of the first video, and based on the detection result of the target object, perform the target operation on the original frame image to obtain a special effect frame image, thereby generating a second video on this basis.

[0021] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present disclosure, nor are they intended to limit the scope of the present disclosure. Other features of the present disclosure will become readily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0023] In order to more clearly illustrate the embodiments of the present disclosure or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0024] FIG1 is a flow chart of a video processing method provided by an embodiment of the present disclosure;

[0025] FIG2 is a flow chart of a video processing method provided by an embodiment of the present disclosure;

[0026] FIG3 is a schematic structural diagram of a video processing device provided by an embodiment of the present disclosure;

[0027] FIG4 is a schematic structural diagram of an electronic device provided by an embodiment of the present disclosure. DETAILED DESCRIPTION

[0028] Existing video special effects functions cannot yet achieve good camera movement effects and still need to be improved. In order to more clearly understand the above-mentioned purposes, features and advantages of the present disclosure, the scheme of the present disclosure will be further described below. It should be noted that, in the absence of conflict, the embodiments of the present disclosure and the features therein can be combined with each other.

[0029] In the following description, many specific details are set forth to facilitate a full understanding of the present disclosure, but the present disclosure may also be implemented in other ways different from those described herein; it is obvious that the embodiments in the specification are only part of the embodiments of the present disclosure, rather than all of the embodiments.

[0030] FIG1 is a flow chart of a video processing method provided by an embodiment of the present disclosure. The method can be executed by a video processing device, wherein the device can be implemented using software and / or hardware and can generally be integrated into an electronic device. As shown in FIG1 , the method mainly includes the following steps S102 to S108:

[0031] Step S102: Obtain a first video to be processed. The present embodiment does not limit the content of the first video. For example, any video that requires special effects processing such as camera movement can be used as the first video.

[0032] Step S104: Detect the target object in the original frame image of the first video. In practical applications, target object detection can be performed for each frame image in the first video. The present disclosure does not limit the target object. For example, the target object can be a human face, a person's entire body, an animal, or a vehicle. In practical applications, the target object can be detected using an object detection algorithm. The object detection algorithm can be implemented with reference to related technologies and will not be described in detail here.

[0033] Step S106, based on the detection result of the target object, perform a target operation on the original frame image to obtain a special effect frame image. Exemplarily, the target operation may include one or more of cropping, rotation, scaling, and replacement. In some specific implementation examples, when the target object is detected, the target operation may include cropping, rotation, and scaling; when the target object is not detected, the target operation may include replacement. In actual applications, the target object in each frame image in the first video may be detected separately to obtain a detection result of the target object for each frame image (hereinafter referred to as the object detection result), and then based on the object detection result of each frame image, the target operation is performed on each frame image to obtain a special effect frame image corresponding to each frame image.

[0034] Step S108: Generate a second video based on the special effect frame images. The special effect frame images corresponding to the original frame images in the first video can be combined in sequence to obtain the second video, which is also the special effect video generated based on the first video.

[0035] Since the above method processes the frame images in the first video based on the object detection results, it can effectively ensure the rationality of the target object presentation method in the second video, and thus help to better achieve the camera movement effects of following the target object on this basis. For example, based on the detection results of the target object, operations such as cropping can be performed according to the position of the target object, thereby achieving the camera movement operation of following the target object.

[0036] In some embodiments, the above step S106, i.e., the step of performing the target operation on the original frame image based on the detection result of the target object to obtain the special effect frame image, can be performed with reference to the following steps 1 to 4:

[0037] Step 1: Based on the target object detection results, the original frame images are divided into object frame images and environment frame images; the object frame images include the target object, while the environment frame images do not. That is, based on the target object detection results, the original frame images in the first video can be divided into two categories: object frame images and environment frame images. Exemplarily, for each original frame image in the first video, if the target object is detected in the frame image, the frame image is considered the object frame image; if the target object is not detected in the frame image, the frame image is considered the environment frame image.

[0038] Step 2: Obtain target key points from the object frame image, and perform a target operation on the object frame image based on the target key points to obtain a first frame image. The disclosed embodiments do not limit the method for setting the target key points. For example, the target key points can be one or more key points of the target object, and the corresponding key points can be the same or different depending on the processing method adopted for the target operation, and the specific setting can be flexible.

[0039] Step 3: Based on the preset transition video, a second frame image is obtained to replace the environment frame image. The transition video can be a preset animation video, and the content of the transition video can be flexibly set.

[0040] Step 4: Obtain a special effect frame image based on the first frame image and the second frame image. For example, the first frame image and the second frame image can both be directly used as special effect frame images.

[0041] Through the above steps 1 to 4, the original frame image can be processed in a targeted manner based on the object detection results of the original frame image. Not only can reasonable target operations be performed on the object frame image based on the target key points, but the environment frame image can also be replaced to obtain a special effect frame image with a strong visual effect.

[0042] In some specific implementation examples, the target key points include a first key point, a second key point, and a third key point; the step of performing a target operation on the object frame image based on the target key points to obtain the first frame image in the above step 2 can be performed with reference to the following steps A to C:

[0043] Step A: cropping the object frame image based on the first key point to obtain a cropped frame image.

[0044] Exemplarily, when there are multiple target objects, the first key point is the designated point of the target object with the largest proportion among the multiple target objects, or the first key point is the designated point of the minimum bounding box of the multiple target objects. For example, the above-mentioned designated point can be a center point or a specified key point, which can be flexibly set. The embodiment of the present disclosure fully takes into account that one or more target objects may appear in the target frame image. When there are multiple target objects, the target object with the largest proportion (such as the largest face among multiple faces) can be selected, and the multiple target objects can also be analyzed as a whole, which can be flexibly set according to needs. The above method can better process the boundaries of complex scenes such as changes in the number of target objects, thereby ensuring the video processing effect.

[0045] Through the above-mentioned cropping process, the picture layout can be better adjusted and the position of the target object in the picture can be effectively adjusted, such as showing that the target object is continuously located at the center of the lens, thereby achieving the effect of following the target object. Even if the lens movement is large, the cropping process based on the position information of the target object can still make the target object located at the center of the cropped frame image or other lens lock positions, presenting a large-scale lens following effect of the target object. The cropped frame image mostly presents a partial picture in the original frame image, and the partial picture usually contains the target object. The size of the cropped image can then be adjusted to be consistent with the size of the target frame image. At this time, the user can also be presented with a visual perception of an enlarged partial picture, which has a stronger visual impact.

[0046] In some implementation examples, step A may be performed with reference to the following steps A1 to A3:

[0047] Step A1: Obtain a relative position relationship between a first key point and a center point of an object frame image. The relative position relationship may indicate a relative direction and distance between the first key point and the center point of the object frame image.

[0048] Step A2: Determine the target position information of the target object in the cropped frame image based on the relative position relationship. The first key point can be a designated key point such as the center point of the target object. The cropped frame image is the image obtained after cropping. In the disclosed embodiment, to effectively achieve a lens lock effect on the target object, the position of the first key point of the target object in the cropped frame image can be first determined based on the relative position relationship between the first key point of the target object and the center point of the object frame image, thereby obtaining the position information of the target object in the cropped frame image. For example, if the target object is a face and the first key point is the center point of the face, the relative position relationship between the center point of the face and the center point of the object frame image can be determined based on the relative position relationship between the center point of the face and the center point of the object frame image. For example, the two can be made consistent to obtain the position of the cropped frame image. Regardless of how the face moves, the above method can also ensure that the face remains at the center of the frame, such as the lens lock position, achieving a face tracking camera movement effect, and is not limited to the camera movement amplitude.

[0049] In some specific implementation examples, step A2 may be performed with reference to the following steps A2.1 to A2.2:

[0050] Step A2.1: If the target frame image is not the first frame image, obtain first position information of the target object in the target frame image, and obtain second position information of the target object in a frame image in the first video that precedes the target frame image. The frame images in the first video that precede the target frame image are historical frame images. The number of historical frame images can be flexibly set, such as setting the historical frame images to be the first M frames before the target frame image.

[0051] Step A2.2: Based on the relative position relationship, the first position information, and the second position information, a preset smoothing algorithm is used to determine the target position information of the target object in the cropped frame image.

[0052] The initial position information of the target object in the cropped frame image can be first determined based on the first position information and the relative position relationship. Then, combined with the second position information of the target object in the historical frame image, a preset smoothing algorithm is used to optimize the initial position information, and the optimized position information is used as the target position information. This method can further improve the accuracy and rationality of the target object's target position information in the cropped frame image by combining the target object's historical position and the current relative position relationship. For example, a real-time smoothing algorithm can be used to smooth and filter the position trajectory of a face in multiple frames, thereby making the position changes of the target object in the final video more stable and smooth, and presenting a more natural visual experience to the user.

[0053] Step A3: cropping the target frame image based on the target position information. In some embodiments, the target frame image is cropped to ensure that the position of the target object in the cropped image (ie, the cropped frame image) conforms to the target position information.

[0054] In practical applications, preset cropping adjustment parameters can also be obtained; based on the cropping adjustment parameters and target position information, the image to be cropped is cropped. The cropping adjustment parameters can be used to indicate the cropping selection range of the canvas. In practical applications, the system can preset default cropping adjustment parameters, or the user can be provided with parameter setting controls such as sliders to allow the user to set the cropping adjustment parameters. In this manner, cropping can be performed in conjunction with the cropping condition parameters and target position information, ensuring that the target object is located at a specified position in the cropped image while satisfying the canvas cropping selection range indicated by the cropping adjustment parameters.

[0055] In step B, the cropped frame image is rotated based on the second key point to obtain a rotated frame image. In this way, a rotating dynamic visual effect can be presented to the user. In some implementation examples, the number of second key points is two, and step B can be performed with reference to the following steps B1 to B2:

[0056] Step B1, based on the angle between the line connecting the two second key points and the specified direction, obtain the deflection angle information of the target object in the cropped frame image. The second key point may be the same as or different from the aforementioned first key point. For example, if the target object is a face, the two second key points may be the left eye key point and the right eye key point. If the target object is a vehicle, the two second key points may be the key points of the left and right headlights. The deflection angle information of the target object can be simply and accurately represented by the angle between the line connecting the two second key points and the specified direction (such as the horizontal direction). The specified direction may be the horizontal direction or the vertical direction, etc., which can be flexibly set. The above is only an exemplary description. In actual applications, other methods can also be used to measure the deflection angle of the target object, which is not limited here.

[0057] In step B2, the cropped frame image is rotated based on the deflection angle information. The rotation angle is positively correlated with the angle indicated by the deflection angle information. This method allows the image to rotate in accordance with the deflection of the target object. For example, the image can rotate as a face rotates, creating a strong visual impact for the user.

[0058] In some specific implementation examples, when step B2 is executed, the following steps may be referred to: obtaining preset rotation adjustment parameters; rotating the cropped frame image based on the rotation adjustment parameters and the deflection angle information. In actual applications, the system may preset default rotation adjustment parameters, or provide the user with parameter setting controls such as sliders, and the user may set the rotation adjustment parameters. In actual applications, the user may be provided with a rotation control for setting the rotation adjustment parameters alone, or an intensity control for setting the overall effect intensity may be provided to the user. The user may change the rotation adjustment parameters individually based on the rotation control, or may change the rotation adjustment parameters or other parameters that affect the overall effect, such as the zoom adjustment parameters, as a whole, through the intensity control. This is not limited here. In actual applications, the image rotation angle may be determined based solely on the deflection angle information of the target object with reference to the aforementioned relevant content, or the rotation adjustment parameters may be fully considered, such as using the parameter as a coefficient of the rotation angle, so as to adjust the amplitude of the rotation angle determined based on the deflection angle information of the target object to ensure that the overall rotation amplitude meets the user's needs.

[0059] In addition, in order to further enhance the rotation effect, a blur parameter can be determined based on the deflection angle information; the image obtained by the rotation processing is blurred based on the blur parameter, and the blurred image is used as a rotated frame image. It can be understood that the degree of blur is related to the image rotation angle, and the image rotation angle depends on the deflection angle of the target object. Therefore, the blur parameter can be positively correlated with the angle indicated by the deflection angle information, such as the greater the deflection angle, the greater the degree of blur. In some specific examples, the blur processing can be radial blur processing, and the blur parameter can be a radial blur parameter. In this way, the image can be rotated following the deflection of the target object and present a radial blur effect. The degree of blur varies with the deflection angle (rotation angle), presenting a realistic dynamic rotation perception to the user.

[0060] Step C: Scaling the rotated frame image based on the third key point to obtain the first frame image. This scaling process can effectively adjust the size of the target object in the first frame image, presenting the user with the desired visual effect. In some implementation examples, the number of third key points is two, and Step C can be performed with reference to Steps C1 to C2 as follows:

[0061] Step C1: obtaining a proportion value of the target object in the rotated frame image based on the distance between the two third key points and the target side length of the rotated frame image.

[0062] The third key point can be the same as or different from the aforementioned second key point. For example, if the target object is a face, the two third key points can also be the left eye key point and the right eye key point. If the target object is a vehicle, the two third key points can be the key points of the left and right headlights. By comparing the distance between the two third key points with the target side length of the rotating frame image, the proportion of the target object in the rotating frame image can be determined simply and reliably. For example, the above distance and target side length can be normalized, and then the ratio between the normalized distance and the target side length is used to determine the proportion value. The larger the ratio, the larger the proportion value. The above target side length can be the long side or the short side in the target frame image, which can be set flexibly. The above is only an exemplary explanation. In actual applications, the ratio between the area of ​​the target object and the area of ​​the target frame image can also be used to represent the proportion value, which is not limited here.

[0063] Step C2: scaling the rotated frame image based on the ratio value.

[0064] It can be understood that the proportion value can be used to characterize the size of the target object in the target frame image. In order to ensure the visual effect, the embodiment of the present disclosure can scale the image based on the proportion value, thereby adjusting the size of the target object in the frame image, and presenting the desired visual effect to the user, such as presenting a visual effect that the size of the target object in the frame image is basically stable.

[0065] In some specific embodiments, step C2 can be performed as follows: when the proportion value is greater than a preset proportion threshold, the reduction ratio of the image obtained after rotation is determined based on the proportion value, and the rotated frame image is reduced based on the reduction ratio; when the proportion value is not greater than the proportion threshold, the magnification ratio of the rotated frame image is determined based on the proportion value, and the rotated frame image is magnified based on the magnification ratio. That is, if the target object occupies a large proportion of the screen, a reduction process can be performed to reduce the screen proportion of the target object; if the target object occupies a small proportion of the screen, a magnification process can be performed to enlarge the screen proportion of the target object. Among them, the mapping relationship between the proportion value and the reduction ratio and the mapping relationship between the proportion value and the magnification ratio can be pre-set, such as, the larger the proportion value, the larger the reduction ratio, and the smaller the proportion value, the larger the magnification ratio. In the above manner, the proportion of the target object in the screen can be kept as unchanged as possible, presenting a more stable visual experience to the user.

[0066] In addition, the above-mentioned step of scaling the image to be scaled based on the proportion value can also be performed with reference to the following steps: obtaining a preset scaling adjustment parameter; scaling the rotated frame image based on the scaling adjustment parameter and the proportion information. The above-mentioned scaling adjustment parameter can be used to indicate the image scaling amplitude. In actual applications, the system can preset a default scaling adjustment parameter, or provide the user with a parameter setting control such as a slider, and the user sets the scaling adjustment parameter. In actual applications, the user can be provided with a scaling control for setting the scaling adjustment parameter separately, or an intensity control for setting the overall effect intensity can be provided to the user. The user can change the scaling adjustment parameter separately based on the scaling control, or change the scaling adjustment parameter or other parameters that affect the overall effect such as the rotation adjustment parameter as a whole through the intensity control. There is no limitation here. In actual applications, the scaling ratio can be determined based on the proportion value and the proportion threshold only with reference to the above-mentioned relevant content, or the scaling adjustment parameter can be fully considered, such as using the parameter as a coefficient of the scaling ratio, so as to adjust the amplitude of the scaling ratio determined based on the proportion value and the proportion threshold to ensure that the overall scaling amplitude meets the user's needs.

[0067] In some embodiments, the above step three, i.e., the step of obtaining a second frame image for replacing the environment frame image based on the preset transition video, can be performed with reference to the following steps a to c:

[0068] Step a: Obtain the target video segment to which the environmental frame image belongs. The target video segment is the segment of the first video that does not contain the target object. For example, assuming the first video has a total of 100 frames, frames 1 to 30 contain the target object, frames 31 to 50 do not, and frames 51 to 100 contain the target object, then the target video segment is the video segment consisting of frames 31 to 50.

[0069] Step b: Determine the transition frame image in the preset transition video to replace the environment frame image based on the number of frame images included in the target video segment and the position of the environment frame image in the target video segment. For example, the total number of all transition frame images required in the transition video can be determined based on the number of frame images included in the target video segment. Then, based on the position of the environment frame image in the target video segment, the transition frame image used to replace the environment frame image can be determined from all the transition frame images required in the transition video. In some simple methods, the replacement can be carried out in sequence. For example, if the environment frame image is located in the nth frame of the target video segment, the nth frame in the transition video is used as the transition frame image to replace the environment frame image. In other embodiments, the appropriate transition frame image can be selected from the transition video for replacement based on the front, middle, or back position of the environment frame image in the target video segment. The specific setting can be flexible and is not limited here.

[0070] Step c: Obtain a second frame image based on the connecting frame image. In some embodiments, the connecting frame image can be directly used as the second frame image. In other embodiments, to make the video transition more natural, the connecting frame image can be further processed and the processed connecting frame image can be used as the second frame image. In some specific implementation examples, step c can be performed with reference to the following steps c1 to c3:

[0071] In step c1, when the environment frame image is N frames before the target video segment, first specified parameter information corresponding to the image frame preceding the target video segment in the first video and second specified parameter information corresponding to the first image frame in the transition video are obtained; wherein N is a preset integer value. The specific value of N can be flexibly set. The specified parameter information may include information such as a rotation angle and a scaling ratio.

[0072] Step c2: Determine processing information for the transition frame image based on the first specified parameter information, the second specified parameter information, and the position of the transition frame image in the transition video. For example, if the first video has a total of 100 frames, the 1st to 30th frames have a target object, the 31st to 50th frames do not have a target object, and the 51st to 100th frames have a target object, then the target video segment is the 31st to 50th frames. The first 20 frames of the transition video can be selected to sequentially replace the frame images in the target video segment. Assuming that the specified parameter value of the 30th frame image (corresponding to the first specified parameter information) is a, and the specified parameter value of the first frame image of the transition video (corresponding to the second specified parameter information) is b, then the specified parameters of the first N frames (assuming the first 5 frames) of the transition video can be transitionally adjusted from a to b, thereby achieving a good transition connection.

[0073] Step c3 processes the connecting frame image based on the processing information to obtain a second frame image. The processing information may include, for example, scaling adjustment information and rotation adjustment information. Based on this processing information, the connecting frame image can be scaled and rotated so that the processed connecting frame image can more naturally replace the surrounding frame image. In other words, based on the first video, the connecting video can more naturally replace the target video segment that does not contain the target object.

[0074] Furthermore, the video processing method provided by the embodiment of the present disclosure further includes: performing smoothing based on the second video. The embodiment of the present disclosure does not limit the smoothing method. Through the above method, the video can be further smoothed to ensure the smooth viewing experience of the second video.

[0075] Based on the above, referring to the flowchart of a video processing method shown in FIG2 , the method mainly includes the following steps:

[0076] Step S202: Obtain a first video to be processed.

[0077] Step S204: Detect the target object in the original frame image of the first video.

[0078] Step S206: Based on the detection result of the target object, the original frame image is divided into an object frame image and an environment frame image, wherein the target object is included in the object frame image and the target object is not detected in the environment frame image.

[0079] In step S208, the target key points in the target frame image are obtained, and the target frame image is sequentially cropped, rotated, and scaled based on the target key points to obtain a first frame image. This sequential processing method allows for efficient and effective image processing. The resulting first frame image not only achieves a camera movement effect that follows the target object, but also presents the user with a dynamic rotation effect of the target object, effectively ensuring the target object's proportion in the frame, and presenting the user with a stable, large-scale camera movement experience.

[0080] Step S210: Based on the preset connection video, a second frame image is obtained to replace the environment frame image. Through this processing method, the second frame image can replace the environment frame image, thereby presenting a better connection feeling to the user.

[0081] Step S212: obtaining a special effect frame image based on the first frame image and the second frame image.

[0082] Step S214: Generate a second video based on the special effect frame image.

[0083] Step S216: Smoothing the second video to obtain a target video.

[0084] Through the above-mentioned method provided by the embodiment of the present disclosure, a better object-following camera movement effect can be achieved, which not only helps to achieve a smoother large-scale camera movement effect, but also can incorporate more reasonable and realistic zooming, rotation and other special effects, with a strong visual impact, and can provide users with a better viewing experience of camera movement special effects.

[0085] Corresponding to the aforementioned video processing method, FIG3 is a schematic structural diagram of a video processing device provided by an embodiment of the present disclosure. The device can be implemented by software and / or hardware and can generally be integrated into an electronic device. As shown in FIG3 , the video processing device includes:

[0086] The video acquisition module 302 is used to acquire a first video to be processed;

[0087] The object detection module 304 is configured to detect a target object in the original frame image of the first video;

[0088] An operation execution module 306 is configured to execute a target operation on the original frame image based on the detection result of the target object to obtain a special effect frame image;

[0089] The video generation module 308 is configured to generate a second video based on the special effect frame image.

[0090] Since the above-mentioned device processes the frame images in the first video based on the object detection results, it can effectively ensure the rationality of the target object presentation method in the obtained second video, and thus on this basis also helps to better achieve the camera movement effect of following the target object.

[0091] In some embodiments, the operation execution module 306 is specifically used to: divide the original frame image into an object frame image and an environment frame image based on the detection result of the target object; wherein the target object is included in the object frame image, and the target object is not detected in the environment frame image; obtain the target key points in the object frame image, and perform the target operation on the object frame image based on the target key points to obtain a first frame image; based on a preset connection video, obtain a second frame image for replacing the environment frame image; and obtain a special effect frame image based on the first frame image and the second frame image.

[0092] In some embodiments, the target key points include a first key point, a second key point, and a third key point; the operation execution module 306 is specifically used to: crop the object frame image based on the first key point to obtain a cropped frame image; rotate the cropped frame image based on the second key point to obtain a rotated frame image; and scale the rotated frame image based on the third key point to obtain a first frame image.

[0093] In some embodiments, the operation execution module 306 is specifically used to: obtain the relative position relationship between the first key point and the center point of the object frame image; determine the target position information of the target object in the cropped frame image based on the relative position relationship; and crop the object frame image based on the target position information.

[0094] In some embodiments, the operation execution module 306 is specifically used to: when the object frame image is not the first frame image, obtain the first position information of the target object in the object frame image, and obtain the second position information of the target object in the frame image that is located before the object frame image in the first video; based on the relative position relationship, the first position information and the second position information, use a preset smoothing algorithm to determine the target position information of the target object in the cropped frame image.

[0095] In some embodiments, when the number of the target object is one, the first key point is the designated point of the target object; when the number of the target objects is multiple, the first key point is the designated point of the target object with the largest proportion among the multiple target objects, or the first key point is the designated point of the minimum enclosing box of the multiple target objects.

[0096] In some embodiments, the number of the second key points is two, and the operation execution module 306 is specifically used to: obtain the deflection angle information of the target object in the cropped frame image based on the angle between the line connecting the two second key points and the specified direction; and rotate the cropped frame image based on the deflection angle information.

[0097] In some embodiments, the number of the third key points is two, and the operation execution module 306 is specifically used to: obtain the proportion value of the target object in the rotated frame image based on the distance between the two third key points and the target side length of the rotated frame image; and scale the rotated frame image based on the proportion value.

[0098] In some embodiments, the operation execution module 306 is specifically used to: when the proportion value is greater than a preset proportion threshold, determine the reduction ratio of the image obtained after rotation based on the proportion value, and reduce the rotated frame image based on the reduction ratio; when the proportion value is not greater than the proportion threshold, determine the enlargement ratio of the rotated frame image based on the proportion value, and enlarge the rotated frame image based on the enlargement ratio.

[0099] In some embodiments, the operation execution module 306 is specifically used to: obtain the target video segment to which the environmental frame image belongs; wherein the target video segment is a segment in the first video that does not contain the target object; based on the number of frame images contained in the target video segment and the position of the environmental frame image in the target video segment, determine the connection frame image in the preset connection video for replacing the environmental frame image; and obtain a second frame image based on the connection frame image.

[0100] In some embodiments, the video generation module 308 is specifically used to: when the environmental frame image belongs to the first N frames of the target video segment, obtain the first specified parameter information corresponding to the previous frame image in the first video located in the target video segment, and the second specified parameter information corresponding to the first frame image in the connecting video; wherein N is a preset integer value; based on the first specified parameter information, the second specified parameter information and the position of the connecting frame image in the connecting video, determine the processing information of the connecting frame image; and process the connecting frame image based on the processing information to obtain the second frame image.

[0101] The video processing device provided in the embodiments of the present disclosure can execute the video processing method provided in any embodiment of the present disclosure, and has the corresponding functional modules and beneficial effects of the execution method.

[0102] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the above-described device embodiment can refer to the corresponding process in the method embodiment, and will not be repeated here.

[0103] An embodiment of the present disclosure provides an electronic device, which includes: a storage device storing a computer program; and a processing device configured to execute the computer program in the storage device to implement the steps of any one of the methods in the present disclosure.

[0104] Reference is now made to FIG4 , which illustrates a schematic diagram of the structure of an electronic device 400 suitable for implementing embodiments of the present disclosure. Terminal devices in embodiments of the present disclosure may include, but are not limited to, mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. The electronic device illustrated in FIG4 is merely an example and should not limit the functionality or scope of use of embodiments of the present disclosure.

[0105] As shown in Figure 4, electronic device 400 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 401, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 402 or a program loaded from a storage device 408 into a random access memory (RAM) 403. Various programs and data required for the operation of electronic device 400 are also stored in RAM 403. Processing device 401, ROM 402, and RAM 403 are connected to each other via a bus 404. An input / output (I / O) interface 405 is also connected to bus 404.

[0106] Typically, the following devices may be connected to the I / O interface 405: an input device 406 including, for example, a touch screen, a touchpad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 407 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 408 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 409. The communication device 409 may allow the electronic device 400 to communicate with other devices wirelessly or by wire to exchange data. Although FIG4 shows the electronic device 400 with various devices, it should be understood that not all of the devices shown are required to be implemented or present. More or fewer devices may alternatively be implemented or present.

[0107] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication device 409, or installed from the storage device 408, or installed from the ROM 402. When the computer program is executed by the processing device 401, the above-mentioned functions defined in the method of the embodiment of the present disclosure are performed.

[0108] In addition to the above-mentioned methods and devices, the embodiments of the present disclosure may also be a computer program product, which includes computer program instructions, which, when executed by a processor, cause the processor to perform the image processing method provided by the embodiments of the present disclosure. The computer program product may be written in any combination of one or more programming languages ​​to write program codes for performing the operations of the embodiments of the present disclosure, the programming languages ​​including object-oriented programming languages ​​such as Java, C++, etc., and also conventional procedural programming languages ​​such as "C" language or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0109] In addition, the embodiment of the present disclosure may also be a computer-readable storage medium having computer program instructions stored thereon. When the computer program instructions are executed by a processor, the processor is enabled to execute the video processing method provided by the embodiment of the present disclosure.

[0110] The computer-readable storage medium can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can, for example, include but is not limited to a system, device or component of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination thereof. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof.

[0111] The embodiments of the present disclosure further provide a computer program product, including a computer program / instruction, which implements the video processing method in the embodiments of the present disclosure when executed by a processor.

[0112] It is understandable that before using the technical solutions disclosed in the various embodiments of this disclosure, the type, scope of use, usage scenarios, etc. of the personal information involved in this disclosure should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.

[0113] For example, in response to a user's active request, a prompt message is sent to the user to clearly inform the user that the operation requested will require the acquisition and use of the user's personal information. This allows the user to independently choose whether to provide personal information to the electronic device, application, server, storage medium, or other software or hardware that performs the operations of the disclosed technical solution based on the prompt message.

[0114] As an optional but non-limiting implementation, in response to receiving a user's active request, the prompt information may be sent to the user in the form of a pop-up window, in which the prompt information may be presented in text form. Furthermore, the pop-up window may also contain a selection control for the user to select "agree" or "disagree" to provide personal information to the electronic device.

[0115] It is understandable that the above notification and user authorization process are merely illustrative and do not constitute a limitation on the implementation of the present disclosure. Other methods that comply with relevant laws and regulations may also be applied to the implementation of the present disclosure.

[0116] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or device comprising the element.

[0117] The foregoing description is intended only to provide specific embodiments of the present disclosure, intended to enable those skilled in the art to understand and implement the present disclosure. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present disclosure. Therefore, the present disclosure is not intended to be limited to the embodiments described herein, but rather to be construed in the broadest manner consistent with the principles and novel features disclosed herein.

Claims

1. A video processing method, comprising: Get the first video to be processed; Detecting a target object in an original frame image of the first video; Based on the detection result of the target object, performing a target operation on the original frame image to obtain a special effect frame image; A second video is generated based on the special effect frame image.

2. The method according to claim 1, wherein performing a target operation on the original frame image based on the detection result of the target object to obtain a special effect frame image comprises: Based on the detection result of the target object, the original frame image is divided into an object frame image and an environment frame image; wherein the target object is included in the object frame image, and the target object is not detected in the environment frame image; Acquire target key points in the object frame image, and perform a target operation on the object frame image based on the target key points to obtain a first frame image; Based on the preset connection video, a second frame image is obtained to replace the environment frame image; A special effect frame image is obtained based on the first frame image and the second frame image.

3. The method according to claim 2, wherein the target key points include a first key point, a second key point, and a third key point; and performing a target operation on the object frame image based on the target key points to obtain the first frame image comprises: Performing cropping processing on the object frame image based on the first key point to obtain a cropped frame image; performing a rotation process on the cropped frame image based on the second key point to obtain a rotated frame image; The rotated frame image is scaled based on the third key point to obtain a first frame image.

4. The method according to claim 3, wherein the cropping process of the object frame image based on the first key point comprises: Acquire a relative positional relationship between the first key point and a center point of the object frame image; Based on the relative position relationship, determining target position information of the target object in the cropped frame image; The object frame image is cropped based on the target position information.

5. The method according to claim 4, wherein determining the target position information of the target object in the cropped frame image based on the relative position relationship comprises: When the target frame image is not the first frame image, obtaining first position information of the target object in the target frame image, and obtaining second position information of the target object in a frame image preceding the target frame image in the first video; Based on the relative position relationship, the first position information and the second position information, a preset smoothing algorithm is used to determine the target position information of the target object in the cropped frame image.

6. The method according to claim 3, wherein when the number of the target object is one, the first key point is a designated point of the target object; In the case where there are multiple target objects, the first key point is a designated point of a target object with the largest proportion among the multiple target objects, or the first key point is a designated point of a minimum bounding box of the multiple target objects.

7. The method according to claim 3, wherein the number of the second key points is two, and the rotating the cropped frame image based on the second key points comprises: Obtaining deflection angle information of the target object in the cropped frame image based on an angle between a line connecting two of the second key points and a specified direction; The cropped frame image is rotated based on the deflection angle information.

8. The method according to claim 3, wherein the number of the third key points is two, and the scaling processing of the rotated frame image based on the third key points comprises: Based on the distance between the two third key points and the target side length of the rotated frame image, obtaining a proportion value of the target object in the rotated frame image; The rotated frame image is scaled based on the ratio value.

9. The method according to claim 8, wherein scaling the rotated frame image based on the ratio value comprises: When the proportion value is greater than a preset proportion threshold, determining a reduction ratio of the rotated frame image based on the proportion value, and performing a reduction process on the rotated frame image based on the reduction ratio; When the proportion value is not greater than the proportion threshold, the magnification ratio of the rotated frame image is determined based on the proportion value, and the rotated frame image is magnified based on the magnification ratio.

10. The method according to claim 2, wherein the acquiring of a second frame image for replacing the environment frame image based on a preset transition video comprises: Acquire a target video segment to which the environment frame image belongs; wherein the target video segment is a segment of the first video that does not contain a target object; Determining, based on the number of frame images included in the target video segment and the position of the environment frame image in the target video segment, a connecting frame image in a preset connecting video for replacing the environment frame image; A second frame image is obtained based on the connected frame image.

11. The method according to claim 10, wherein obtaining the second frame image based on the connected frame image comprises: When the environment frame image belongs to the first N frames of the target video segment, obtaining first specified parameter information corresponding to the frame image in the first video that is located before the target video segment, and second specified parameter information corresponding to the first frame image in the transition video; wherein N is a preset integer value; determining processing information of the connecting frame image based on the first specified parameter information, the second specified parameter information, and a position of the connecting frame image in the connecting video; The connected frame image is processed based on the processing information to obtain a second frame image.

12. A video processing device, comprising: A video acquisition module, configured to acquire a first video to be processed; An object detection module, configured to detect a target object in an original frame image of the first video; an operation execution module, configured to execute a target operation on the original frame image based on the detection result of the target object to obtain a special effect frame image; A video generation module is used to generate a second video based on the special effect frame image.

13. An electronic device, comprising: a storage device having a computer program stored thereon; A processing device, configured to execute the computer program in the storage device to implement the steps of the video processing method according to any one of claims 1 to 11. 14 . A computer-readable storage medium storing a computer program, wherein the computer program is configured to execute the video processing method according to claim 1 .

Images