Video processing method and apparatus, electronic device, and storage medium

Abstract

The present disclosure relates to a video processing method and apparatus, an electronic device, and a storage medium. The method comprises: acquiring an original video and a video speed change parameter corresponding to the original video; on the basis of the video speed change parameter, determining a first playback time period and a second playback time period, wherein the first playback time period comprises a first frame extraction moment, the second playback time period comprises a second frame extraction moment, the playback speed during the first playback time period is the same as the playback speed of the original video, and the playback speed during the second playback time period is slower than the playback speed of the original video; on the basis of the original video, determining a first image frame corresponding to the first frame extraction moment and a second image frame corresponding to the second frame extraction moment; and on the basis of the first image frame corresponding to the first frame extraction moment and the second image frame corresponding to the second frame extraction moment, obtaining a target video.

Description

Video processing methods, apparatus, electronic devices and storage media

[0001] Cross-reference to related applications

[0002] This application claims priority to Chinese Patent Application No. 202410505870.8, filed on April 25, 2024, entitled "Video Processing Method, Apparatus, Electronic Device and Storage Medium", the entire contents of which are incorporated herein by reference. Technical Field

[0003] This disclosure relates to the field of video processing technology, and in particular to a video processing method, apparatus, electronic device, and storage medium. Background Technology

[0004] With the rapid development of digital video technology, people have placed higher demands on the visual effects of videos. Adding special effects to videos has become an important technical means to enhance their viewing experience and artistry. Special effects can bring richer visual effects to videos, making them more attractive and valuable for viewing. Summary of the Invention

[0005] This disclosure provides a video processing method, apparatus, electronic device, and storage medium.

[0006] Firstly, this disclosure provides a video processing method, including:

[0007] Obtain the original video and the corresponding video speed parameters;

[0008] Based on the video speed adjustment parameters, a first playback period and a second playback period are determined; the first playback period includes a first frame skipping moment, and the second playback period includes a second frame skipping moment; the playback speed of the first playback period is the same as the playback speed of the original video; the playback speed of the second playback period is less than the playback speed of the original video.

[0009] Based on the original video, a first image frame corresponding to the first frame extraction time and a second image frame corresponding to the second frame extraction time are determined.

[0010] The target video is obtained based on the first image frame corresponding to the first frame extraction time and the second image frame corresponding to the second frame extraction time.

[0011] Secondly, this disclosure also provides a video processing apparatus, comprising:

[0012] The acquisition module is used to acquire the original video and the video speed adjustment parameters corresponding to the original video;

[0013] A first determining module is used to determine a first playback period and a second playback period based on the video speed adjustment parameters; the first playback period includes a first frame-dropping moment, and the second playback period includes a second frame-dropping moment; the playback speed of the first playback period is the same as the playback speed of the original video; the playback speed of the second playback period is less than the playback speed of the original video.

[0014] The second determining module is used to determine, based on the original video, a first image frame corresponding to the first frame extraction time and a second image frame corresponding to the second frame extraction time;

[0015] The stitching module is used to obtain the target video based on the first image frame corresponding to the first frame extraction time and the second image frame corresponding to the second frame extraction time.

[0016] Thirdly, this disclosure also provides an electronic device, the electronic device comprising:

[0017] One or more processors;

[0018] Storage device for storing one or more programs;

[0019] When the one or more programs are executed by the one or more processors, the one or more processors implement the video processing method as described above.

[0020] Fourthly, this disclosure also provides a computer-readable storage medium having a computer program stored thereon that, when executed by a processor, implements the video processing method described above.

[0021] The technical solution provided in this disclosure involves setting and acquiring the original video and the corresponding video speed parameters; determining a first playback period and a second playback period based on the video speed parameters; the first playback period includes a first frame-dropping moment, and the second playback period includes a second frame-dropping moment; the playback speed of the first playback period is the same as the playback speed of the original video; the playback speed of the second playback period is less than the playback speed of the original video; determining a first image frame corresponding to the first frame-dropping moment and a second image frame corresponding to the second frame-dropping moment based on the original video; and obtaining the target video based on the first image frame corresponding to the first frame-dropping moment and the second image frame corresponding to the second frame-dropping moment. Attached Figure Description

[0022] While some software and applications on the market offer the function of adding special effects, the variety of effects they provide is limited, making it difficult to meet the diverse needs of users. The embodiments disclosed herein can solve the above-mentioned technical problems, or at least partially solve them.

[0023] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0024] To more clearly illustrate the technical solutions in the embodiments of this disclosure or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, those skilled in the art can obtain other drawings based on these drawings without creative effort.

[0025] Figure 1 is a flowchart of a video processing method provided in an embodiment of this disclosure;

[0026] Figure 2 is a schematic diagram of a target timeline of a target video provided in an embodiment of this disclosure;

[0027] Figure 3 is a schematic diagram of a video processing method provided in an embodiment of this disclosure;

[0028] Figure 4 is a schematic diagram of a method for processing two third candidate image frames using optical flow to obtain an interpolated image, according to an embodiment of this disclosure.

[0029] Figure 5 is a schematic diagram of the structure of a video processing device according to an embodiment of the present disclosure;

[0030] Figure 6 is a schematic diagram of the structure of an electronic device according to an embodiment of this disclosure. Detailed Implementation

[0031] To better understand the above-mentioned objectives, features, and advantages of this disclosure, the solutions disclosed herein will be further described below. It should be noted that, unless otherwise specified, the embodiments and features described herein can be combined with each other.

[0032] Numerous specific details are set forth in the following description in order to provide a full understanding of this disclosure, but this disclosure may also be implemented in other ways different from those described herein; obviously, the embodiments in the specification are only some, and not all, of the embodiments of this disclosure.

[0033] Figure 1 is a flowchart of a video processing method provided in an embodiment of this disclosure. This embodiment is applicable to adding special effects to a video on a client side. The method can be executed by a video processing device, which can be implemented in software and / or hardware. The device can be configured in an electronic device, such as a terminal, including but not limited to smartphones, PDAs, tablets, wearable devices with displays, desktop computers, laptops, all-in-one computers, smart home devices, etc. Alternatively, this embodiment can be applied to adding special effects to a video on a server side. The method can be executed by a video processing device, which can be implemented in software and / or hardware. The device can be configured in an electronic device, such as a server.

[0034] As shown in Figure 1, the method may specifically include:

[0035] S110, Obtain the original video and the corresponding video speed adjustment parameters.

[0036] The original video could be, for example, a video to which a speed-changing effect needs to be added. The resulting video after adding the speed-changing effect to the original video will be used as the target video.

[0037] During the original video playback, the playback speed of the image frames is constant, referred to as the standard speed in this application. However, during the playback of the target video, the playback speed of the image frames changes continuously; specifically, image frames are played at the standard speed for a period of time, and at a slower speed for another period of time. In this application, "slow speed" refers to a playback speed less than the standard speed.

[0038] For example, referring to Figure 2, the target timeline corresponds to a target video (i.e., a video with a speed-changing effect added), describing the changes in the playback speed of the target video during its playback. The target timeline includes multiple first playback periods and multiple second playback periods, which are arranged alternately. When the target video plays to the first playback period, image frames are played at standard speed; when it plays to the second playback period, image frames are played at slow speed.

[0039] Video speed-changing parameters can be parameters used in the process of speeding up the original video, and their values ​​determine the visual performance of the video after speed-changing. This application does not limit the specific parameters included in the video speed-changing parameters. For example, video speed-changing parameters may include information describing the playback speed of image frames in different time periods, information describing the duration of the time period corresponding to the playback speed of a certain image frame, and information describing the arrangement of the time periods corresponding to different image frame playback speeds on the time axis.

[0040] For example, information describing the playback speed of image frames in different time periods may include: the playback speed of image frames in the second playback period, the playback speed of image frames in the first playback period, or the ratio of the playback speed of image frames in the second playback period to the playback speed of image frames in the first playback period, etc.

[0041] Information about the duration of a time period corresponding to the playback speed of a certain image frame may include: the speed change period, the proportion of the first playback period in the speed change period, the proportion of the second playback period in the speed change period, and the ratio of the duration of the first playback period to the duration of the second playback period.

[0042] The arrangement of time periods corresponding to different image frame playback speeds on the time axis can include: the number of first playback periods, the number of second playback periods, the arrangement of the first and second playback periods, the duration of the first playback period or the duration of the second playback period, etc.

[0043] Image frame playback speed refers to the number of consecutive image frames that can be played within a unit of time (e.g., 1 minute, 1 second).

[0044] A speed change cycle can be, for example, the smallest repeating unit of the image frame playback speed change. For instance, in Figure 2, one speed change cycle includes a first playback period and a second playback period.

[0045] Because the playback speed of image frames in the original video is fixed at a standard speed, the number of original image frames played within one speed-changing cycle is constant if the target timeline is aligned with the original video timeline. Therefore, in some scenarios, the number of original image frames can be used to represent the cycle. For example, a speed-changing cycle of 30 means that the length of one speed-changing cycle is equal to the time required to play 30 consecutive image frames from the original video.

[0046] The proportion of the first playback segment in the speed-changing cycle could be, for example, the ratio of the duration of the first playback segment to the duration of a speed-changing cycle. Similarly, the proportion of the second playback segment in the speed-changing cycle could be, for example, the ratio of the duration of the second playback segment to the duration of a speed-changing cycle.

[0047] For example, assuming a speed-changing cycle includes only a first playback period and a second playback period, the length of the first playback period and the length of the second playback period can be obtained based on the proportion of the first playback period in the speed-changing cycle; similarly, the length of the first playback period and the length of the second playback period can be obtained based on the proportion of the second playback period in the speed-changing cycle.

[0048] For example, if the speed change period is T and the proportion of the second playback period in the speed change period is a, then the duration of the second playback period is T·a and the duration of the first playback period is T·(1-a).

[0049] In some embodiments, a general video speed parameter can be pre-specified, that is, regardless of the original video obtained, the pre-specified general video speed parameter is used as the video speed parameter corresponding to the original video.

[0050] In other embodiments, the video speed-changing parameters may not be universal but determined based on the characteristics of the original video. For example, obtaining the video speed-changing parameters corresponding to the original video includes: determining the video speed-changing parameters corresponding to the original video based on the rhythm of the background music in the original video. The purpose of this setting is to select video speed-changing parameters that match the rhythm of the background music in the original video, and use these parameters as the corresponding video speed-changing parameters for the original video. The advantage of this setting is that it associates the slow-motion playback period (i.e., the second playback period) of the final target video with the rhythm of the background music, forming a regular visual rhythm, increasing the dynamic aesthetics of the video, and creating a rhythmic visual impact.

[0051] In practice, optionally, when the video speed adjustment parameters include multiple parameters, some parameters can be set to be determined according to the rhythm of the background music of the original video, while other parameters are pre-specified fixed values.

[0052] S120. Based on the video speed adjustment parameters, determine the first playback period and the second playback period; the first playback period includes the first frame skipping moment, and the second playback period includes the second frame skipping moment; the playback speed of the first playback period is the same as the playback speed of the original video; the playback speed of the second playback period is less than the playback speed of the original video.

[0053] The principle of adding speed-changing effects to the original video using the technical solution of this application is to extract image frames from the original video according to the speed-changing requirements, and then reassemble the extracted image frames to form the target video.

[0054] The first frame extraction time could be, for example, the moment during the first playback period when an image frame needs to be extracted. The second frame extraction time could be, for example, the moment during the second playback period when an image frame needs to be extracted.

[0055] There are multiple ways to implement this step, and this application does not limit them. For example, the video speed adjustment parameters include the duration of the first playback period, the duration of the second playback period, the playback speed of the first playback period, and the playback speed of the second playback period. The implementation method of this step includes: determining the first playback period and the second playback period on the target timeline based on the duration of the first playback period and the duration of the second playback period; alternating the first playback period and the second playback period on the target timeline; determining the first frame-dropping moment within the first playback period based on the playback speed of the first playback period; and determining the second frame-dropping moment within the second playback period based on the playback speed of the second playback period.

[0056] When determining the first frame-dropping moment within the first playback period, the duration of the first playback period can be divided by the playback speed (i.e., standard speed) to obtain the time interval t1 (hereinafter referred to as the first time interval) between two adjacent image frames within the first playback period. The first time interval t1 is also the time interval between two adjacent first frame-dropping moments. Based on this first time interval t1, multiple first frame-dropping moments can be determined within the first playback period.

[0057] In another embodiment, since the playback speed of the first playback period is the same as the playback speed of the original video, the timeline of the original image frame in the original video can be aligned with the target timeline, and the time indicated by the timestamp of the original image frame in the first playback period can be used as the first frame extraction time.

[0058] In one embodiment, when determining the second frame-dropping moment within the second playback period, the duration of the second playback period can be divided by the playback speed of the second playback period to obtain the time interval t2 (hereinafter referred to as the second time interval) between two adjacent image frames within the second playback period. The second time interval t2 is also the time interval between two adjacent second frame-dropping moments. Based on this second time interval t2, multiple second frame-dropping moments can be determined within the second playback period.

[0059] S130. Based on the original video, determine the first image frame corresponding to the first frame extraction time and the second image frame corresponding to the second frame extraction time.

[0060] There are multiple specific implementation methods for "determining the first image frame corresponding to the first frame extraction time based on the original video", and this application does not limit this one. For example, a specific implementation method for "determining the first image frame corresponding to the first frame extraction time based on the original video" may include: determining a first candidate image frame in the original video, wherein the timestamp of the first candidate image frame is consistent with the first frame extraction time; and obtaining the first image frame corresponding to the first frame extraction time based on the first candidate image frame.

[0061] For example, referring to Figure 3, for the first frame extraction time M, an image frame with a timestamp consistent with the first frame extraction time M is searched from the original video. Since the timestamp of the first original image frame is consistent with the first frame extraction time M, the first original image frame is selected as the first candidate image frame. In this way, the first candidate image frames corresponding to each first frame extraction time can be obtained.

[0062] Based on the first candidate image frame, a first image frame corresponding to the first frame extraction time is obtained. For example, this may include using the first candidate image frame as the first image frame corresponding to the first frame extraction time; or, it may include: in the original video, determining a second candidate image frame, wherein the absolute value of the time difference between the timestamp of the second candidate image frame and the first frame extraction time is greater than 0 and less than or equal to a first set time threshold; and fusing the first candidate image frame and the second candidate image frame to obtain the first image frame corresponding to the first frame extraction time.

[0063] The first set time threshold is predetermined, and this application does not restrict the specific value of the first set time threshold. For example, the first set time threshold is determined based on the number of original image frames cached in the original image. For example, if the electronic device executing this application caches 30 original image frames at a time, the first set time threshold is determined to be the product of the inter-frame time of the original video and 29. The inter-frame time of the original video can be, for example, the absolute value of the difference between the timestamps of two consecutive original image frames in the original video.

[0064] Referring to Figure 3, the first set time threshold is set to the product of the inter-frame time in the original video and 19. For the first frame extraction time M, the original image frames from the 2nd frame to the 30th frame are all used as the second candidate image frames corresponding to the first frame extraction time M. Since the original image frame of the 1st frame is the first candidate image frame corresponding to the first frame extraction time M, the original image frames from the 1st frame to the 30th frame are subsequently fused.

[0065] In practice, there are various methods for fusing the first candidate image frame and the second candidate image frame, and this application does not limit this method. For example, when fusing the first candidate image frame and the second candidate image frame, the fusing can be based on the timestamps of each candidate image frame (including the first candidate image frame and the second candidate image frame). Specifically, a weight value for each candidate image frame is determined based on its timestamp; based on the weight values ​​of each candidate image frame, the candidate image frames are superimposed to obtain the fused image.

[0066] By fusing the first candidate image frame and the second candidate image frame, a first image frame corresponding to the first frame extraction moment can be obtained, which can create a motion blur effect and enhance the sense of speed.

[0067] In practice, since the playback speed of the second playback period differs from that of the original video, two scenarios may occur: one is that the original video includes an original image frame with a timestamp consistent with the second frame extraction time; the other is that the original video does not include an original image frame with a timestamp consistent with the second frame extraction time. To address this, optionally, a third candidate image frame is determined in the original video, where the absolute value of the time difference between the timestamp of the third candidate image frame and the second frame extraction time is less than the inter-frame time of the original video; based on the third candidate image frame, the second image frame corresponding to the second frame extraction time is obtained.

[0068] Specifically, assuming the inter-frame time in the original video is t3, if a certain second frame extraction time is t0, then the original image frame with timestamp ta satisfying t0-t3 < ta < t0+t3 is searched in the original video. This original image frame with timestamp ta satisfying t0-t3 < ta < t0+t3 is selected as the third candidate image frame. For example, referring to Figure 3, for the second frame extraction time N, the original image frame needs to be queried within the time range of (N-t3, N+t3). After the query, the 9th and 10th original image frames can be obtained as the third candidate image frames corresponding to the second frame extraction time N. For the second frame extraction time P, the original image frame needs to be queried within the time range of (P-t3, P+t3). After the query, the 12th original image frame can be obtained as the third candidate image frame corresponding to the second frame extraction time P.

[0069] Obtaining a second image frame corresponding to the second frame extraction time based on the third candidate image frame may include: if there is only one third candidate image frame, using that third candidate image frame as the second image frame corresponding to the second frame extraction time; if there are two third candidate image frames, obtaining an interpolated image based on the two third candidate image frames; and using the interpolated image as the second image frame corresponding to the second frame extraction time.

[0070] In practice, there are various methods for "obtaining an interpolated image based on two third candidate image frames", and this application does not limit this method. For example, "obtaining an interpolated image based on two third candidate image frames" may include: mixing the two third candidate image frames based on the second frame extraction time to obtain an interpolated image; and / or processing the two third candidate image frames using optical flow to obtain an interpolated image.

[0071] Furthermore, based on the timestamps of the two third candidate image frames and the second frame extraction time, the two third candidate image frames are mixed to obtain the interpolated image.

[0072] Alternatively, the two third candidate image frames can be blended based on the following formula to obtain the interpolated image: Interpolated image = (1-w(t)) × third candidate image frame 1 + w(t) × third candidate image frame 2

[0073] In this system, the timestamp of the third candidate image frame 1 is earlier than the timestamp t of the interpolated image, and the timestamp of the interpolated image is earlier than the timestamp of the third candidate image frame 2. The timestamp t of the interpolated image is the second frame extraction time. w(t) is the weight value of the third candidate image frame 2, and (1-w(t)) is the weight value of the third candidate image frame 1. w(t) is a function of time t, which can be a function that increases with time. The larger t is, the smaller the time interval between the second frame extraction time and the timestamp of the third candidate image frame 2. The larger w(t) is, the greater the influence of the third candidate image frame 2 on the interpolated image, and the smaller the influence of the third candidate image frame 1 on the interpolated image.

[0074] If the two third candidate image frames are the first reference image frame and the second reference image frame, where the timestamp of the first reference image frame is earlier than the timestamp of the second reference image frame, the two third candidate image frames are processed using optical flow to obtain an interpolated image. Referring to Figure 4, this process may include: obtaining a forward optical flow map and a backward optical flow map based on the first and second reference image frames; obtaining a predicted forward optical flow map corresponding to the second frame extraction time based on the second frame extraction time and the forward optical flow map; obtaining a predicted backward optical flow map corresponding to the second frame extraction time based on the second frame extraction time and the backward optical flow map; obtaining a first predicted frame based on the first reference image frame and the predicted forward optical flow map; obtaining a second predicted frame based on the second reference image frame and the predicted backward optical flow map; and mixing the first predicted frame and the second predicted frame to obtain the interpolated image.

[0075] The forward optical flow map is the optical flow map from the first reference image frame to the second reference image frame, used to reflect the motion information from the first reference image frame to the second reference image frame; the backward optical flow map is the optical flow map from the second reference image frame to the first reference image frame, used to reflect the motion information from the second image frame to the first image frame. Optionally, a deep learning network can be used to obtain the forward and backward optical flow maps based on the first and second reference image frames.

[0076] The second frame extraction time is the timestamp of the desired interpolated image.

[0077] The predicted forward optical flow graph is the optical flow graph from the first reference image frame to the interpolated image, used to reflect the motion information from the first reference image frame to the interpolated image. The predicted backward optical flow graph is the optical flow graph from the second reference image frame to the interpolated image, used to reflect the motion information from the second reference image frame to the interpolated image.

[0078] Optionally, obtaining the predicted forward optical flow map corresponding to the second frame extraction time based on the second frame extraction time and the forward optical flow map can include: obtaining the predicted forward optical flow map corresponding to the second frame extraction time based on the time interval between the timestamp of the first reference image frame and the second frame extraction time, the time interval between the timestamp of the first reference image frame and the first reference image frame, and the forward optical flow map. In some scenarios, the pixels in the image frame can be considered to move at a constant speed, and then the predicted forward optical flow map can be obtained by combining the law of constant speed movement. The method for obtaining the predicted backward optical flow map is similar to the method for obtaining the predicted forward optical flow map, and will not be described in detail here.

[0079] "Obtaining a first predicted frame based on a first reference image frame and a predicted forward optical flow map" may include, for example, moving pixels in the first reference image frame according to the predicted forward optical flow map, resulting in the first predicted frame. "Obtaining a second predicted frame based on a second reference image frame and a predicted backward optical flow map" may include, for example, moving pixels in the second reference image frame according to the predicted backward optical flow map, resulting in the second predicted frame.

[0080] The first and second predicted frames are mixed to obtain an interpolated image. For example, the first and second predicted frames can be superimposed to obtain the interpolated image. The timestamp of the resulting interpolated image is the second frame extraction time; therefore, the interpolated image is the candidate for the second image frame corresponding to the second frame extraction time.

[0081] S140. Based on the first image frame corresponding to the first frame extraction time and the second image frame corresponding to the second frame extraction time, the target video is obtained.

[0082] Optionally, the first image frame corresponding to the first extraction time and the second image frame corresponding to the second extraction time can be stitched together according to the chronological order of the first extraction time and the chronological order of the second extraction time to obtain the target video. In the target video, the timestamp of the first image frame is consistent with its corresponding first extraction time, and the timestamp of the second image frame is consistent with its corresponding second extraction time.

[0083] The above technical solution involves setting and acquiring the original video and corresponding video speed parameters; determining a first playback period and a second playback period based on the video speed parameters; the first playback period includes a first frame drop moment, and the second playback period includes a second frame drop moment; the playback speed of the first playback period is the same as the playback speed of the original video; the playback speed of the second playback period is less than the playback speed of the original video; determining a first image frame corresponding to the first frame drop moment and a second image frame corresponding to the second frame drop moment based on the original video; and obtaining the target video based on the first image frame corresponding to the first frame drop moment and the second image frame corresponding to the second frame drop moment. Essentially, it proposes a new special effect that presents the effect of playing image frames at standard speed for a period of time and then playing them at slow speed for another period of time during the playback of the target video. During the period of playing image frames at standard speed, the video content can be faithfully reproduced, maintaining the continuity of image frames. During the period of playing image frames at slow speed, subtle movements can be made more clearly visible.

[0084] It should be noted that in practice, because the playback speed of the second playback period is lower than that of the first playback period, not all original image frames in the original video may be used when determining the second image frame corresponding to the second frame extraction time. For example, referring to Figure 3, in this example, frames 7, 8, 11, 13, 16, 19, 24, 27, and 30 of the original video are not used. This results in the target video appearing as if the motion is broken down and fast-forwarded during the second playback period.

[0085] It should also be noted that in the above example, a variable speed cycle includes a standard speed playback period and a slow speed playback period. This is a specific example of this application and not a limitation thereof. In practice, a variable speed cycle may include multiple standard speed playback periods and multiple slow speed playback periods. Optionally, the playback speed of at least some of the slow speed playback periods may be set to be different. For example, a variable speed cycle includes one standard speed playback period and two slow speed playback periods, where the playback speed of one slow speed playback period is half the standard speed, and the playback speed of the other slow speed playback period is one-quarter of the standard speed.

[0086] Based on the above technical solution, optionally, after S140, it may further include: determining image frames in the target video that are associated with the rhythm points of the background music; adding a glowing effect to the image frames associated with the rhythm points of the background music.

[0087] Image frames associated with the rhythm points of the background music can be, for example, image frames in the target video whose timestamps are near the time corresponding to the rhythm points of the background music.

[0088] Using the time corresponding to the rhythm point of the background music as the center, preset time lengths are calculated forward and backward to obtain a reference time period; the image frame associated with the rhythm point of the background music can be, for example, an image frame with a timestamp in the reference time period.

[0089] This application does not limit the specific value of the preset time length. For example, if the preset time length is tm, and the time corresponding to a certain rhythm point of the background music is tn, then the start time of the reference time period corresponding to that rhythm point is tn-tm, and the end time is tn+tm. The image frame with the timestamp in that reference time period is the image frame associated with that rhythm point.

[0090] Adding a glowing effect to an image frame that is set to be associated with the rhythm of the background music can enhance the artistic expression of the target video.

[0091] It is understood that before using the technical solutions disclosed in the various embodiments of this disclosure, users should be informed of the types, scope of use, and usage scenarios of the personal information involved in this disclosure in an appropriate manner in accordance with relevant laws and regulations, and user authorization should be obtained.

[0092] For example, upon receiving a user's active request, a prompt message is sent to the user to explicitly inform them that the requested operation will require the acquisition and use of the user's personal information. This allows the user to independently choose whether to provide personal information to the software or hardware, such as the electronic device, application, server, or storage medium performing the operations of this disclosed technical solution, based on the prompt message.

[0093] As an optional but non-limiting implementation, in response to a user's active request, sending a prompt message to the user can be done via a pop-up window, where the prompt message can be presented in text format. Furthermore, the pop-up window can also include a selection control allowing the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0094] It is understood that the above notification and user authorization process are merely illustrative and do not constitute a limitation on the implementation of this disclosure. Other methods that comply with relevant laws and regulations may also be applied to the implementation of this disclosure.

[0095] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that the present invention is not limited to the described order of actions, because according to the present invention, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to the present invention.

[0096] Figure 5 is a schematic diagram of a video processing device according to an embodiment of this disclosure. The video processing device provided in this embodiment can be configured in a client or in a server. Referring to Figure 5, the video processing device specifically includes:

[0097] The acquisition module 310 is used to acquire the original video and the video speed change parameters corresponding to the original video;

[0098] The first determining module 320 is used to determine a first playback period and a second playback period based on the video speed adjustment parameters; the first playback period includes a first frame-dropping moment, and the second playback period includes a second frame-dropping moment; the playback speed of the first playback period is the same as the playback speed of the original video; the playback speed of the second playback period is less than the playback speed of the original video.

[0099] The second determining module 330 is used to determine, based on the original video, a first image frame corresponding to the first frame extraction time and a second image frame corresponding to the second frame extraction time.

[0100] The stitching module 340 is used to obtain the target video based on the first image frame corresponding to the first frame extraction time and the second image frame corresponding to the second frame extraction time.

[0101] Furthermore, module 310 is used for:

[0102] Based on the background music rhythm of the original video, determine the video speed adjustment parameters corresponding to the original video.

[0103] Further, the video speed adjustment parameters include the duration of the first playback period, the duration of the second playback period, the playback speed of the first playback period, and the playback speed of the second playback period; the first determining module 320 is used for:

[0104] Based on the duration of the first playback period and the duration of the second playback period, a first playback period and a second playback period are determined on the target timeline, and the first playback period and the second playback period are arranged alternately on the target timeline;

[0105] Based on the playback speed of the first playback period, the first frame extraction time is determined within the first playback period;

[0106] The second frame-dropping moment is determined within the second playback period based on the playback speed of the second playback period.

[0107] Furthermore, the second determining module 330 is used for:

[0108] In the original video, a first candidate image frame is determined, and the timestamp of the first candidate image frame is consistent with the first frame extraction time.

[0109] Based on the first candidate image frame, the first image frame corresponding to the first frame extraction time is obtained.

[0110] Furthermore, the second determining module 330 is used for:

[0111] In the original video, a second candidate image frame is determined, wherein the absolute value of the time difference between the timestamp of the second candidate image frame and the first frame extraction time is greater than 0 and less than or equal to a first set time threshold.

[0112] The first candidate image frame and the second candidate image frame are fused to obtain the first image frame corresponding to the first frame extraction time.

[0113] Furthermore, the second determining module 330 is used for:

[0114] In the original video, a third candidate image frame is determined, wherein the absolute value of the time difference between the timestamp of the third candidate image frame and the second frame extraction time is less than the inter-frame time of the original video.

[0115] Based on the third candidate image frame, a second image frame corresponding to the second frame extraction time is obtained.

[0116] Furthermore, the second determining module 330 is used for:

[0117] If there are two third candidate image frames, an interpolated image is obtained based on the two third candidate image frames;

[0118] The interpolated image is used as the second image frame corresponding to the second frame extraction time.

[0119] Furthermore, the second determining module 330 is used for:

[0120] Based on the second frame extraction time, the two third candidate image frames are blended to obtain the interpolated image; and / or,

[0121] The two third candidate image frames are processed using optical flow to obtain interpolated images.

[0122] Furthermore, the device also includes a light emission effect addition module, which is used to determine the image frame associated with the rhythm point of the background music in the target video after obtaining the target video based on the first image frame corresponding to the first frame-dropping time and the second image frame corresponding to the second frame-dropping time.

[0123] Add a glowing effect to the image frames associated with the rhythm points of the background music.

[0124] The video processing apparatus provided in this disclosure can execute the steps performed by the client or server in the video processing method provided in this disclosure, and has the execution steps and beneficial effects, which will not be described in detail here.

[0125] Figure 6 is a schematic diagram of the structure of an electronic device according to an embodiment of this disclosure. Referring specifically to Figure 6 below, it shows a schematic diagram of the structure suitable for implementing the electronic device 1000 in the embodiments of this disclosure. The electronic device 1000 in the embodiments of this disclosure may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), in-vehicle terminals (e.g., in-vehicle navigation terminals), wearable electronic devices, etc., as well as fixed terminals such as digital TVs, desktop computers, smart home devices, etc. The electronic device shown in Figure 6 is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this disclosure.

[0126] As shown in FIG6, the electronic device 1000 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 1001, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1002 or a program loaded from a storage device 1008 into a random access memory (RAM) 1003 to implement the video processing method as described in the embodiments of the present disclosure. The RAM 1003 also stores various programs and information required for the operation of the electronic device 1000. The processing device 1001, ROM 1002, and RAM 1003 are interconnected via a bus 1004. An input / output (I / O) interface 1005 is also connected to the bus 1004.

[0127] Typically, the following devices can be connected to the I / O interface 1005: input devices 1006 including, for example, a touchscreen, touchpad, keyboard, mouse, camera, microphone, accelerometer, gyroscope, etc.; output devices 1007 including, for example, a liquid crystal display (LCD), speaker, vibrator, etc.; storage devices 1008 including, for example, magnetic tape, hard disk, etc.; and communication devices 1009. Communication device 1009 allows electronic device 1000 to communicate wirelessly or wiredly with other devices to exchange information. Although Figure 6 shows an electronic device 1000 with various devices, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively.

[0128] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts, thereby implementing the video processing method as described above. In such embodiments, the computer program can be downloaded and installed from a network via communication device 1009, or installed from storage device 1008, or installed from ROM 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of embodiments of this disclosure.

[0129] It should be noted that the computer-readable medium described in this disclosure can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this disclosure, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium can include information signals propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated information signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0130] In some implementations, clients and servers may communicate using any known or future network protocol, such as HTTP (Hypertext Transfer Protocol), and may interconnect with any form or medium of digital information communication (e.g., a communication network). Examples of communication networks include local area networks (“LANs”), wide area networks (“WANs”), the Internet (e.g., the Internet of Things), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any known or future network.

[0131] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device.

[0132] The aforementioned computer-readable medium carries one or more programs that, when executed by the electronic device, cause the electronic device to:

[0133] Obtain the original video and the corresponding video speed parameters;

[0134] Based on the video speed adjustment parameters, a first playback period and a second playback period are determined; the first playback period includes a first frame skipping moment, and the second playback period includes a second frame skipping moment; the playback speed of the first playback period is the same as the playback speed of the original video; the playback speed of the second playback period is less than the playback speed of the original video.

[0135] Based on the original video, a first image frame corresponding to the first frame extraction time and a second image frame corresponding to the second frame extraction time are determined.

[0136] The target video is obtained based on the first image frame corresponding to the first frame extraction time and the second image frame corresponding to the second frame extraction time.

[0137] Optionally, when one or more of the above-described procedures are executed by the electronic device, the electronic device may also perform other steps described in the above embodiments.

[0138] Computer program code for performing the operations of this disclosure can be written in one or more programming languages ​​or a combination thereof, including but not limited to object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0139] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0140] The units described in the embodiments of this disclosure can be implemented in software or hardware. The names of the units are not, in some cases, intended to limit the specific unit.

[0141] The functions described above in this document can be performed, at least in part, by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: Field Programmable Gate Arrays (FPGAs), Application-Specific Integrated Circuits (ASICs), Application Standard Products (ASSPs), System-on-Chip (SoCs), Complex Programmable Logic Devices (CPLDs), and so on.

[0142] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0143] According to one or more embodiments of this disclosure, this disclosure provides an electronic device, including:

[0144] One or more processors;

[0145] Memory, used to store one or more programs;

[0146] When the one or more programs are executed by the one or more processors, the one or more processors implement any of the video processing methods provided in this disclosure.

[0147] According to one or more embodiments of the present disclosure, the present disclosure provides a computer-readable storage medium having a computer program stored thereon that, when executed by a processor, implements a video processing method as described in any of the present disclosure.

[0148] This disclosure also provides a computer program product, which includes a computer program or instructions that, when executed by a processor, implement the video processing method described above.

[0149] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0150] The above description is merely a specific embodiment of this disclosure, enabling those skilled in the art to understand or implement it. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this disclosure. Therefore, this disclosure is not to be limited to the embodiments described herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A video processing method, comprising: Obtain the original video and the corresponding video speed parameters; Based on the video speed adjustment parameters, the first playback period and the second playback period are determined; The first playback period includes the first frame-dropping moment, and the second playback period includes the second frame-dropping moment; The playback speed of the first playback period is the same as the playback speed of the original video; The playback speed of the second playback period is less than the playback speed of the original video; Based on the original video, a first image frame corresponding to the first frame extraction time and a second image frame corresponding to the second frame extraction time are determined. The target video is obtained based on the first image frame corresponding to the first frame extraction time and the second image frame corresponding to the second frame extraction time.

2. The method according to claim 1, wherein obtaining the video speed adjustment parameters corresponding to the original video includes: Based on the background music rhythm of the original video, determine the video speed adjustment parameters corresponding to the original video.

3. The method according to claim 1, wherein the video speed adjustment parameters include the duration of the first playback period, the duration of the second playback period, the playback speed of the first playback period, and the playback speed of the second playback period; The step of determining the first playback period and the second playback period based on the video speed adjustment parameters includes: Based on the duration of the first playback period and the duration of the second playback period, a first playback period and a second playback period are determined on the target timeline, and the first playback period and the second playback period are arranged alternately on the target timeline; Based on the playback speed of the first playback period, the first frame extraction time is determined within the first playback period; The second frame-dropping moment is determined within the second playback period based on the playback speed of the second playback period.

4. The method according to claim 1, wherein determining the first image frame corresponding to the first frame extraction time based on the original video includes: In the original video, a first candidate image frame is determined, and the timestamp of the first candidate image frame is consistent with the first frame extraction time. Based on the first candidate image frame, the first image frame corresponding to the first frame extraction time is obtained.

5. The method according to claim 4, wherein obtaining the first image frame corresponding to the first frame extraction time based on the first candidate image frame includes: In the original video, a second candidate image frame is determined, wherein the absolute value of the time difference between the timestamp of the second candidate image frame and the first frame extraction time is greater than 0 and less than or equal to a first set time threshold. The first candidate image frame and the second candidate image frame are fused to obtain the first image frame corresponding to the first frame extraction time.

6. The method according to claim 1, wherein determining the second image frame corresponding to the second frame extraction time based on the original video comprises: In the original video, a third candidate image frame is determined, wherein the absolute value of the time difference between the timestamp of the third candidate image frame and the second frame extraction time is less than the inter-frame time of the original video. Based on the third candidate image frame, a second image frame corresponding to the second frame extraction time is obtained.

7. The method according to claim 6, wherein obtaining the second image frame corresponding to the second frame extraction time based on the third candidate image frame comprises: If there are two third candidate image frames, an interpolated image is obtained based on the two third candidate image frames; The interpolated image is used as the second image frame corresponding to the second frame extraction time.

8. The method according to claim 7, wherein obtaining the interpolated image based on the two third candidate image frames comprises: Based on the second frame extraction time, the two third candidate image frames are mixed to obtain the interpolated image; And / or, The two third candidate image frames are processed using optical flow to obtain interpolated images.

9. The method according to claim 1, wherein after obtaining the target video based on the first image frame corresponding to the first frame extraction time and the second image frame corresponding to the second frame extraction time, the method further includes: Identify image frames in the target video that are associated with the rhythm points of the background music; Add a glowing effect to the image frames associated with the rhythm points of the background music.

10. A video processing apparatus, comprising: The acquisition module is used to acquire the original video and the video speed adjustment parameters corresponding to the original video; The first determining module is used to determine the first playback period and the second playback period based on the video speed change parameters; The first playback period includes the first frame-dropping moment, and the second playback period includes the second frame-dropping moment; The playback speed of the first playback period is the same as the playback speed of the original video; The playback speed of the second playback period is less than the playback speed of the original video; The second determining module is used to determine, based on the original video, a first image frame corresponding to the first frame extraction time and a second image frame corresponding to the second frame extraction time; The stitching module is used to obtain the target video based on the first image frame corresponding to the first frame extraction time and the second image frame corresponding to the second frame extraction time.

11. An electronic device, the electronic device comprising: One or more processors; Storage device for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1-9.

12. A computer-readable storage medium having a computer program stored thereon that, when executed by a processor, implements the method as described in any one of claims 1-9.

Images