Method, device, apparatus and medium for video processing

Abstract

According to embodiments of this disclosure, a method, apparatus, device, and medium for video processing are provided. The method includes, for multiple frames of a target video, determining multiple optical flow features between adjacent frames, the optical flow features indicating the motion direction and amplitude of multiple points in the frames; determining a sequence of motion change intensities of the multiple frames over time based on the multiple optical flow features; selecting at least one target motion change intensity from the sequence of motion change intensities; and determining at least one video beat point in the target video based on at least one frame corresponding to the at least one target motion change intensity. This enables the extraction of video beat points based on the motion change intensity in video content, improving the accuracy of video beat point extraction and significantly enhancing the efficiency and effectiveness of beat-based video production.

Description

Technical Field

[0001] Example embodiments of the present disclosure generally relate to the field of computer technology, and more particularly, to methods, devices, apparatuses, and computer-readable storage media for video processing. Background Art

[0002] The rhythm or beat information of a video is mainly reflected by the action changes of the target or the scene switching in the video. The extraction of the beat information of the video can help various uses of the video in subsequent tasks. For example, in video editing, it may be expected that the video and music can match each other in rhythm to improve the combination of video effects and music rhythm. Summary of the invention

[0003] In a first aspect of the present disclosure, a method for video processing is provided. The method comprises: for a plurality of frames of a target video, determining a plurality of optical flow features between adjacent frames in the plurality of frames, the optical flow features indicating the motion direction and motion amplitude of a plurality of points in the frames; based on the plurality of optical flow features, determining a sequence of motion change strengths of the plurality of frames over time; selecting at least one target motion change strength from the sequence of motion change strengths; and determining at least one video beat point in the target video based on at least one frame corresponding to the at least one target motion change strength.

[0004] In a second aspect of the present disclosure, a device for video processing is provided. The device includes: an optical flow feature determination module, configured to determine, for multiple frames of a target video, multiple optical flow features between adjacent frames in the multiple frames, the optical flow features indicating the motion direction and motion amplitude of multiple points in the frames; a sequence determination module, configured to determine, based on the multiple optical flow features, a sequence of motion change intensities of the multiple frames over time; a selection module, configured to select at least one target motion change intensity from the sequence of motion change intensities; and a video beat point determination module, configured to determine at least one video beat point in the target video based on at least one frame corresponding to the at least one target motion change intensity.

[0005] In a third aspect of the present disclosure, an electronic device is provided. The device includes at least one processing unit; and at least one memory, the at least one memory is coupled to the at least one processing unit and stores instructions for execution by the at least one processing unit. When the instructions are executed by the at least one processing unit, the device executes the method of the first aspect.

[0006] In a fourth aspect of the present disclosure, a computer-readable storage medium is provided, wherein a computer program is stored on the medium, and when the computer program is executed by a processor, the method of the first aspect is implemented.

[0007] It should be understood that the content described in this section is not intended to limit the key features or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. In the accompanying drawings, the same or similar reference numerals represent the same or similar elements, wherein:

[0009] Figure 1 A schematic diagram showing an example environment in which embodiments of the present disclosure can be implemented;

[0010] Figure 2 A flowchart showing a process of video processing according to some embodiments of the present disclosure is shown;

[0011] Figure 3 A flowchart showing a process of determining a sequence of motion variation intensities according to some embodiments of the present disclosure;

[0012] Figure 4 A flowchart showing a process of making a merged video according to some embodiments of the present disclosure is shown;

[0013] Figure 5 A schematic diagram showing an example mapping relationship between video beat points and audio beat points according to some embodiments of the present disclosure;

[0014] Figure 6 A block diagram showing an apparatus for video processing according to some embodiments of the present disclosure; and

[0015] Figure 7 A block diagram of an electronic device is shown in which one or more embodiments of the present disclosure may be implemented. DETAILED DESCRIPTION

[0016] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments set forth herein. On the contrary, these embodiments are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0017] In the description of the embodiments of the present disclosure, the term "including" and similar terms should be understood as open inclusion, that is, "including but not limited to". The term "based on" should be understood as "based at least in part on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The term "some embodiments" should be understood as "at least some embodiments". Other explicit and implicit definitions may also be included below.

[0018] It is understandable that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) shall comply with the requirements of relevant laws, regulations and relevant provisions.

[0019] It is understandable that before using the technical solutions disclosed in the embodiments of the present disclosure, the types, scope of use, usage scenarios, etc. of the personal information involved in the present disclosure should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.

[0020] For example, in response to receiving an active request from a user, a prompt message is sent to the user to clearly prompt the user that the operation requested to be performed will require obtaining and using the user's personal information, so that the user can independently choose whether to provide personal information to software or hardware such as electronic devices, applications, servers or storage media that execute operations of the technical solution of the present disclosure based on the prompt message.

[0021] As an optional but non-limiting implementation, in response to receiving an active request from the user, the prompt information is sent to the user in a manner such as a pop-up window, in which the prompt information can be presented in text form. In addition, the pop-up window can also carry a selection control for the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0022] It is understandable that the above notification and the process of obtaining user authorization are merely illustrative and do not constitute a limitation on the implementation of the present disclosure. Other methods that meet relevant laws and regulations may also be applied to the implementation of the present disclosure.

[0023] As briefly mentioned above, the beat information of a video (also called rhythm information) is very useful in many scenarios. For example, in video editing, one must not only master the skills of cutting and splicing, but also pay attention to the coordination between the video content and the music. Changes in character movements and video transitions in the video can be used as rhythm features of the video. The video rhythm changes are extracted based on the visual content in the video, and the rhythm changes are further extracted as video beat points. The video beat points are aligned with the beat points of the target audio to generate a merged video. Such a merged video is sometimes also called a card point video.

[0024] Currently, there is no good solution for extracting the beat information of a video. Therefore, when making a merged video, a possible solution is often to use a transitional video synthesis with the target music and multiple images as input, or with the target music and multiple video clips as input. The goal of this type of method is to display the image or video clip at the beat time of the target music. In the merged video, it is manifested as: the video content transitions to another image or another video clip at the beat of the music. This type of method does not pay attention to the rhythm of the video content, and cannot handle the card point requirements when a single video is used as input.

[0025] Embodiments of the present disclosure provide a scheme for video processing. According to various embodiments of the present disclosure, multiple optical flow features between adjacent frames in multiple frames of a target video are determined. Based on the multiple optical flow features, a sequence of motion change intensities of multiple frames over time is determined. At least one target motion change intensity is selected from the sequence of motion change intensities. Based on at least one frame corresponding to the at least one target motion change intensity, at least one video beat point in the target video is determined.

[0026] In the embodiments of the present disclosure, the intensity of motion changes in video content is analyzed based on the principle of optical flow, and the beat points of the video are automatically extracted. In this way, the video beat point extraction based on the intensity of motion changes in the video content is achieved, and the extraction accuracy of the video beat point is improved. This helps the efficiency and effect of video production that relies on beat points, and increases the overall coordination, rhythm and fluency of the video.

[0027] Example embodiments of the present disclosure are described below with reference to the accompanying drawings.

[0028] Figure 1 1 is a schematic diagram of an example environment 100 in which embodiments of the present disclosure can be implemented. Figure 1 As shown, environment 100 involves a video beat extraction system 110 , an audio beat extraction system 120 , and a beat point matching system 130 .

[0029] exist Figure 1 In the example of , the video beat extraction system 110 is configured to receive the target video 101 as input and output one or more video beat points 102 of the target video 101. The video beat extraction system 110 extracts beat points, also called rhythm points, in the target video 110. Such beat points can be video transitions, screen switching, rhythm changes, human body motion switching, object motion switching, and the like.

[0030] The audio beat extraction system 120 is configured to receive the target audio 103 as input and output one or more audio beat points 104 of the target audio 103. The target audio 103 may include music, speech, or a combination of music and speech. Beat is the combination rule of strong beats and weak beats of music. The audio beat extraction system 120 extracts the beat points in the target audio 103, which are also called rhythm points. Such beat points can be strong beats, weak beats or a combination thereof in music, or can be sudden changes in intonation, sudden changes in volume, etc. in speech, or can be key points with certain characteristics in audio, such as "drum beats", etc.

[0031] In some embodiments, the environment 100 further includes a beat point matching system 130, which is configured to match the video beat points 102 with the audio beat points 104, and output a merged video 105. Beat point matching is to make the video rhythm stuck on the music rhythm, so the merged video 105 can also be called a stuck video. In some embodiments, the beat point matching system 130 establishes a time mapping relationship between the video beat points 102 and the audio beat points 104, thereby generating a video with a video screen and a music beat stuck video.

[0032] In some embodiments, the environment 100 may further include a video beat library 140. The video beat library 140 is configured to store the extracted video beat points 102 for subsequent other uses. In some embodiments, the video beat points 102 extracted from the target video 101 may also be presented to the user, accessed according to video synthesis requirements, and the like. In some embodiments, in addition to being used to match with audio to produce a merged video, the beat information of the video (e.g., one or more video beat points 102) may also be used to perform video clustering, video search, video matching, and the like based on the beat information.

[0033] It should be understood that the structure and functionality of environment 100 are described for exemplary purposes only and do not imply any limitation on the scope of the present disclosure. Figure 1 , multiple separate systems are shown, such as a video beat extraction system 110, an audio beat extraction system 120, and a beat point matching system 130, but two or more of these systems may also be integrated into a single device or system. In some scenarios, Figure 1 Some systems and / or elements may also be omitted.

[0034] Some example embodiments of the present disclosure will be described below with continued reference to the accompanying drawings.

[0035] Figure 2 FIG. 2 is a flowchart showing a process 200 of video processing according to some embodiments of the present disclosure. The process 200 may be implemented in Figure 1 For ease of discussion, reference will be made to the video beat extraction system 110. Figure 1 The process 200 is described below.

[0036] In block 210 , a plurality of frames in a target video are grouped together, and the video beat extraction system 110 determines a plurality of optical flow features between adjacent frames in the group.

[0037] The optical flow feature indicates the motion information of multiple points in the frame, including the direction and amplitude of motion. That is, the motion information of each point can be expressed in the form of a vector, also called an optical flow vector or a motion vector. The direction of the optical flow vector indicates the direction of motion, and the size of the optical flow vector indicates the amplitude of motion. The frames in the video (also called "video frames") can be considered as two-dimensional images. The points indicated by each optical flow feature can correspond to pixels in the two-dimensional image. In the real three-dimensional space, each point in the three-dimensional space will reach a new position after a certain period of movement, and this displacement process can be described by a motion field. In the space of computer vision, the signals received by the computer are often two-dimensional image information, so it is suitable to use the optical flow field to describe the motion vector field of the pixels reflected in the two-dimensional image of the moving objects in the three-dimensional space.

[0038] For example, multiple frames of the target video 101 can be represented as a video frame sequence I = [I1, I2, ..., I n ], where n is a positive integer. The video beat extraction system 110 calculates the optical flow between every two adjacent frames in the video frame sequence I, and obtains an optical flow feature sequence O = [O1, O2, ... n-1 ],O i represents the i-th optical flow feature (t=1, 2, ...n-1).

[0039] In some embodiments, the video beat extraction system 110 may use a dense optical flow method (eg, GunnarFarneback algorithm) to determine the neighboring frames. t ,I t+1 ] between the dense optical flow, and obtain the optical flow feature O t In some other embodiments, the video beat extraction system 110 may use a sparse optical flow method (eg, Lucas-Kanade algorithm) to determine and calculate adjacent frames [I t ,I t+1 ] between the sparse optical flow, and obtain the optical flow feature O t. Dense optical flow is used to describe the optical flow information of all pixels in a frame, and sparse optical flow is used to describe the optical flow information of some pixels in a frame. Generally, dense optical flow captures the most motion information; sparse optical flow has the fastest calculation speed. The video beat extraction system 110 can choose to calculate dense optical flow or sparse optical flow based on different types of video content and / or different application needs. For example, a video recorded in a fruit and vegetable market has rich picture content and is suitable for dense optical flow. For another example, a video recorded in a dance studio has a clean and monotonous background. When capturing the dancer's movements, sparse optical flow is suitable.

[0040] In some embodiments, before determining the optical flow features between adjacent frames, the video beat extraction system 110 extracts multiple frames (n frames) from the target video 101 at preset time intervals for video beat extraction. Video is composed of continuous static image screens. According to the principle of visual persistence, if the number of frames of a video exceeds 24 frames, the human eye cannot distinguish a single static picture, and the perception is a smooth and continuous visual effect. Therefore, the motion change between adjacent frames of the target video 101 is very small. Extracting multiple frames from all frames of the target video 101 for processing not only saves time, but also improves the user experience.

[0041] In box 220, the video beat extraction system 110 determines the sequence of motion change intensity of the group of frames over time based on the extracted multiple optical flow features. The motion change intensity indicates the picture change intensity value of the video in the time dimension. A series of motion change intensities can be determined from multiple optical flow features to form a sequence of motion change intensities. Continuing with the example of the dance studio, the dancer waves his right hand, and the right hand moves upward to the apex with acceleration and then falls down. This motion information in three-dimensional space is reflected in the continuous frames of the video as a change in the two-dimensional vector value of multiple pixels. The motion information indicated by the optical flow feature can be used to determine the motion change intensity.

[0042] A single optical flow feature may indicate a change in motion in adjacent frames. In some embodiments, the video beat extraction system 110 may determine a plurality of optical flow change features based on differences between adjacent optical flow features in the plurality of optical flow features. Each optical flow change feature may be used to indicate a degree of motion change in consecutive frames. The video beat extraction system 110 may determine a motion change intensity corresponding to the optical flow change feature based on a plurality of optical flow change features.

[0043] Figure 3 FIG. 3 is a flowchart showing a process 300 for determining a sequence of motion change intensities according to some embodiments of the present disclosure. The process 300 may be implemented in Figure 1 Video beat extraction system 110.

[0044] In box 310, the video beat extraction system 110 divides the range of motion direction into multiple direction intervals. In a frame, the motion direction indicated by the optical flow feature usually varies within a direction angle range of 0 to 360° (i.e., 0 to 2π). By dividing this range into multiple direction intervals, the motion direction of the object can be quantified. For example, the range can be divided into k direction intervals, k is a positive integer greater than 1, and the size of each direction interval is 2π / k. The more direction intervals are divided, the more motion direction information is retained later. Such division can be equal or unequal. The granularity and number of divisions of the direction intervals can be determined according to the actual application.

[0045] In box 320, the video beat extraction system 110 determines the number of points that fall into multiple directional intervals (for example, k directional intervals) in each of the multiple optical flow features, and obtains multiple optical flow histograms corresponding to each of the multiple optical flow features. A histogram is a statistic of the distribution of a continuous variable. Performing histogram statistics on the optical flow features can obtain the overall motion information of the subject in the video frame. Specifically, for each optical flow feature, each point (for example, the corresponding optical flow vector) is divided into a corresponding directional interval based on its motion direction. By converting each optical flow feature into a k-dimensional histogram statistical feature, the video beat extraction system 110 obtains the corresponding optical flow histogram (also called a video optical flow spectrum) H = [H1, H2, ..., H n-1 ], where H i ∈R k .

[0046] In some video scenes, the optical flow feature captures a large number of subtle motion changes between adjacent frames. For example, in a lighting scene, there will be light changes between adjacent frames, but the main body of the picture does not move. A large number of points in the extracted optical flow features will indicate changes caused by light. In some embodiments, in the statistics of the optical flow histogram, the video beat extraction system 110 can filter points whose motion amplitude is lower than the amplitude threshold from the optical flow features to obtain filtered optical flow features. The video beat extraction system 110 can determine the number of points that fall into multiple direction intervals in the filtered optical flow features to obtain an optical flow histogram corresponding to a given optical flow feature. For example, when the histogram is counted, it is first determined whether the motion amplitude indicated by each point of the optical flow feature (for example, the modulus of the optical flow vector) exceeds the amplitude threshold, and then only the points exceeding the amplitude threshold are counted. In some embodiments, for each optical flow feature, the amplitude threshold used can be determined based on the motion amplitude of all points indicated by the optical flow feature, for example, it can be the average value, median value, etc. of the motion amplitude of all points. Thus, it is possible to avoid counting small motions in the picture in the optical flow histogram.

[0047] In block 330, the video beat extraction system 110 determines a plurality of optical flow change features based on a plurality of differences between adjacent optical flow histograms in a plurality of optical flow histograms. A corresponding optical flow histogram may be determined for each optical flow feature. The differences between the optical flow histograms may be used to indicate the differences between the corresponding optical flow features, i.e., the optical flow change feature. In some embodiments, the optical flow change feature may be the difference in the number of directional intervals of adjacent optical flow histograms in each of a plurality of directional intervals. Thus, the video beat extraction system 110 performs an optical flow histogram H=[H1, H2, ..., H n-1 The adjacent optical flow histograms in ] are subtracted to obtain the optical flow change feature D = [D1, D2, ..., D n-2 ], where D i ∈R k In some embodiments, when calculating the difference between each direction interval in adjacent optical flow histograms, the difference may be calculated and then modulo-calculated as the corresponding value in the optical flow change feature.

[0048] The optical flow change feature can indicate the degree of motion change between consecutive frames. Continuing the example of the dance studio, when the dancer waves her right hand upward, the optical flow feature can be captured in the corresponding video frame as being concentrated in some directional intervals facing upward, and the number of these directional intervals in the optical flow histogram is the largest. If the next video frame captures the dancer's right hand pausing, the number of these directional intervals in the optical flow histogram corresponding to the corresponding optical flow feature is significantly reduced. The difference between these adjacent optical flow histograms will be large, showing a large degree of motion change.

[0049] In block 340 , the video beat extraction system 110 determines a sequence of motion variation strengths based on the plurality of optical flow variation features.

[0050] In some embodiments, for each optical flow change feature among the multiple optical flow change features, the video beat extraction system 110 determines the motion change intensity corresponding to the optical flow change feature by adding multiple differences indicated by the optical flow change feature in multiple direction intervals.

[0051] For example, for each D in the optical flow change feature sequence D i , the sum of the values ​​corresponding to all the directional intervals can be calculated. In this way, the video beat extraction system 110 can obtain the sequence P = [P1, P2, ..., P n-2 ], where each P i is a scalar.

[0052] Some examples of sequences of intensity motion varying intensities are discussed above. Figure 2In block 230, the video beat extraction system 110 selects at least one target motion change intensity from the sequence of motion change intensities. In some embodiments, the selected target motion change intensity may be an intensity that is significantly different from one or more motion change intensities before and after it in the entire sequence. These intensities generally correspond to turning points, scene switching points, etc., of motion changes of people or objects in the video, and thus are the video beat points to be detected.

[0053] Continuing with the dance studio example, the dancer is waving her right hand. The video beat extraction system 110 extracts the tempo of the motion variation intensity sequence P = [P1, P2, ..., P n-2 ], select one or more target motion change intensities. Such target motion change intensities may be reflected as the right hand reaching or about to reach the top, or may be reflected as the right hand falling to the vicinity of the top of the head after reaching the top.

[0054] In some embodiments, the video beat extraction system 110 selects at least one intensity extreme value from the sequence of motion change intensities as the target motion change intensity. If the sequence of motion change intensities is represented by a curve, the intensity extreme value can be represented as a peak or a trough on the curve, corresponding to a position with a large difference compared to the previous and next motion change intensities. Of course, in addition to selecting the extreme value point in the sequence, the target motion change intensity can also be selected according to other criteria, and the embodiments of the present disclosure are not limited in this respect.

[0055] In block 240 , the video beat extraction system 110 determines the video beat point 102 in the target video 101 based on at least one frame corresponding to at least one target motion change intensity.

[0056] Looking back at the previous video processing process, the video beat extraction system 110 extracts the video frame sequence I = [I1, I2, ..., I n After processing two adjacent frames in ], we get the optical flow feature sequence O = [O1, O2, ... n-1 The video beat extraction system 110 performs histogram statistics on the optical flow feature sequence O to obtain an optical flow histogram H = [H1, H2, ..., H n-1 After the video beat extraction system 110 processes the adjacent histograms in the optical flow histogram H in pairs, the optical flow change feature D = [D1, D2, ..., D n-2 ], and then get the sequence of motion change intensity P = [P1, P2, ..., P n-2]. Therefore, the target motion change intensity selected in the sequence of motion change intensity may correspond to three frames extracted from the target video 101. These three frames may correspond to a certain time point or a short time interval of the target video 102. The video beat extraction system 110 may determine the time point or time interval as the video beat point 102. When the sampling frame rate is high, these frames corresponding to the target motion change intensity can all be used as video beat points 102. In some embodiments, the video beat points can also be determined at the granularity of the frame. For example, the first frame, the middle frame, or the last frame can be selected from the multiple frames corresponding to the target motion change intensity as the video beat point 102.

[0057] In some embodiments, the determined target motion change strength may be relatively large, and the video beat extraction system 110 may regard the video beat point determined from the frame corresponding to the target motion change strength as at least one candidate video beat point. Then, the video beat extraction system 110 may select at least one video beat point 102 from the at least one candidate video beat point based on beat dynamic programming.

[0058] For example, in a rhythmic dance, if all candidate beat points are used as video beat points 102, the resulting merged video may cause discomfort to viewers due to the overly messy rhythm. In some embodiments, candidate video beat points are screened based on such a rule: a threshold number of video beat points within a certain duration. For example, within a duration of 1 second, the threshold number of video beat points is 2. If there are three candidate beat points, the first and second ones at the front can be selected.

[0059] Through the above-mentioned exemplary embodiments, the video beat extraction system 110 calculates the object motion information between adjacent frames of the video according to the optical flow, and further calculates the video beat point 102 according to the change of the motion information. In this way, the video beat point extraction based on the video content is realized, and the application in the automatic video editing and synthesis can significantly improve the efficiency of video production, and increase the coordination, rhythm and fluency of the video.

[0060] By extracting accurate motion information and the degree of motion change, the video beat extraction disclosed in the present invention is also particularly suitable for accurate beat extraction in videos where the motion of the object changes a lot but the large-area background changes little. For example, in a dance video, the changes in the movements of the characters are large, but the overall picture changes little. The traditional image analysis method based on scene switching is often ineffective in extracting video beats, but accurate extraction can be achieved based on the embodiments of the present invention.

[0061] Back to Figure 1After obtaining the video beat points 102, they can be stored in the video beat library 140 for subsequent use. When making the merged video 105, the video beat points 102 can be mapped to the audio beat points 104 of the target audio 103 through the beat point matching system 130, and then the target video 101 and the target audio 103 are merged to obtain the merged video 105. Figure 4 FIG. 4 is a flowchart showing a process 400 for producing a merged video according to some embodiments of the present disclosure. The process 400 may be implemented in Figure 1 environment 100.

[0062] In block 410, the video beat extraction system 110 determines one or more video beat points 102 in the target video 101. The video beat points 102 may be determined according to the above-described exemplary embodiments, for example, a video beat point sequence V beat =[t1,t2,…,t m ].

[0063] In block 420, the audio beat extraction system 120 determines one or more audio beat points 104 in the target audio 103. For example, the audio beat points 104 are obtained by a common algorithm for music beat detection, such as an audio beat point sequence M. beat =[T1,T2,...,T l ].

[0064] In block 430, the beat point matching system 130 determines the mapping relationship between the video beat points 102 and the audio beat points. beat and audio beat sequence M beat are all a set of time series. In order to generate the merged video 105, the visual beat point sequence V beat To the audio beat point sequence M beat Alignment, that is, calculating the video frames corresponding to the audio beat points. For example, align the number of audio beats with the number of video beats, and the number of beats retained after processing is the smaller value of the number of audio beats and the number of video beats, that is, k=min(l,m). In this way, if the number of video beats is more than k, the number of video beats can be deleted. The same is true for audio beats. In some embodiments, in order to ensure that the audio is not distorted, the number of beats k retained after processing is the number of audio beat points l.

[0065] Further, for the video beat point sequence and the audio beat point sequence that are aligned in number, the beat point matching system 130 can construct the video beat point sequence V beat To the audio beat point sequence M beatThe mapping function f can indicate which audio beat point each video beat point is mapped to. In some embodiments, in order to ensure that the time points at the beat points are accurately matched, the mapping function f can be designed as a piecewise linear function.

[0066] Figure 5 A schematic diagram of an example 500 of a mapping relationship according to some embodiments of the present disclosure is shown. Example 500 shows a video beat point sequence V beat To the audio beat point sequence M beat The vertical axis is the video beat point sequence V beat =[t1,t2,t3,...,t k ], the horizontal axis is the audio beat point sequence M beat =[T1,T2,T3,...,T k ]. As can be seen from the figure, the duration of each segment is not the same. While ensuring that the audio is not distorted, the video segment (i.e., the duration between adjacent video beat points) can be lengthened or compressed, for example, by speeding up or slowing down the playback speed.

[0067] In block 440, the beat point matching system 130 merges the target video 101 and the target audio 103 according to the mapping relationship to obtain a merged video. The time track of the merged video 105 is the same as the time track of the target audio 104, so it is only necessary to calculate the video time corresponding to the current time according to the time track of the target audio 104 according to the mapping relationship to obtain the picture frame. In this way, the target video frame sequence and the audio track are synthesized to obtain the merged video 105 after the beat points are aligned.

[0068] In summary, the various embodiments of the present disclosure align the visual beat points with the music beat points in chronological order, determine the time period for each visual beat segment to be displayed in the final merged video while ensuring that the audio playback speed remains unchanged, and adjust the playback rate of the video screen to achieve the matching of the two time series. Therefore, based on the principle of optical flow change to analyze the changes in video content, the video beat points can be effectively extracted, and the effect of the synthesized card point video is greatly improved. The card point success rate on the tested merged video reached 70%. According to user evaluation, the availability rate exceeds 90%, which is suitable for application in scenarios such as automatic card point editing of videos.

[0069] Figure 6 FIG. 6 is a schematic structural block diagram of a video processing apparatus 600 according to some embodiments of the present disclosure. The apparatus 600 may be implemented as or included in Figure 1 Each module / component in the device 600 may be implemented by hardware, software, firmware or any combination thereof.

[0070] As shown in the figure, the device 600 includes an optical flow feature determination module 610, which is configured to determine multiple optical flow features between adjacent frames in multiple frames of the target video, and the optical flow features indicate the motion direction and motion amplitude of multiple points in the frame. The device 600 also includes a sequence determination module 620, which is configured to determine the sequence of motion change intensity of multiple frames over time based on multiple optical flow features. The device 600 also includes a selection module 630, which is configured to select at least one target motion change intensity from the sequence of motion change intensity. The device 600 also includes a video beat point determination module 640, which is configured to determine at least one video beat point in the target video based on at least one frame corresponding to at least one target motion change intensity.

[0071] In some embodiments, the sequence determination module 620 is further configured to: determine a plurality of optical flow change features based on differences between adjacent optical flow features in the plurality of optical flow features; and determine a sequence of motion change intensities based on the plurality of optical flow change features.

[0072] In some embodiments, the sequence determination module 620 is further configured to: divide the range of motion directions into multiple direction intervals; determine the number of points in each of the multiple optical flow features that fall into the multiple direction intervals, and obtain multiple optical flow histograms corresponding to each of the multiple optical flow features; and determine multiple optical flow change features based on multiple differences between adjacent optical flow histograms in the multiple optical flow histograms, the optical flow change features indicating the difference in the number of adjacent optical flow histograms in each of the multiple direction intervals.

[0073] In some embodiments, the sequence determination module 620 is further configured to: for a given optical flow feature among multiple optical flow features, filter points whose motion amplitude is lower than an amplitude threshold from the given optical flow feature to obtain a filtered optical flow feature; and determine the number of points in the filtered optical flow feature that fall into multiple direction intervals to obtain an optical flow histogram corresponding to the given optical flow feature.

[0074] In some embodiments, the sequence determination module 620 is further configured to: for each optical flow change feature among the multiple optical flow change features, determine the motion change intensity corresponding to the optical flow change feature by adding multiple differences indicated by the optical flow change feature in multiple direction intervals.

[0075] In some embodiments, the selection module 630 is further configured to select at least one intensity extreme value from the sequence of motion variation intensities.

[0076] In some embodiments, the video beat point determination module 640 is further configured to determine at least one candidate video beat point based on at least one frame corresponding to at least one target motion change intensity; and select at least one video beat point from at least one candidate video beat point based on beat dynamic programming.

[0077] In some embodiments, the device 600 also includes: an audio beat point determination module, configured to determine at least one audio beat point in the target audio; a mapping relationship determination module, configured to determine the mapping relationship between at least one video beat point and at least one audio beat point; and a merging module, configured to merge the target video and the target audio based on the mapping relationship to obtain a merged video.

[0078] Figure 7 1 shows a block diagram of an electronic device 700 in which one or more embodiments of the present disclosure may be implemented. It should be understood that Figure 7 The electronic device 700 shown is merely exemplary and should not constitute any limitation on the functionality and scope of the embodiments described herein. Figure 7 The electronic device 700 shown can be used to implement Figure 1 The electronic device 700 may include or be implemented as one or more of the video beat extraction system 110, the audio beat extraction system 120, and the beat point matching system 130. Figure 6 Device 600.

[0079] like Figure 7 As shown, the electronic device 700 is in the form of a general-purpose computing device. The components of the electronic device 700 may include, but are not limited to, one or more processors or processing units 710, a memory 720, a storage device 730, one or more communication units 740, one or more input devices 750, and one or more output devices 760. The processing unit 710 may be an actual or virtual processor and is capable of performing various processes according to a program stored in the memory 720. In a multi-processor system, multiple processing units execute computer executable instructions in parallel to improve the parallel processing capabilities of the electronic device 700.

[0080] The electronic device 700 typically includes a plurality of computer storage media. Such media may be any available media accessible to the electronic device 700, including but not limited to volatile and non-volatile media, removable and non-removable media. The memory 720 may be a volatile memory (e.g., a register, a cache, a random access memory (RAM)), a non-volatile memory (e.g., a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), flash memory), or some combination thereof. The storage device 730 may be a removable or non-removable medium, and may include a machine-readable medium, such as a flash drive, a disk, or any other medium, which may be capable of being used to store information and / or data (e.g., training data for training) and may be accessed within the electronic device 700.

[0081] The electronic device 700 may further include additional removable / non-removable, volatile / non-volatile storage media. Figure 7 As shown in , a disk drive for reading or writing from a removable, non-volatile disk (e.g., a "floppy disk") and an optical drive for reading or writing from a removable, non-volatile optical disk may be provided. In these cases, each drive may be connected to the bus (not shown) by one or more data media interfaces. The memory 720 may include a computer program product 725 having one or more program modules that are configured to perform various methods or actions of various embodiments of the present disclosure.

[0082] The communication unit 740 implements communication with other electronic devices through a communication medium. Additionally, the functions of the components of the electronic device 700 can be implemented with a single computing cluster or multiple computing machines that can communicate through a communication connection. Therefore, the electronic device 700 can operate in a networked environment using a logical connection with one or more other servers, a network personal computer (PC), or another network node.

[0083] The input device 750 may be one or more input devices, such as a mouse, a keyboard, a tracking ball, etc. The output device 760 may be one or more output devices, such as a display, a speaker, a printer, etc. The electronic device 700 may also communicate with one or more external devices (not shown) through the communication unit 740 as needed, such as a storage device, a display device, etc., communicate with one or more devices that allow a user to interact with the electronic device 700, or communicate with any device that allows the electronic device 700 to communicate with one or more other electronic devices (e.g., a network card, a modem, etc.). Such communication may be performed via an input / output (I / O) interface (not shown).

[0084] According to an exemplary implementation of the present disclosure, a computer-readable storage medium is provided, on which computer-executable instructions are stored, wherein the computer-executable instructions are executed by a processor to implement the method described above. According to an exemplary implementation of the present disclosure, a computer program product is also provided, which is tangibly stored on a non-transitory computer-readable medium and includes computer-executable instructions, and the computer-executable instructions are executed by a processor to implement the method described above.

[0085] Various aspects of the present disclosure are described herein with reference to the flowcharts and / or block diagrams of the methods, devices, equipment, and computer program products implemented according to the present disclosure. It should be understood that each box in the flowchart and / or block diagram and the combination of each box in the flowchart and / or block diagram can be implemented by computer-readable program instructions.

[0086] These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processing unit of the computer or other programmable data processing device, a device that implements the functions / actions specified in one or more boxes in the flowchart and / or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0087] Computer-readable program instructions can be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operational steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, so that the instructions executed on the computer, other programmable data processing apparatus, or other device implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0088] The flow chart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system, method and computer program product according to multiple implementations of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a part of a module, program segment or instruction, and a part of a module, program segment or instruction includes one or more executable instructions for realizing the logical function of the specification. In some implementations as replacements, the function marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous square boxes can actually be executed substantially in parallel, and they can sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be realized by a special hardware-based system that performs the function or action of the specification, or can be realized by a combination of special hardware and computer instructions.

[0089] The above descriptions of various implementations of the present disclosure are exemplary, non-exhaustive, and not limited to the disclosed implementations. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described implementations. The selection of terms used herein is intended to best explain the principles of the implementations, practical applications, or improvements to the technology in the market, or to enable other persons of ordinary skill in the art to understand the various implementations disclosed herein.

Claims

1. A video processing method, comprising: For a plurality of frames of a target video, determining a plurality of optical flow features between adjacent frames in the plurality of frames, the optical flow features indicating movement directions and movement amplitudes of a plurality of points in the frames; Based on the multiple optical flow features, a sequence of motion change strengths of the multiple frames over time is determined, wherein determining the sequence of motion change strengths comprises: The sequence of motion change intensities is determined based on a plurality of optical flow change features, wherein the plurality of optical flow change features are determined in the following manner: Divide the motion direction range into multiple direction intervals. Determine the number of points in each of the plurality of optical flow features that fall into the plurality of direction intervals, and obtain a plurality of optical flow histograms corresponding to each of the plurality of optical flow features; and Determine the plurality of optical flow variation features based on a plurality of differences between adjacent optical flow histograms in the plurality of optical flow histograms, the optical flow variation features indicating a difference in the number of points of adjacent optical flow histograms in each of the plurality of direction intervals; and Wherein, based on the multiple optical flow change features, determining the sequence of the motion change intensity comprises: For each optical flow change feature of the multiple optical flow change features, determining a motion change intensity corresponding to the optical flow change feature by adding a plurality of differences indicated by the optical flow change feature in the multiple direction intervals; selecting at least one target motion change intensity from said sequence of motion change intensities; and At least one video beat point in the target video is determined based on at least one frame corresponding to the at least one target motion change intensity.

2. The method according to claim 1, wherein determining the number of points in each of the plurality of optical flow features that fall into the plurality of direction intervals comprises: For a given optical flow feature among the plurality of optical flow features, filtering points whose motion amplitude is lower than an amplitude threshold from the given optical flow feature to obtain a filtered optical flow feature; as well as The number of points in the filtered optical flow feature that fall into the multiple direction intervals is determined to obtain an optical flow histogram corresponding to the given optical flow feature.

3. The method according to claim 1, wherein the selecting at least one target motion change intensity comprises: At least one intensity extreme is selected from the sequence of motion variation intensities.

4. The method according to claim 1, wherein determining the at least one video beat point comprises: Determine at least one candidate video beat point based on the at least one frame corresponding to the at least one target motion change intensity; as well as Based on beat dynamic programming, the at least one video beat point is selected from the at least one candidate video beat point.

5. The method according to claim 1, further comprising: Determining at least one audio beat point in the target audio; Determine a mapping relationship between the at least one video beat point and the at least one audio beat point; as well as Based on the mapping relationship, the target video and the target audio are merged to obtain a merged video.

6. A device for video processing, comprising: An optical flow feature determination module is configured to determine, for a plurality of frames of a target video, a plurality of optical flow features between adjacent frames in the plurality of frames, the optical flow features indicating a motion direction and a motion amplitude of a plurality of points in the frames; A sequence determination module is configured to determine a sequence of motion change intensities of the multiple frames over time based on the multiple optical flow features, and the sequence determination module is further configured to: The sequence of motion change intensities is determined based on a plurality of optical flow change features, wherein the optical flow change features are determined in the following manner: Divide the motion direction range into multiple direction intervals. Determine the number of points in each of the plurality of optical flow features that fall into the plurality of direction intervals, and obtain a plurality of optical flow histograms corresponding to each of the plurality of optical flow features; and Determine the plurality of optical flow variation features based on a plurality of differences between adjacent optical flow histograms in the plurality of optical flow histograms, the optical flow variation features indicating a difference in the number of points of adjacent optical flow histograms in each of the plurality of direction intervals; and Wherein the sequence determination module is further configured to: For each optical flow change feature of the multiple optical flow change features, determining a motion change intensity corresponding to the optical flow change feature by adding a plurality of differences indicated by the optical flow change feature in the multiple direction intervals; A selection module configured to select at least one target motion change intensity from the sequence of motion change intensities; as well as The video beat point determination module is configured to determine at least one video beat point in the target video based on at least one frame corresponding to the intensity of the at least one target motion change.

7. An electronic device comprising: at least one processing unit; as well as At least one memory, the at least one memory being coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit, the instructions causing the device to perform the method according to any one of claims 1 to 5 when executed by the at least one processing unit.

8. A computer-readable storage medium having a computer program stored thereon, wherein the computer program implements the method according to any one of claims 1 to 5 when executed by a processor.

9. A computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the method according to any one of claims 1 to 5 is implemented.

Images