Video splitting method and system applying short video to form video by one key

Abstract

The invention provides a video mirror splitting method and system applying a short video to form a piece by one key, and the method comprises the steps: S101, obtaining all frames in a target video, and extracting the SIFT features and HSV color histogram features of each frame; s102, aiming at each frame, taking the frame as an end, and according to the SIFT feature difference degree and the HSV color histogram feature difference degree, calculating the difference degree between the frame and other frames in the window with the size of k; s103, according to the difference degree between the current frame and other frames in the window, obtaining a split score of the current frame, and introducing inter-frame distance attenuation into calculation of the split score; s104, determining a splitting node according to the splitting score and a splitting threshold, and splitting the target video based on the splitting node; wherein the split threshold is dynamically adjusted through the variance of the split score in the window. The method can better adapt to the characteristics of different types of videos, improves the accuracy and robustness of lens splitting, and enables the lens splitting to be more reasonable and stable.

Description

Technical Field

[0001] The present invention relates to the field of information technology, and in particular to a video storyboard method and system for creating a short video with one click. Background Art

[0002] The core challenge of video automatic storyboarding technology is how to accurately identify the change points of video content to achieve intelligent scene segmentation. The complexity and diversity of video content make it difficult to extract features and calculate differences. For example, in situations such as light changes, camera movement, or object motion, how to effectively distinguish between true scene switching and continuous content changes is a key issue.

[0003] In addition, differences in video length, resolution, and frame rate also increase the difficulty of automatic storyboarding. In large-scale video processing scenarios, how to ensure the accuracy of storyboarding while improving processing efficiency is also an important challenge. Summary of the invention

[0004] In view of this, an object of the present invention is to provide a video storyboard method and system for one-click generation of short videos, so as to improve the above-mentioned problems.

[0005] The embodiment of the present invention provides a video storyboard method for one-click formation of short videos, comprising: S101, acquiring all frames in a target video, and extracting SIFT features and HSV color histogram features of each frame; S102, for each frame, taking the frame as the end, calculating the difference between the frame and other frames in a window of size k according to SIFT feature differences and HSV color histogram feature differences; S103, obtaining a storyboard score of the current frame according to the difference with other frames in the window, and the calculation of the storyboard score introduces inter-frame distance attenuation; S104, determining a storyboard node according to the storyboard score and a storyboard threshold, so as to storyboard the target video based on the storyboard node; wherein the storyboard threshold is dynamically adjusted by the variance of the storyboard score in the window.

[0006] Preferably, in step S101: the SIFT feature F of the i-th frame SIFT,i The calculation method is as follows:

[0007] Where N represents the total number of frames of the target video; The calculation process of HSV color histogram features is as follows: Convert each frame image to HSV space

[0008] Where I(i,j) represents the pixel in the i-th row and j-th column of the input image; Calculate the HSV color histogram features in HSV space .

[0009] Preferably, step S102 specifically includes: using a nearest neighbor matching algorithm to find matching pairs of two sets of SIFT features:

[0010] Among them, the matching condition is represented by Lowe's ratio test; Keep the matching pairs that meet the conditions to obtain the set Mgood, and calculate the similarity of SIFT features based on the set Mgood:

[0011] Determine the SIFT feature difference D based on the complement of the SIFT feature similarity SIFT ; According to the chi-square distance of the HSV color histogram features of the two frames, the HSV color histogram difference D is expressed CHI ; According to the SIFT feature difference D SIFT Difference D from HSV color histogram CHI Calculate the final difference D between the two frames of images.

[0012] Preferably, the HSV color histogram difference D CHI The calculation formula is:

[0013] Preferably, the calculation formula of the final difference degree D is: .

[0014] Preferably, in step S103, the storyboard score is calculated as follows:

[0015] Among them, S t is the score of the tth frame, and λ is the attenuation coefficient, which is used to introduce the relative distance information between frames.

[0016] Preferably, step S104 specifically includes: For the t-th frame image, calculate the variance of the score of the previous m frames

[0017] Among them, V t is the variance of the tth frame, S i is the score of the i-th frame, μ t is the average score of the previous m frames, where m is the window size; Determine the storyboard threshold based on the variance of the storyboard scores of the previous m frames T t =T base +p*V t Among them, T t is the frame splitting threshold of the tth frame, T base is the basic threshold, p is a constant.

[0018] The embodiment of the present invention further provides a video storyboard system for forming a short video into a film with one click, which includes: A feature extraction module is used to obtain all frames in a target video and extract SIFT features and HSV color histogram features of each frame; a difference calculation module is used to calculate the difference between each frame and other frames in a window of size k based on SIFT feature differences and HSV color histogram feature differences, with the frame as the end; a storyboard score calculation module is used to obtain a storyboard score of the current frame based on the difference with other frames in the window, and the calculation of the storyboard score introduces inter-frame distance attenuation; a storyboard module is used to determine a storyboard node based on the storyboard score and a storyboard threshold, so as to storyboard the target video based on the storyboard node; wherein the storyboard threshold is dynamically adjusted by the variance of the storyboard score in the window.

[0019] The technical solution provided by the embodiment of the present invention may have the following beneficial effects: The present invention extracts SIFT features and HSV color histogram features of video frames, calculates the feature difference between the current frame and the previous k frames, and performs weighted summation on the difference to obtain a storyboard score. The weight decays as the distance between frames increases to highlight the influence of recent frames. By calculating the variance and average of the storyboard scores of the previous m frames, the storyboard threshold of the current frame is dynamically determined, and the frames whose storyboard scores exceed the threshold are determined as storyboard nodes, thereby dividing the video into multiple segments. This method can adaptively detect changes in video content, accurately identify scene switching points, and realize intelligent storyboarding of videos. Compared with the traditional fixed threshold method, the present invention can better adapt to the characteristics of different types of videos, improve the accuracy and robustness of storyboards, and make storyboards more reasonable and stable. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 A flowchart of a video storyboard method for creating a short video with one click provided in the first embodiment of the present invention.

[0021] Figure 2 A structural diagram of a video storyboard system for creating short videos with one click, provided in accordance with the second embodiment of the present invention. DETAILED DESCRIPTION

[0022] In order to further understand the content of the present invention, the present invention is described in detail in conjunction with the accompanying drawings and embodiments. The present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It is understood that the specific embodiments described herein are only used to explain the relevant inventions, rather than to limit the invention. It is also necessary to explain that, for ease of description, only the parts related to the invention are shown in the accompanying drawings.

[0023] like Figure 1 The first embodiment of the present invention provides a video storyboard method using short videos to form a film in one click, which can be executed by a video storyboard device using short videos to form a film in one click (hereinafter referred to as a storyboard device), and in particular, is executed by one or more processors in the storyboard device to implement the following steps: Step S101, obtain all frames in the target video, and extract SIFT features and HSV color histogram features of each frame.

[0024] In this embodiment, the split mirror device may be a device with data processing capability such as a smart mobile terminal, a laptop computer, a desktop computer or a server, and the present invention does not make a specific selection.

[0025] In this embodiment, for the target video to be divided into two parts, assuming that the target video contains N frames of images, SIFT features are first extracted for each frame of the target video to obtain a SIFT feature set corresponding to each frame of the image. Among them, the SIFT feature F of the i-th frame SIFT,i The calculation method is as follows:

[0026] Then, for each frame of the target video, convert the image to the HSV color space and get:

[0027] Next, the color histogram features in the HSV space are calculated to obtain the HSV color histogram features corresponding to each frame of the image. ist HS The calculation formula is: .

[0028] S102 , for each frame, taking this frame as the end, calculating the difference between this frame and other frames in a window of size k according to the SIFT feature difference and the HSV color histogram feature difference.

[0029] Specifically, first: Use the nearest neighbor matching algorithm to find matching pairs of two sets of SIFT features:

[0030] Here, the matching condition is represented by Lowe's ratio test.

[0031] Keep the matching pairs that meet the conditions to obtain the set Mgood, and calculate the similarity of SIFT features based on the set Mgood:

[0032] Determine the SIFT feature difference D based on the complement of the SIFT feature similarity SIFT ; According to the chi-square distance of the HSV color histogram features of the two frames, the HSV color histogram difference D is expressed CHI ;

[0033] According to the SIFT feature difference D SIFT Difference D from HSV color histogram CHI Calculate the final difference D between the two frames of images: .

[0034] S103, obtaining a storyboard score of the current frame according to the difference between the current frame and other frames in the window, wherein the calculation of the storyboard score introduces inter-frame distance attenuation.

[0035] The score calculation method for storyboard is as follows:

[0036] Among them, S t is the score of the tth frame, and λ is the attenuation coefficient, which is used to introduce the relative distance information between frames.

[0037] S104, determining a storyboard node according to the storyboard score and a preset storyboard threshold, so as to storyboard the target video based on the storyboard node; wherein the storyboard threshold is dynamically adjusted by the variance of the storyboard score in the window.

[0038] Among them, for the t-th frame image, calculate the variance of the score of the previous m frames

[0039] V t is the variance of the tth frame, S i is the score of the i-th frame, μ t is the average score of the previous m frames, where m is the window size; Determine the threshold based on the variance of the storyboard scores of the previous m frames T t =Tbase +p*V t Among them, T t is the threshold of the tth frame, T base is the basic threshold, p is a constant.

[0040] For example, for a video containing 1000 frames, SIFT features and HSV color histogram features are extracted for each frame. The 500th frame is taken as the current frame, and the images of the previous 100 frames are obtained. For the 500th frame and each of the previous 100 frames, the nearest neighbor matching algorithm is used to find the matching pairs of SIFT features, and the matching pairs are screened by Lowe's ratio test. If the matching pairs that meet the conditions account for more than 80% of the total number of features, the SIFT similarity is 0.8 and the difference is 0.2.

[0041] At the same time, the chi-square distance of the two frames of HSV color histograms is calculated to obtain the difference D of the HSV color histograms. CHI The final difference D is the arithmetic mean of the SIFT difference and the color histogram difference.

[0042] Assume that the distance attenuation coefficient is 0.9, and calculate the score of the 500th frame according to the formula. Get the score of the 50 frames before the 500th frame, and calculate the mean and variance. If the constant p =1, the variance is 0.1, and the basic threshold is 0.5. The storyboard threshold of the 500th frame is 0.5+0.1*1=0.6. Compare the storyboard score of the 500th frame with the storyboard threshold. If it is greater than the storyboard threshold, it is determined as a storyboard node. Repeat the above process until all frames are traversed, and finally obtain the storyboard node sequence of the video.

[0043] In this embodiment, after the storyboard node sequence is obtained, the target video can be divided into a plurality of video segments according to the storyboard node sequence, and each of the video segments includes an image frame located between two adjacent storyboard nodes.

[0044] This embodiment extracts SIFT features and HSV color histogram features of video frames, calculates the feature difference between the current frame and the previous k frames, and performs weighted summation on the difference to obtain the storyboard score. The weight decays as the distance between frames increases to highlight the influence of recent frames. By calculating the variance and average of the storyboard scores of the previous m frames, the storyboard threshold of the current frame is dynamically determined, and the frames whose storyboard scores exceed the threshold are determined as storyboard nodes, thereby dividing the video into multiple segments. This method can adaptively detect changes in video content, accurately identify scene switching points, and realize intelligent storyboarding of videos. Compared with the traditional fixed threshold method, the present invention can better adapt to the characteristics of different types of videos, improve the accuracy and robustness of storyboards, and make storyboards more reasonable and stable.

[0045] See also Figure 2 The second embodiment of the present invention further provides a video storyboard system for forming a short video into a film with one click, which includes: The feature extraction module 210 is used to obtain all frames in the target video and extract the SIFT features and HSV color histogram features of each frame; the difference calculation module 220 is used to calculate the difference between each frame and other frames in a window of size k based on the SIFT feature difference and the HSV color histogram feature difference, with the frame as the end; the storyboard score calculation module 230 is used to obtain the storyboard score of the current frame based on the difference with other frames in the window, and the calculation of the storyboard score introduces inter-frame distance attenuation; the storyboard module 240 is used to determine the storyboard node based on the storyboard score and the storyboard threshold, so as to storyboard the target video based on the storyboard node; wherein the storyboard threshold is dynamically adjusted by the variance of the storyboard score in the window.

[0046] The third embodiment of the present invention also provides a video storyboard device for using short videos to create a film with one click, which includes a memory and a processor, wherein the memory stores a computer program, and the computer program can be executed by the processor to implement the video storyboard method for using short videos to create a film with one click as described above.

[0047] The fourth embodiment of the present invention further provides a computer-readable storage medium storing a computer program, wherein the computer program can be executed by the processor to implement the video storyboarding method of applying short videos to form a film with one click as described above.

[0048] Exemplarily, the computer program described in the third embodiment and the fourth embodiment of the present invention can be divided into one or more modules, and the one or more modules are stored in the memory and executed by the processor to complete the present invention. The one or more modules can be a series of computer program instruction segments that can complete specific functions, and the instruction segments are used to describe the execution process of the computer program in the error correction device. For example, the system described in the second embodiment of the present invention.

[0049] The processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor, etc. The processor is the control center of the error correction method, and uses various interfaces and lines to connect the various parts of the entire method for realizing high-confidence intelligent live broadcast reply.

[0050] The memory can be used to store the computer program and / or module, and the processor realizes various functions of an error correction method by running or executing the computer program and / or module stored in the memory, and calling the data stored in the memory. The memory can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, an application required for at least one function (such as a sound playback function, a text conversion function, etc.), etc.; the data storage area can store data created according to the use of the mobile phone (such as audio data, text message data, etc.), etc. In addition, the memory can include a high-speed random access memory, and can also include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.

[0051] Wherein, if the implemented module is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on such an understanding, the present invention implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Wherein, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electric carrier signal, telecommunication signal and software distribution medium. It should be noted that the content contained in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electric carrier signals and telecommunication signals.

[0052] It should be noted that the device embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. In addition, in the accompanying drawings of the device embodiments provided by the present invention, the connection relationship between the modules indicates that there is a communication connection between them, which may be specifically implemented as one or more communication buses or signal lines. A person of ordinary skill in the art may understand and implement it without paying any creative effort.

[0053] The above are only preferred implementations of the present invention. The protection scope of the present invention is not limited to the above embodiments. All technical solutions under the concept of the present invention belong to the protection scope of the present invention.

Claims

1. A video storyboard method for forming a short video into a film with one click, characterized in that: include: S101, obtain all frames in the target video, and extract SIFT features and HSV color histogram features of each frame; S102, for each frame, take the frame as the end, calculate the difference between it and other frames in a window of size k according to the SIFT feature difference and the HSV color histogram feature difference; S103, obtain the storyboard score of the current frame according to the difference with other frames in the window, and the calculation of the storyboard score introduces inter-frame distance attenuation; S104, determine the storyboard node according to the storyboard score and the storyboard threshold, so as to storyboard the target video based on the storyboard node; wherein the storyboard threshold is dynamically adjusted by the variance of the storyboard score in the window.

2. The video storyboard method for using short videos to create a film with one click according to claim 1, characterized in that: In step S101: SIFT feature F of the i-th frame SIFT,i The calculation method is as follows: Where N represents the total number of frames of the target video; The calculation process of HSV color histogram features is as follows: Convert each frame image to HSV space Where I(i,j) represents the pixel in the i-th row and j-th column of the input image; Calculate the HSV color histogram features in HSV space: 。 3. The video storyboard method for using short videos to create a film with one click as claimed in claim 2, characterized in that: Step S102 specifically includes: using the nearest neighbor matching algorithm to find matching pairs of two sets of SIFT features: Among them, the matching condition is represented by Lowe's ratio test; Keep the matching pairs that meet the conditions to obtain the set Mgood, and calculate the similarity of SIFT features based on the set Mgood: Determine the SIFT feature difference D based on the complement of the SIFT feature similarity SIFT ; According to the chi-square distance of the HSV color histogram features of the two frames, the HSV color histogram difference D is expressed CHI ; According to the SIFT feature difference D SIFT Difference D from HSV color histogram CHI Calculate the final difference D between the two frames of images.

4. The video storyboard method for using short videos to create a film with one click as claimed in claim 3, characterized in that: HSV color histogram difference D CHI The calculation formula is:

5. The video storyboard method for using short videos to create a film with one click according to claim 3, characterized in that: The calculation formula of the final difference D is: 。 6. The video storyboard method for using short videos to create a film with one click according to claim 1, characterized in that: In step S103, the storyboard score is calculated as follows: Among them, S t is the score of the tth frame, and λ is the attenuation coefficient, which is used to introduce the relative distance information between frames.

7. The video storyboard method for using short videos to create a film with one click according to claim 6, characterized in that: Step S104 specifically includes: For the t-th frame image, calculate the variance of the score of the previous m frames Among them, V t is the variance of the tth frame, S i is the score of the i-th frame, μ t is the average score of the previous m frames, where m is the window size; The storyboard threshold is determined based on the variance of the storyboard scores of the previous m frames: T t =T base +p*V t Among them, T t is the frame splitting threshold of the tth frame, T base is the basic threshold, p is a constant.

8. A video storyboard system that uses short videos to create a film in one click, characterized in that: include: The feature extraction module is used to obtain all frames in the target video and extract the SIFT features and HSV color histogram features of each frame; A difference calculation module is used to calculate the difference between each frame and other frames in a window of size k based on SIFT feature difference and HSV color histogram feature difference for each frame, with the frame as the end; a storyboard score calculation module is used to obtain the storyboard score of the current frame based on the difference with other frames in the window, and the calculation of the storyboard score introduces inter-frame distance attenuation; a storyboard module is used to determine the storyboard node based on the storyboard score and the storyboard threshold, so as to storyboard the target video based on the storyboard node; wherein the storyboard threshold is dynamically adjusted by the variance of the storyboard score in the window.

Images