Video mirror moving analysis method and device and electronic equipment

Abstract

The embodiment of the invention provides a video mirror operation analysis method. The method comprises the following steps: extracting an optical flow graph between video frames of a video; and on the basis of the extracted optical flow graph between the video frames, analyzing the operation mirror type of the video by using a pre-trained convolutional neural network model. According to the video operation mirror analysis method and the optical flow analysis method based on the video frames, information irrelevant to motion in the video is filtered out, the complexity of an operation mirror analysis algorithm is reduced, the operation mirror is calculated only based on the motion information, and the precision of the operation mirror algorithm is improved. The embodiment of the invention further provides a video mirror moving analysis device and electronic equipment.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of video processing technology, and specifically, to a video camera movement analysis method, device, and electronic device. Background Art

[0002] AIGC technology is a pioneering application of current artificial intelligence technologies. Its application in video algorithms encompasses multiple approaches, including generative, editing, customization, and digital human video. In video generation algorithms, content consistency is difficult to ensure, especially when generating long videos, where visual jitter becomes noticeable. Therefore, AIGC algorithms require video camera analysis algorithms to enhance the coherence and stability of video content, reduce jitter, and ensure the continuity of action. Furthermore, AIGC algorithms require video camera algorithms for more refined video control, including precise control of camera movement and character movements. Furthermore, in video editing algorithms, AI-based algorithms splice videos from multiple sources together based on screenwriter requirements and content coherence. This often results in inconsistent camera directions in video clips, resulting in poor visual quality. Therefore, video camera analysis algorithms are needed to analyze video clips and avoid splicing together videos with inconsistent camera directions.

[0003] However, existing video camera movement analysis algorithms have problems such as insufficient training data, insufficient model robustness, inability to recognize multiple camera movement behaviors, and inability to eliminate target movement interference. Summary of the Invention

[0004] In response to the problems existing in the above-mentioned prior art, the embodiments of the present application provide a video camera movement analysis method, device, and electronic device. Based on the optical flow analysis method between video frames, it filters out information in the video that is not related to motion, reduces the complexity of the camera movement analysis algorithm, and calculates the camera movement only based on motion information, thereby improving the accuracy of the camera movement algorithm.

[0005] In a first aspect, an embodiment of the present application provides a video camera movement analysis method, comprising the following steps:

[0006] extracting an optical flow map between video frames of the video; and

[0007] Based on the extracted optical flow map between the video frames, a pre-trained convolutional neural network model is used to analyze the camera movement type of the video.

[0008] Furthermore, extracting the optical flow map between video frames includes:

[0009] The video frames are extracted according to a specific frame rate, and the video frame sequence is extracted based on an optical flow recognition model to obtain an optical flow map between the video frames.

[0010] Furthermore, the pre-training process of the convolutional neural network model includes:

[0011] Extract the video frames at a specific frame rate, and extract the optical flow maps between the video frames as a training sample set; and

[0012] Construct a convolutional neural network model, and input the training sample set to train the convolutional neural network model.

[0013] Further, the extracting the video frames at a specific frame rate, and extracting the optical flow maps between the video frames as a training sample set includes:

[0014] Collect videos of various camera movement types and perform reverse operations to expand the training data set; and

[0015] Extract the optical flow maps between the video frames of the training data set based on an optical flow recognition model as the training sample set.

[0016] Further, the camera movement types of the video include upward movement, downward movement, leftward movement, rightward movement, forward movement, backward movement, clockwise rotation, counterclockwise rotation, and stillness.

[0017] Further, before the extracting the optical flow maps between the video frames of the video, it includes:

[0018] Decode the video into video frames and unify the sizes of the video frames.

[0019] Further, after the analyzing the camera movement type of the video based on the extracted optical flow maps between the video frames by using a pre-trained convolutional neural network model, it further includes:

[0020] Summarize the analysis results of the camera movement type of the video frames, and output the analysis results of the camera movement type at the timestamp level.

[0021] In a second aspect, an embodiment of the present application further provides a video camera movement analysis device, including:

[0022] An optical flow map extraction module for extracting the optical flow maps between the video frames of the video; and

[0023] A camera movement analysis module for analyzing the camera movement type of the video based on the extracted optical flow maps between the video frames by using a pre-trained convolutional neural network model.

[0024] In a third aspect, an embodiment of the present application further provides an electronic device, including a memory, a processor, and a computer program stored on the memory and executable on the processor, where the processor is used to implement the video camera movement analysis method according to the first aspect above when executing the program.

[0025] Fourthly, an embodiment of the present application also provides a computer-readable storage medium, on which a computer program is stored, and the computer program is used to implement the video camera movement analysis method according to the first aspect described above.

[0026] The embodiments of the present application bring the following beneficial effects:

[0027] In the video camera movement analysis method provided by the embodiments of the present application, first, the initial video is segmented into multiple scene segments to independently analyze the stability of each segment. For the scene segments determined to be unstable, the method further calculates the camera movement trajectory in the video frames. Finally, based on the obtained movement trajectory, the jittery video frames are converted into stable video frames, thus significantly improving the visual stability of the video. The video camera movement analysis method provided by the embodiments of the present application ensures effective processing of each unstable segment through detailed scene division and accurate jitter analysis. Its advantage lies in being able to handle complex scenes and large jitters, providing a smoother and more stable viewing experience for the audience. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the drawings in the following description are only some embodiments of the present application. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on the structures shown in these drawings.

[0029] Figure 1 It is a schematic flow framework diagram of the video camera movement analysis method provided by the embodiments of the present application;

[0030] Figure 2 It is a structural block diagram of the video camera movement analysis device provided by the embodiments of the present application;

[0031] Figure 3 It is a schematic structural diagram of the electronic device provided by the embodiments of the present application.

[0032] The realization, functional features and advantages of the purpose of the present application will be further described in conjunction with the embodiments and with reference to the drawings. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0033] The following will clearly and completely describe the technical solutions in the embodiments of the present application with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all of the embodiments. Based on the embodiments of the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present application.

[0034] In the description and claims of this application and the above-mentioned drawings, the terms "first" and "second" are only used for descriptive purposes and cannot be construed as indicating or implying relative importance or implicitly specifying the quantity of the indicated technical features. Thus, features defined with "first" and "second" may explicitly or implicitly include one or more of such features. In the description of this application, unless otherwise stated, the meaning of "a plurality" is two or more. For those of ordinary skill in the art, the specific meanings of the above terms in this application can be understood according to specific circumstances.

[0035] Referring to Figure 1 , Figure 1 is a flowchart framework of a video camera movement analysis method according to an embodiment of this application. As Figure 1 shown, the video camera movement analysis method according to an embodiment of this application includes the following steps:

[0036] S101: Extract the optical flow map between video frames of the video; and

[0037] S102: Based on the extracted optical flow map between video frames, use a pre-trained convolutional neural network model to analyze the camera movement type of the video.

[0038] Specifically, the optical flow map can show the moving direction and speed of pixel points in a video between consecutive video frames and is an important tool for understanding the dynamic content of the video. The optical flow map is usually represented in the form of a vector field, where each vector represents the moving direction and speed of a certain point in the image. The displacement of pixels between adjacent frames can be calculated using optical flow algorithms (such as the Lucas-Kanade method, the Farneback method, etc.) to generate the optical flow map between video frames. A convolutional neural network (CNN) model pre-trained with a large amount of data has powerful feature extraction and classification capabilities and can receive the optical flow map as input and intelligently identify the camera movement type of the video. Whether it is translation, zoom, rotation, or other complex camera movement methods, the model can give accurate and reliable prediction results.

[0039] In the video camera movement analysis method provided by the embodiment of this application, first, the optical flow map between video frames of the video is extracted, and then, based on the extracted optical flow map between video frames, a pre-trained convolutional neural network model is used to analyze the camera movement type of the video. The video camera movement analysis method provided by the embodiment of this application is based on the optical flow analysis method between video frames, filters out the information irrelevant to movement in the video, reduces the complexity of the camera movement analysis algorithm, and calculates the camera movement only based on movement information, improving the accuracy of the camera movement algorithm.

[0040] In addition, the video camera movement analysis method provided by the embodiments of the present application directly extracts the original motion information of the camera based on dense optical flow analysis, and then restores it to camera movement techniques. The video camera movement analysis method provided by the embodiments of the present application allows multiple camera movement behaviors to exist in one video, is applicable to various videos shot by non-professionals, and can also perform timestamp-level evaluation on the camera movement results of AI-generated videos. For the simultaneous movement of the picture target and the picture content, the existing methods cannot exclude the interference of the target movement on the recognition result. The video camera movement analysis method provided by the embodiments of the present application can significantly reflect the difference between the camera movement (picture content movement) and the target movement based on the extracted optical flow information, thereby improving the recognition accuracy. At the same time, compared with the existing method that jointly judges and outputs the camera movement result based on three neural networks, the video camera movement analysis method provided by the embodiments of the present application is based on the existing optical flow analysis model, and only needs to add a neural network model to complete the video camera movement analysis, and only needs to complete the training of one neural network model, which reduces the difficulty of algorithm optimization.

[0041] Further, in some embodiments of the present application, the extracting the optical flow map between video frames includes:

[0042] Extracting the video frames at a specific frame rate, and based on an optical flow recognition model, extracting the video frame sequence to obtain the optical flow map between the video frames.

[0043] Specifically, a series of video frames need to be evenly extracted from the video at a preset specific frame rate, so as to ensure that a sufficient number of key pictures in the video are captured, and at the same time avoid the excessive processing burden caused by too high a frame rate. For example, if 3 frames are extracted from a 1s video to calculate the optical flow, the fps is 3; if 10 frames are extracted from a 1s video to calculate the optical flow, the fps is 10. Based on this operation, the model can adapt to different types of camera movements with different speeds.

[0044] Subsequently, using a pre-trained optical flow recognition model, these extracted video frame sequences are deeply analyzed. The core function of the optical flow recognition model is that it can accurately calculate the displacement of pixel points between adjacent video frames, and these displacement information are integrated and visualized in the form of an optical flow map. Each vector in the optical flow map represents the movement direction and speed of a certain pixel point in the image from the previous frame to the next frame. The optical flow map provides us with intuitive and accurate information about the movement mode of the camera in the video. This information is conducive to the subsequent analysis of video camera movement types, can better understand the movement trajectory and speed change of the camera, and lays a good foundation for the subsequent analysis of camera movement types.

[0045] Further, in some embodiments of the present application, the pre-training process of the convolutional neural network model includes:

[0046] Extract the video frames at a specific frame rate and extract the optical flow maps between the video frames as a training sample set; and

[0047] Construct a convolutional neural network model and input the training sample set to train the convolutional neural network model.

[0048] Specifically, extract video frames from the video at a specific frame rate and use the optical flow algorithm to extract the optical flow maps between adjacent frames. These optical flow maps record the movement trajectories of pixel points in the video and reflect the movement mode of the camera, which are important sample sets for training the CNN model. Then, construct a convolutional neural network model, which consists of multiple convolutional layers, pooling layers, and fully connected layers, aiming to extract features from the input optical flow maps and classify them. To train this model, input the optical flow map sample set into the model, and through multiple iterations and adjustment of the model parameters, the model gradually learns the mapping relationship between the optical flow map and the camera movement type. During the training process, use a loss function to measure the prediction accuracy of the model and optimize the model through the backpropagation algorithm. After a large amount of training, the model gradually converges and the prediction ability is significantly improved.

[0049] Finally, a pre-trained CNN model can be obtained, which can receive new optical flow maps as input and accurately predict the camera movement type of the video, thus providing strong technical support for video camera movement analysis.

[0050] Furthermore, in some embodiments of the present application, the extracting the video frames at a specific frame rate and extracting the optical flow maps between the video frames as a training sample set includes:

[0051] Collect videos of multiple camera movement types and perform reverse operations to expand the training data set; and

[0052] Extract the optical flow maps between the video frames of the training data set based on the optical flow recognition model as the training sample set.

[0053] First, it is necessary to widely collect video materials covering multiple camera movement types. These videos not only have rich content but also diverse camera movement methods, ensuring the diversity and representativeness of the training samples. To further expand the training data set, reverse operations can also be taken, that is, play some videos in reverse, so as to effectively increase the number and diversity of training samples without increasing additional shooting costs.

[0054] Subsequently, using the pre-trained optical flow recognition model, these augmented training data sets are deeply processed. This model can accurately calculate the motion information of pixel points between adjacent video frames and integrate this information into an optical flow map. These optical flow maps not only intuitively show the motion trajectories of objects in the video, but more importantly, they imply the motion mode of the camera, providing information for the subsequent training of the CNN model. By using this sample set for training, a CNN model that can accurately identify the types of video camera movements can be obtained, providing strong support for video camera movement analysis.

[0055] Further, in some embodiments of the present application, the types of camera movements of the video include upward movement, downward movement, leftward movement, rightward movement, forward movement, backward movement, clockwise rotation, counterclockwise rotation, and stillness.

[0056] Specifically, as described above, in order to train the deep convolutional neural network model, a training sample set containing rich optical flow maps needs to be constructed. This sample set covers all target camera movement types, including upward movement, downward movement, leftward movement, rightward movement, forward movement, backward movement, clockwise rotation, counterclockwise rotation, and stillness. The high-quality sample set lays a solid foundation for the training of the CNN model, enabling the model to accurately identify various camera movement types.

[0057] Further, in some embodiments of the present application, before extracting the optical flow map between video frames of the video, it includes:

[0058] Decoding the video into video frames and unifying the sizes of the video frames.

[0059] Specifically, before extracting the optical flow map between video frames, the video needs to be preprocessed to ensure the smooth progress of subsequent steps. That is, decoding the video and splitting it into a series of independent video frames. This step is the basis of video processing, enabling frame-by-frame analysis of video content and providing the possibility for subsequent optical flow map extraction and camera movement type recognition. Immediately afterwards, in order to maintain the consistency and efficiency of processing, the sizes of all video frames need to be unified. Since different videos may be recorded at different resolutions, the sizes of video frames may vary. Unifying the sizes can not only ensure the correct application of the optical flow algorithm, but also improve the processing speed and accuracy. During the process of unifying the sizes, a suitable resolution is usually selected as the standard, and then other video frames are adjusted to this resolution through methods such as scaling and cropping.

[0060] Further, in some embodiments of the present application, after analyzing the types of camera movements of the video by using the pre-trained convolutional neural network model based on the extracted optical flow map between video frames, it further includes:

[0061] Summarize the analysis results of the video camera movement types of the video frames and output the analysis results of the video camera movement types at the timestamp level.

[0062] Specifically, after analyzing the camera movement types of the video by using the pre-trained convolutional neural network model based on the extracted optical flow maps between the video frames, it is also necessary to summarize and organize the analysis results. Specifically, arrange the camera movement types of each video frame in chronological order and label each camera movement type with the corresponding timestamp to clearly show the changes in the camera movement methods in the video and their occurrence times.

[0063] Finally, output the analysis results of the video camera movement types including timestamps. This result not only intuitively shows the distribution of various camera movement methods in the video but also provides the exact time information of their occurrences, providing strong support for fields such as video editing and special effect adding, and also providing data resources for subsequent applications and research.

[0064] Figure 2 It is the structural block diagram of the video camera movement analysis device 200 provided by the embodiments of the present application. As Figure 2 shown, the video camera movement analysis device 200 of the embodiments of the present application includes: an optical flow map extraction module 210 and a camera movement analysis module 220, where:

[0065] The optical flow map extraction module 210 is used to extract the optical flow maps between the video frames of the video; and

[0066] The camera movement analysis module 220 is used to analyze the camera movement types of the video by using the pre-trained convolutional neural network model based on the extracted optical flow maps between the video frames.

[0067] In the video camera movement analysis device provided by the embodiments of the present application, first extract the optical flow maps between the video frames of the video, and then analyze the camera movement types of the video by using the pre-trained convolutional neural network model based on the extracted optical flow maps between the video frames. The video camera movement analysis device provided by the embodiments of the present application is based on the optical flow analysis method between video frames, filters out the information irrelevant to the movement in the video, reduces the complexity of the camera movement analysis algorithm, and calculates the camera movement only based on the movement information, improving the accuracy of the camera movement algorithm.

[0068] It should be noted that the specific implementation manner of the video camera movement analysis device of the embodiments of the present application is similar to the specific implementation manner of the video camera movement analysis method of the embodiments of the present application. For details, please refer to the description in the method part and will not be elaborated here.

[0069] Figure 3 It is the structural schematic diagram of the electronic device 300 of the embodiments of the present application.

[0070] As Figure 3As shown in the figure, the electronic device 300 includes a central processing unit (CPU) 301, which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM) 302 or the program loaded from the storage section 302 into the random access memory (RAM) 303. In the RAM 303, various programs and data required for the operation of the electronic device 300 are also stored. The CPU 301, ROM 302, and RAM 303 are connected to each other via a bus 304. An input / output (I / O) interface 305 is also connected to the bus 304.

[0071] The following components are connected to the I / O interface 305: an input section 306 including a keyboard, a mouse, etc.; an output section 307 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc. and a speaker, etc.; a storage section 308 including a hard disk, etc.; and a communication section 309 including a network interface card such as a LAN card, a modem, etc. The communication section 309 performs communication processing via a network such as the Internet. A drive 310 is also connected to the I / O interface 305 as needed. A removable medium 311, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 310 as needed so that a computer program read from it can be installed into the storage section 308 as needed.

[0072] Specifically, according to an embodiment of the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present application includes a computer program product, which includes a computer program carried on a machine-readable medium, and the computer program includes program codes for performing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication section 309, and / or installed from the removable medium 311. When the computer program is executed by the central processing unit (CPU) 301, the above functions defined in the electronic device of the present application are executed.

[0073] It should be noted that the computer-readable medium shown in the present application can be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium can be, for example, but not limited to, an electronic device, device, or component of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination of the above. More specific examples of the computer-readable storage medium can include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0074] In the present application, a computer-readable storage medium may be any tangible medium that contains or stores a program, which can be used by or in conjunction with an instruction-executing electronic device, apparatus, or device. In the present application, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries computer-readable program code. Such a propagated data signal may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the foregoing. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in conjunction with an instruction-executing electronic device, apparatus, or device. The program code contained on a computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0075] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architectures, functions, and operations of a processing receiving device, method, and computer program product according to various embodiments of the present application. In this regard, each block in a flowchart or block diagram may represent a module, program segment, or part of code that contains one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than marked in the accompanying drawings. For example, two consecutive blocks shown may actually be executed substantially in parallel, and they may sometimes be executed in the reverse order, depending on the functions involved. It should also be noted that each block in the block diagram and / or flowchart, and combinations of blocks in the block diagram and / or flowchart, may be implemented by a dedicated hardware-based electronic device that performs the specified functions or operations, or may be implemented by a combination of dedicated hardware and computer instructions.

[0076] The units or modules described in the embodiments of the present application may be implemented in software or in hardware. The described units or modules may also be provided in a processor, and when the processor executes the program, it implements the video camera movement analysis method:

[0077] Extract the optical flow map between video frames of the video; and

[0078] Based on the extracted optical flow map between video frames, use a pre-trained convolutional neural network model to analyze the camera movement type of the video.

[0079] As another aspect, the present application also provides a computer-readable storage medium, which may be included in the electronic device described in the foregoing embodiments; or may exist alone without being assembled into the electronic device. The above computer-readable storage medium stores one or more programs, and when the foregoing program is executed by one or more processors to perform the video camera movement analysis method described in the present application:

[0080] Extract the optical flow map between video frames of the video; and

[0081] Based on the extracted optical flow map between video frames, use a pre-trained convolutional neural network model to analyze the camera movement type of the video.

[0082] As another aspect, the present application also provides a computer program product, which may be included in the electronic device described in the foregoing embodiments; or may exist alone without being assembled into the electronic device. The above computer program product stores one or more programs, and when the foregoing program is executed by one or more processors to perform the video camera movement analysis method described in the present application:

[0083] Extract the optical flow map between video frames of the video; and

[0084] Based on the extracted optical flow map between video frames, use a pre-trained convolutional neural network model to analyze the camera movement type of the video.

[0085] The foregoing are only the preferred embodiments of the present application, and do not limit the patent scope of the present application. Any equivalent structural transformation made by using the description of the present application's specification and drawings under the application concept of the present application, or direct / indirect application in other related technical fields, is included in the patent protection scope of the present application.

Claims

1. A video camera movement analysis method, characterized in that, Including the following steps: Extracting the optical flow map between video frames of the video; and Based on the extracted optical flow map between video frames, using a pre-trained convolutional neural network model to analyze the camera movement type of the video.

2. The video camera movement analysis method according to claim 1, wherein The extracting the optical flow map between video frames includes: Sampling the video frames at a specific frame rate and extracting the video frame sequence based on an optical flow recognition model to obtain the optical flow map between video frames.

3. The video camera movement analysis method according to claim 1, wherein The pre-training process of the convolutional neural network model includes: Sampling the video frames at a specific frame rate and extracting the optical flow map between video frames as a training sample set; and Constructing a convolutional neural network model and inputting the training sample set to train the convolutional neural network model.

4. The video camera movement analysis method according to claim 3, wherein, The sampling the video frames at a specific frame rate and extracting the optical flow map between video frames as a training sample set includes: Collecting videos of various camera movement types and performing reverse order operations to expand the training data set; and Extracting the optical flow map between video frames of the training data set based on an optical flow recognition model as the training sample set.

5. The video camera movement analysis method according to claim 4, wherein The camera movement types of the video include upward movement, downward movement, leftward movement, rightward movement, forward movement, backward movement, clockwise rotation, counterclockwise rotation, and stillness.

6. The video camera movement analysis method according to claim 1, wherein Before extracting the optical flow map between video frames of the video, it includes: Decoding the video into video frames and unifying the sizes of the video frames.

7. The video camera movement analysis method according to claim 1, wherein After analyzing the camera movement type of the video based on the extracted optical flow map between video frames using a pre-trained convolutional neural network model, it further includes: Summarizing the analysis results of the camera movement type of the video frames and outputting the analysis results of the camera movement type of the video at the timestamp level.

8. A video camera movement analysis device, characterized in that, Including: An optical flow map extraction module for extracting the optical flow map between video frames of the video; and A camera movement analysis module for analyzing the camera movement type of the video based on the extracted optical flow map between video frames using a pre-trained convolutional neural network model.

9. An electronic device, characterized in that, Including a memory, a processor, and a computer program stored on the memory and executable on the processor, where the processor is used to implement the video camera movement analysis method according to any one of claims 1-7 when executing the program.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, and the computer program is used to implement the video camera movement analysis method according to any one of claims 1-7.

Images