A method, apparatus, device and program product for detecting video shake

Abstract

The application provides a video shaking detection method, device, equipment and program product, and relates to the technical field of video processing. The video shaking detection method comprises the following steps: extracting a plurality of video frames arranged in time sequence in at least one video segment corresponding to an input video; obtaining an optical flow feature vector of the at least one video segment according to the plurality of video frames; determining a playing state of the at least one video segment according to the optical flow feature vector; and obtaining a video segment with shaking in the input video based on the playing state of the at least one video segment. According to the embodiment of the application, the time point with shaking in the complete video can be automatically detected.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of video processing, in particular to a video shaking detection method, device, equipment and program product. BACKGROUND

[0002] With the development of Internet content ecology, people's demand for high-quality videos is increasing. However, a large number of low-quality videos still flood the network, among which videos with dramatic camera shaking are easy to make viewers feel strong dizziness and even cause physical discomfort when playing.

[0003] The currently commonly used method for detecting shaking segments in a video is mainly traditional methods such as optical flow method, block matching method, feature point matching method and gray projection method, which set a threshold to judge the shaking state of the video. However, these methods have the problems of low generalization, difficulty in setting the threshold, and easy misjudgment in some special scenes (special effects, scene transitions). SUMMARY

[0004] According to an aspect of the present application, a video shaking detection method is provided, including: extracting a plurality of video frames arranged in time sequence in at least one video segment corresponding to an input video; obtaining an optical flow feature vector of the at least one video segment according to the plurality of video frames; determining a playing state of the at least one video segment according to the optical flow feature vector; and obtaining a video segment with shaking in the input video based on the playing state of the at least one video segment.

[0005] According to some embodiments, extracting a plurality of video frames arranged in time sequence in at least one video segment corresponding to an input video includes: dividing the input video into at least one video segment in time sequence with a preset number of consecutive video frames; and scaling the plurality of video frames in the at least one video segment according to a preset image size.

[0006] According to some embodiments, obtaining an optical flow feature vector of the at least one video segment according to the plurality of video frames includes: selecting corresponding pixel points between any two consecutive video frames in the plurality of video frames; calculating displacement information of the corresponding pixel points in horizontal and vertical directions respectively by a first model as optical flow feature information between the two consecutive video frames; obtaining a plurality of optical flow feature matrices in the at least one video segment according to the optical flow feature information between the two consecutive video frames; and converting the plurality of optical flow feature matrices into the optical flow feature vector of the at least one video segment.

[0007] According to some embodiments, the playing states include no shaking, slight shaking and severe shaking; determining the playing state of the at least one video segment according to the optical flow feature vector includes: analyzing the optical flow feature vector of the at least one video segment by a second model to obtain multi-dimension features of the at least one video segment; obtaining probabilities of the at least one video segment being in different playing states respectively according to the multi-dimension features; and determining the playing state corresponding to the probabilities of the different playing states as the playing state of the at least one video segment.

[0008] According to some embodiments, analyzing the optical flow feature vector of the at least one video segment by the second model to obtain multi-dimension features of the at least one video segment includes: obtaining optical flow feature information between a plurality of continuous two video frames in the optical flow feature vector; analyzing, by the second model, a relationship between any optical flow feature information and a plurality of optical flow feature information other than the any optical flow feature information in the optical flow feature information between the plurality of continuous two video frames; and obtaining the multi-dimension features according to the analysis result.

[0009] According to some embodiments, before determining the playing state of the at least one video segment according to the optical flow feature vector, the second model is trained, including: obtaining an optical flow feature vector of a training video segment; labeling a playing state of the training video segment; and training the second model until the second model converges finally, with the optical flow feature vector of the training video segment as input data and the labeled playing state as a label.

[0010] According to some embodiments, based on the playing state of the at least one video segment, the video segment with shaking in the input video is obtained, including: obtaining the playing state of the at least one video segment; comparing the playing states of two adjacent video segments in the input video; time-point position fusing the two adjacent video segments with the same playing state; and obtaining the video segment with shaking in the input video after time-point position fusion, the video segment with shaking being a video segment in a slight shaking state or a severe shaking state.

[0011] According to an aspect of the present application, a device for detecting video shaking is provided, comprising: a preprocessing module configured to divide an input video into at least one video segment in time sequence according to a preset number of continuous video frames; perform image size scaling on a plurality of video frames in the at least one video segment; a calculation module configured to calculate optical flow feature information between two continuous video frames by a first model to obtain a plurality of groups of optical flow feature matrices in the at least one video segment; convert the plurality of groups of optical flow feature matrices into an optical flow feature vector of the at least one video segment; an analysis module configured to determine a playing state of the at least one video segment according to the optical flow feature vector by a second model; and an output module configured to obtain the playing state of the at least one video segment and obtain a video segment with shaking in the input video.

[0012] According to an aspect of the present application, an electronic device is provided, comprising: one or more processors; a storage device configured to store one or more programs; and when the one or more programs are executed by the one or more processors, the one or more processors implement the method as described above.

[0013] According to an aspect of the present application, a computer program product is provided, comprising computer programs or instructions, which, when executed by a processor, implement the method as described above.

[0014] According to embodiments of the present application, the optical flow feature information between the continuous video frames of the video segment can be quickly extracted, and the optical flow feature vector generated by the optical flow feature information between different continuous video frames can be analyzed by the model to extract the overall time sequence information of the video segment in the same time range, so that more accurate video shaking detection effect can be achieved.

[0015] It should be understood that the foregoing general description and the following detailed description are only exemplary and are not limiting to the present application. BRIEF DESCRIPTION OF DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application.

[0017] Figure 1 A flow chart of a method for detecting video shaking according to an example embodiment of the present application is shown.

[0018] Figure 2 A flow chart of obtaining an optical flow feature vector according to an example embodiment of the present application is shown.

[0019] Figure 3 A flow chart of determining a playing state of a video segment according to an example embodiment of the present application is shown.

[0020] Figure 4 A workflow diagram of a detection device according to an example embodiment of the present application is shown.

[0021] Figure 5 A block diagram of an electronic device according to an example embodiment of the present application is shown. DETAILED DESCRIPTION

[0022] The user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in the present application are all information and data authorized by the user or authorized by all parties, and the collection, use and processing of relevant data need to comply with relevant laws, regulations and standards of the country and region, and provide corresponding operation portal for user to choose authorization or refusal.

[0023] Example embodiments now will be described more fully hereinafter with reference to the accompanying drawings. Example embodiments may, however, be implemented in many different forms and should not be construed as limited to the embodiments set forth herein; rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of example embodiments to those skilled in the art. Like reference numerals refer to like elements throughout the description. Repetitive descriptions of like elements will be omitted for sake of brevity.

[0024] The described features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. In the following description, numerous specific details are provided to give a thorough understanding of embodiments of the application. One skilled in the relevant art will recognize, however, that the application can be practiced without one or more of the specific details, or with other methods, components, materials, and so forth. In these instances, well-known structures, methods, devices, implementations, materials, and so forth are not described in detail.

[0025] The flowcharts shown in the drawings are only exemplary illustrations, and are not necessarily required to include all contents and operations / steps, nor necessarily executed in the described order. For example, some operations / steps can be further decomposed, and some operations / steps can be combined or partially combined, so the actual execution order can be changed according to actual conditions.

[0026] The terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish different objects, not to describe a specific order. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or apparatuses.

[0027] This application provides a method, apparatus, device, and program product for detecting video shake, which can combine optical flow characteristics and video timing information to automatically detect time points in the video that are in a shaking state, and ensure the accuracy of video shake detection.

[0028] The following will describe in detail, with reference to the accompanying drawings, a method, apparatus, device, and program product for detecting video shaking according to embodiments of this application.

[0029] Figure 1 A flowchart illustrating a video shake detection method according to an example embodiment of this application is shown.

[0030] like Figure 1 As shown, in step S110, the detection device extracts multiple video frames arranged in time sequence from at least one video segment corresponding to the input video.

[0031] For example, in step S110, the detection device obtains the input video that needs to be detected for shaking from the video platform, and then divides the input video into at least one video segment for detecting the playback status of the input video at different time periods.

[0032] The detection device obtains multiple video frames arranged in chronological order from the video segment based on the timestamps of the video frames.

[0033] Furthermore, the detection device divides the input video into at least one video segment in time sequence according to a preset number of consecutive video frames. Each video segment in the at least one video segment includes multiple video frames arranged in time sequence, with the number being the preset number of consecutive video frames. Moreover, the detection device can obtain the time point of each video segment based on the multiple video frames arranged in time sequence.

[0034] According to some embodiments, the detection device can divide the input video and record data according to the number of video frames contained in the input video, the preset number of consecutive video frames, and the video resolution. Assuming that the number of video frames of the input video is N frames, the preset number of consecutive video frames is 64, and the resolution of the input video is h*w. The detection device divides the input video into N / 64 video segments, each of which contains 64 consecutive video frames in time sequence, and divides each video segment into 64 groups of consecutive video frames. The detection device records the relevant data of the current video segment in the array [64, 3, h, w], where the number 64 represents the preset number of consecutive video frames (i.e., the number of video frames of the current video segment), the number 3 represents the image color channel of the data frame (i.e., the RGB three channels), and h and w represent the resolution of the current video segment.

[0035] According to some embodiments, the detection device can reduce or enlarge the multiple video frames of the input video according to the preset image size. The detection device can adjust the preset image size according to actual needs. If it is necessary to reduce the amount of data in the calculation process, the detection device can reduce the video frames according to the preset image size; if it is necessary to obtain more accurate detection results, the detection device can enlarge the video frames according to the preset image size. For example, the preset image size is 256*256, and the detection device adjusts the multiple video frames of the input video according to the image size of 256*256, and the array of the relevant data of the current video segment is changed to [64, 3, 256, 256].

[0036] In step S120, the detection device obtains the optical flow feature vector of at least one video segment according to the multiple video frames.

[0037] For example, in step S120, the detection device obtains the optical flow feature information between any two consecutive video frames in the multiple video frames, and generates multiple groups of optical flow feature matrices of at least one video segment according to the multiple optical flow feature information through the first model. The detection device converts the multiple groups of optical flow feature matrices into the optical flow feature vector of at least one video segment.

[0038] The detection device obtains the multiple video frames in at least one video segment, then selects a pixel point in any one video frame, and obtains the same pixel point in an adjacent video frame of the video frame, and uses the two corresponding pixel points to calculate the optical flow feature information between the current two video frames. The detection device obtains the optical flow feature matrix of the current video segment through the optical flow feature information between any two consecutive video frames, and converts the optical flow feature matrix into the optical flow feature vector.

[0039] According to some embodiments, the optical flow feature information includes change information of corresponding pixel points between two continuous video frames, for example, displacement information of a pixel point in a current video frame along horizontal and vertical directions respectively relative to a same pixel point in a neighboring previous video frame.

[0040] According to some embodiments, before obtaining the optical flow feature information between any two continuous video frames, the detection device pre-sets a first model for obtaining an optical flow feature matrix of the video segment, wherein the first model can adopt a RAFT (Recurrent All-Pairs Field Transforms for Optical Flow) convolutional neural network model. The detection device takes any two continuous video frames in at least one video segment as a group as input data of the first model, and obtains multiple groups of optical flow feature matrices of the at least one video segment through the first model.

[0041] According to some embodiments, the detection device inputs any two continuous video frames in a video segment as a group of data into the trained first model, selects corresponding two pixel points between the two continuous video frames through the first model, and obtains position changes of the two pixel points in a plane coordinate system. For example, the detection device calculates displacement information of corresponding pixel points along x-axis and y-axis (i.e., horizontal direction and vertical direction) of the plane coordinate system through the trained first model, i.e., optical flow feature information between the current two video frames.

[0042] According to the optical flow feature information between any two continuous video frames, the detection device obtains an optical flow feature matrix of the current video segment through the first model, and converts the optical flow feature matrix into an optical flow feature vector of the current video segment according to the number of optical flow feature matrices, components of optical flow feature information, and size of the optical flow feature matrix.

[0043] In step S130, the detection device determines the playing state of the at least one video segment according to the optical flow feature vector.

[0044] For example, in step S130, the detection device analyzes the relationship between the optical flow feature information in the optical flow feature vector of the at least one video segment through the second model to obtain multi-dimensional features of the video segments. Based on the multi-dimensional features of the video segments, the detection device calculates probabilities of the video segments being in different playing states respectively, and determines the playing state of the video segments according to the probabilities.

[0045] The detection device first sets a second model for analyzing the relationship between the optical flow feature information between any two continuous video frames in the optical flow feature vector of the current video segment in the at least one video segment and other optical flow feature information in the optical flow feature vector of the current video segment.

[0046] According to some embodiments, the second model can adopt a ViT (Vision Transformer, a deep learning model based on self-attention mechanism for computer vision field) model.

[0047] According to some embodiments, the detection device inputs the optical flow feature vector of the at least one video segment into the second model which has been trained. The detection device analyzes the relationship between the optical flow feature information between any two consecutive data frames in the optical flow feature vector and the other optical flow feature information in the optical flow feature vector through the second model to obtain the multi-dimensional features of each video segment in the at least one video segment, which are used to represent the overall features of each video segment.

[0048] According to some embodiments, a convolution layer is added to the second model set by the detection device, which is used to calculate the playing state corresponding to the video segment through the multi-dimensional features of the video segment.

[0049] After obtaining the multi-dimensional features of each video segment in the at least one video segment, the detection device continues to operate the multi-dimensional features of each video segment through the second model to obtain the probabilities of each video segment being in different playing states (not shaking, slight shaking or serious shaking). The detection device determines the playing state corresponding to the maximum value in the probabilities of each video segment being in different playing states as the final playing state of each video segment.

[0050] According to some embodiments, the detection device inputs the multi-dimensional features of each video segment into the added convolution layer, and obtains the probabilities of the video segment being in not shaking, slight shaking or serious shaking through the calculation of the added convolution layer.

[0051] In step S140, based on the playing states of the at least one video segment, the detection device obtains the video segment with shaking in the input video.

[0052] For example, in step S140, after determining the playing states of the at least one video segment, the detection device determines the video segment with shaking in the input video by comparing the playing states of the adjacent two video segments in the at least one video segment.

[0053] The detection device collects the playing states of each video segment in the at least one video segment of the input video, and compares the playing states of the adjacent two video segments. If the playing states of the adjacent two video segments are the same, the detection device fuses the two video segments with the same playing state in the time point position, merges the two video segments through the time information to obtain a new video segment and the time point position corresponding to the playing state of the new video segment.

[0054] For example, if the input video has N frames, the detection device divides the input video into N / 64 video segments, grouping them into 64 consecutive frames in chronological order. The time point of each video segment corresponds to the time information of the sequentially arranged video frames within that segment. The detection device also obtains the playback status of each of the N / 64 video segments.

[0055] Assuming the input video has a frame rate of 25 frames per second, both the first video segment and the adjacent second video segment exhibit shaky playback. The first video segment starts at 00:00.00 and ends at 00:02.56. The second video segment starts at 00:02.56 and ends at 00:05.12. The detection device fuses the first and second video segments by time points to obtain a third video segment. This third video segment also exhibits shaky playback, and its time points range from 00:00.00 to 00:05.12.

[0056] According to embodiments of this application, the detection device can automatically detect video segments with shaky motion in the input video, eliminating the need for manual inspection of the entire input video and improving the overall efficiency of video quality assessment.

[0057] Figure 2 A flowchart illustrating the acquisition of optical flow feature vectors according to an example embodiment of this application is shown.

[0058] like Figure 2 As shown, in step S210, the detection device obtains the optical flow feature matrix between two consecutive video frames through the trained first model.

[0059] For example, in step S210, the detection device uses a first model trained according to a preset image size as a feature extractor to calculate the optical flow feature information between two consecutive video frames. Based on the optical flow feature information between any two consecutive video frames, the detection device generates an optical flow feature matrix between any two consecutive video frames using the first model.

[0060] According to an embodiment of this application, the first model may employ a RAFT (Recurrent All-Pairs FieldTransforms for Optical Flow) convolutional neural network.

[0061] The detection device inputs the video frames contained in each video segment into a first model in pairs. The detection device calculates the optical flow feature information of the corresponding pixels between the current two video frames along the x-axis and y-axis (i.e., the horizontal and vertical directions) of the planar coordinate system, using the first model as the optical flow feature information between the current two video frames.

[0062] The detection model obtains multiple sets of optical flow feature information along the horizontal and vertical directions between any two consecutive video frames in the current video segment based on the optical flow feature information between the two current video frames. Then, the detection model can obtain multiple sets of optical flow feature matrices of the current video segment from the optical flow feature information between any two consecutive video frames.

[0063] For example, if the current video segment contains 64 consecutive video frames, the detection model inputs any two consecutive video frames as a group into the first model, resulting in 63 groups of optical flow feature information formed by the optical flow feature information between any two consecutive video frames. This information is then used to generate the optical flow feature matrix of the current video segment, denoted as [63, 2, m, n]. Here, 2 represents the two components of the optical flow feature in the horizontal and vertical directions, and m*n represents the size of the optical flow feature matrix output by the first model. The values ​​of m and n are determined by the network structure of the first model and can also be set according to actual needs.

[0064] According to some embodiments, the optical flow feature matrix of the current video segment can be denoted as [63,2,32,32], and the size of the optical flow feature matrix is ​​set to 32*32 according to empirical values.

[0065] In step S220, the detection device converts the optical flow feature matrix into an optical flow feature vector.

[0066] For example, in step S220, the detection device converts the optical flow feature matrix of the current video segment into the optical flow feature vector of the current video segment through calculation.

[0067] For example, if the optical flow feature matrix of the current video segment is [63, 2, 32, 32], the detection device calculates the optical flow feature vector of the current video segment as [63, 2*32*32] to obtain [63, 2048].

[0068] According to embodiments of this application, the detection device can accurately and quickly extract the optical flow features of each video segment in the input video through the first model, providing a basis for ensuring the accuracy of video shake time point analysis.

[0069] Figure 3 A flowchart illustrating the determination of the playback state of a video segment according to an example embodiment of this application is shown.

[0070] like Figure 3 As shown, in step S310, the detection device trains the second model.

[0071] For example, in step S310, the detection device collects a plurality of training video clips, and calculates the optical flow feature vectors of the training video clips according to the method in steps S210 and S220.

[0072] The detection device takes the optical flow feature vectors of the training video clips as input data of the second model, takes the playing states of the training video clips as labels of the second model, and supervises the training and parameter updating of the second model until the second model finally converges.

[0073] According to an embodiment of the present application, the second model can adopt a ViT (Vision Transformer, a deep learning model based on self-attention mechanism in the field of computer vision) model, compare and analyze the relationship between the optical flow feature information between any two continuous data frames in the input optical flow feature vector and the relationship between other optical flow feature information in the current optical flow feature vector, to extract the multi-dimensional features of the video clip corresponding to the current optical flow feature vector, wherein the multi-dimensional features of the video clip include the playing state of the video clip and the timing information of the video clip.

[0074] In step S320, the detection device analyzes the optical flow feature vectors of at least one video clip through the second model to obtain the multi-dimensional features of each video clip.

[0075] For example, in step S320, the detection device inputs the optical flow feature vectors of at least one video clip of the input video into the trained second model to analyze the relationship between the plurality of optical flow features in the input optical flow feature vector through the second model.

[0076] The detection device analyzes the relationship between the optical flow feature information between any two continuous data frames in the input optical flow feature vector and the relationship between other optical flow feature information in the current optical flow feature vector except for this optical flow feature information, to obtain the multi-dimensional features of the video clip corresponding to the current optical flow feature vector, to represent the overall features of this video clip.

[0077] According to some embodiments, the detection device takes the optical flow feature vector of a video clip as the input of the second model, and the second model obtains 1 P-dimensional feature of the current video clip through analysis, wherein the value P is determined by the structure of the last convolutional layer of the second model, and can also be set according to actual needs.

[0078] For example, the detection device takes the optical flow feature vector [63,2048] (i.e., 63 optical flow feature vectors with 2048 dimensions) of the current video segment as the input of the second model. The second model analyzes and obtains a 768-dimensional feature of the current video segment, where the 768 dimensions are the dimension values ​​set according to experience.

[0079] In step S330, based on multi-dimensional features, the detection device obtains the probability that each video segment is in a different playback state.

[0080] For example, in step S330, using the multi-dimensional features of each video segment as input data, the detection device calculates the probability that each video segment is in a different playback state through the second model.

[0081] According to some embodiments, the detection device adds a convolutional layer with an output of 3 to the second model. The detection device takes the multi-dimensional features output by the second model as input data and outputs a 3-dimensional vector [p1, p2, p3] through the convolutional layer, where p1, p2, and p3 represent the probabilities that the current video segment is in a playback state with no shaking, slight shaking, and severe shaking, respectively.

[0082] In step S340, the detection device determines the playback state corresponding to the highest probability as the playback state of each video segment.

[0083] For example, in step S340, the detection device determines the maximum probability among the video clip being in a playback state of no shaking, slight shaking, and severe shaking, and determines the playback state of the current video clip accordingly.

[0084] According to some embodiments, the detection device determines the maximum value of p1, p2, and p3 in step S330 above. The detection device determines the playback state corresponding to the maximum value among p1, p2, and p3 as the playback state of the current video segment.

[0085] According to embodiments of this application, by analyzing the optical flow characteristics between two adjacent data frames with different time ranges, the detection device can better obtain the complete timing information and playback status of the current video segment.

[0086] Figure 4 A schematic diagram illustrating the operation of a detection apparatus according to an example embodiment of this application is shown.

[0087] like Figure 4 As shown, the detection device 400 includes a preprocessing module 410, a calculation module 420, an analysis module 430, and an output module 440.

[0088] The preprocessing module 410 obtains an input video from a video platform, and divides the input video into a plurality of video segments in time sequence according to a preset number of continuous video frames, that is, uniformly divides the input video into a plurality of video segments with the same time range according to a plurality of video frames containing time information, wherein each video segment contains a plurality of video frames with the preset number of continuous video frames.

[0089] The preprocessing module 410 also reduces or enlarges the video frames in the divided video segments according to a preset image size, so as to convert the video frames after image scaling processing into a data format required by the operation module 420.

[0090] The operation module 420 calculates the optical flow feature information between adjacent two continuous video frames of a current video segment through a trained first model, and generates a plurality of groups of optical flow feature matrices of the current video segment through the first model, and then obtains a plurality of groups of optical flow feature matrices of each video segment in the input video.

[0091] The operation module 420 also converts the plurality of groups of optical flow feature matrices of each video segment in the input video into optical flow feature vectors as input data of the analysis module 430.

[0092] The analysis module 430 first collects a plurality of training video segments, and obtains the optical flow feature vectors of the training video segments. The analysis module 430 also labels the playing states of the training video segments. Then, the analysis module 430 trains a second model by taking the optical flow feature vectors of the training video segments as input data of the second model and taking the playing states of the training video segments as labels of the second model, until the second model converges.

[0093] The analysis module 430 analyzes the optical flow feature vectors of each video segment in the input video through the second model to obtain the multi-dimensional features of each video segment.

[0094] The analysis module 430 obtains the probabilities of each video segment being in different playing states (not shaking, slightly shaking or seriously shaking) through the added convolution layer in the second model according to the multi-dimensional features of each video segment, and determines the playing state corresponding to the maximum value in the probabilities of being in different playing states as the playing state of each video segment.

[0095] The output module 440 obtains the playing state detection results of each video segment of the input video, and compares the playing states of two adjacent continuous video segments.

[0096] After determining that the playing states of the two adjacent continuous video segments are the same, the output module 440 fuses the two adjacent continuous video segments with the same playing state at the time point to obtain a new video segment and its playing state.

[0097] The output module 440 outputs the video segment in the input video that is in slight shaking or severe shaking.

[0098] Figure 5 A block diagram of an electronic device according to an example embodiment of the present application is shown.

[0099] As Figure 5 shown, the electronic device 600 is merely an example, and should not bring any limitation to the function and use range of the embodiments of the present application.

[0100] As Figure 5 shown, the electronic device 600 is in the form of a general computing device. The components of the electronic device 600 can include, but are not limited to, at least one processing unit 610, at least one storage unit 620, a bus 630 connecting different system components (including the storage unit 620 and the processing unit 610), a display unit 640, etc. The storage unit stores program codes which can be executed by the processing unit 610, so that the processing unit 610 executes the methods according to various example embodiments of the present application described in the present specification. For example, the processing unit 610 can execute the methods as shown in Figure 1 .

[0101] The storage unit 620 can include a readable medium in the form of a volatile storage unit, such as a random access memory (RAM) 6201 and / or a cache memory unit 6202, and can further include a read-only memory (ROM) 6203.

[0102] The storage unit 620 can further include program / utilities 6204 having a set of (at least one) program modules 6205, such as an operating system, one or more application programs, other program modules, and program data, each of which or some combination of which can include implementation of a network environment.

[0103] The bus 630 can represent one or more of several types of bus structures, including a storage unit bus or storage unit controller, a peripheral bus, a graphics acceleration port, a processing unit bus, or a local bus using any of a variety of bus architectures.

[0104] The electronic device 600 can also communicate with one or more external devices 700 such as a keyboard or a pointing device, a Bluetooth device, etc.; other devices that enable a user to interact with the electronic device 600; and / or any devices (e.g., a router, a modem, a switch, etc.) that enable the electronic device 600 to communicate with one or more other computing devices. Such communication can occur via an input / output (I / O) interface 650. Still yet, the electronic device 600 can communicate with one or more networks, such as a local area network (LAN), a wide area network (WAN), and / or the Internet, through a network adapter 660. The network adapter 660 can communicate with the other components of the electronic device 600 through the bus 630. It should be understood that, although not shown explicitly, other hardware and / or software components could be used in conjunction with the electronic device 600. These include, but are not limited to, microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data archival storage systems, etc.

[0105] From the above description of the embodiments, it is easy for those skilled in the art to understand that the example embodiments described herein can be implemented by software, or by software in combination with necessary hardware. The technical solutions according to the embodiments of the present application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a U disk, a mobile hard disk, etc.) or a network, and includes a number of instructions to enable a computing device (which can be a personal computer, a server, a mobile terminal, or a network device, etc.) to perform the method according to the embodiments of the present application.

[0106] The software product can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium, for example, can be, but is not limited to, an electrical, a magnetic, an optical, an electromagnetic, an infrared, or a semiconductor system, device or apparatus, or any combination of the above. More specific examples (a non-exhaustive list) of the readable storage medium include an electrical connection having one or more wires, a portable disc, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0107] The computer readable storage medium can include a computer-readable medium in baseband or propagated as a carrier wave in a propagated signal, wherein the latter embodiment includes a computer-readable medium. Such computer-readable storage medium or media can take many forms, including but not limited to, volatile, non-volatile, or transitory media, volatile or non-volatile media, whether memory or otherwise, including those that are internal or external to various computing devices.

[0108] The program code for carrying out operations of the present application can be written in any combination of one or more programming languages, including an object oriented programming language such as Java, C++ or the like, and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code can execute entirely on the user's computing device, partly on the user's computing device, as a stand-alone software package, partly on the user's computing device and partly on a remote computing device or entirely on the remote computing device or server. In the latter scenario, the remote computing device can be connected to the user's computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computing device, such as through the Internet using an Internet Service Provider.

[0109] The above computer readable medium has one or more programs stored therein, which, when executed by the device, cause the computer readable medium to implement the foregoing functions.

[0110] Those skilled in the art can understand that the above-mentioned modules can be distributed in the device according to the description of the embodiment, and can also be changed in one or more devices different from the embodiment. The modules of the above-mentioned embodiment can be combined into one module, or can be further split into a plurality of sub-modules.

[0111] According to some embodiments of the present application, the technical solutions of the present application can more accurately and quickly calculate the optical flow features of the video, and better extract the complete timing information of the video by analyzing the changes of the optical flow features between the different adjacent two video frames in the same time range, so as to achieve more accurate video shaking detection effect.

[0112] The above describes the embodiments of the present application in detail, and the above description of the embodiments is only used to help understand the method of the present application and its core idea. Meanwhile, the changes or deformations made by the person skilled in the art according to the idea of the present application, based on the specific implementation and application range of the present application, all belong to the protection range of the present application. In conclusion, the content of the specification should not be understood as the limitation of the present application.

Claims

1. A method of detecting video shake, the method comprising: The method comprises the following steps: extracting a plurality of video frames arranged in sequence in at least one video segment corresponding to an input video; obtaining an optical flow feature vector of the at least one video segment according to the plurality of video frames; determining a playing state of the at least one video segment according to the optical flow feature vector, wherein the playing state comprises no shaking, slight shaking and severe shaking; obtaining a video segment with shaking in the input video based on the playing state of the at least one video segment; wherein, obtaining the optical flow feature vector of the at least one video segment according to the plurality of video frames comprises: selecting corresponding pixel points between any two continuous video frames in the plurality of video frames; calculating displacement information of the corresponding pixel points in the horizontal direction and the vertical direction respectively as optical flow feature information between the two continuous video frames by a first model; obtaining a plurality of optical flow feature matrices in the at least one video segment according to the optical flow feature information between the two continuous video frames; converting the plurality of optical flow feature matrices into the optical flow feature vector of the at least one video segment; wherein, determining the playing state of the at least one video segment according to the optical flow feature vector comprises: analyzing the optical flow feature vector of the at least one video segment by a second model to obtain multi-dimensional features of the at least one video segment; obtaining probabilities of the at least one video segment being in different playing states respectively according to the multi-dimensional features; determining the playing state corresponding to the probabilities of the different playing states as the playing state of the at least one video segment; wherein, analyzing the optical flow feature vector of the at least one video segment by the second model to obtain the multi-dimensional features of the at least one video segment comprises: obtaining optical flow feature information between a plurality of continuous video frames in the optical flow feature vector; analyzing relationships between any optical flow feature information and a plurality of optical flow feature information other than the any optical flow feature information in the optical flow feature information between the plurality of continuous video frames by the second model; obtaining the multi-dimensional features according to the analysis results.

2. The method of claim 1, wherein, extracting a plurality of video frames arranged in sequence in at least one video segment corresponding to an input video comprises: dividing the input video into at least one video segment in sequence with a preset number of continuous video frames; scaling the plurality of video frames in the at least one video segment according to a preset image size.

3. The method of claim 1, wherein, Before determining the playing state of the at least one video segment according to the optical flow feature vector, the second model is trained, comprising: obtaining an optical flow feature vector of a training video segment; labeling a playing state of the training video segment; training the second model with the optical flow feature vector of the training video segment as input data and the labeled playing state as a label until the second model converges finally.

4. The method of claim 1, wherein, obtaining a video segment with shaking in the input video based on the playing state of the at least one video segment comprises: obtaining the playing state of the at least one video segment; comparing the playing states of two adjacent video segments in the input video; Fusing the adjacent two video clips with the same playing state at a time point; Obtaining a video clip with shaking from the input video after the time point fusion, the video clip with shaking being a video clip in a slight shaking playing state or a severe shaking playing state.

5. A device for detecting video shake, characterized by The method comprises the following steps: A preprocessing module is configured to divide an input video into at least one video clip in a time sequence according to a preset number of continuous video frames; An image size scaling module is configured to perform image size scaling on a plurality of video frames in the at least one video clip; An operation module is configured to calculate optical flow feature information between two continuous video frames by a first model to obtain a plurality of groups of optical flow feature matrices in the at least one video clip; The plurality of groups of optical flow feature matrices are converted into an optical flow feature vector of the at least one video clip; An analysis module is configured to determine a playing state of the at least one video clip according to the optical flow feature vector by a second model, the playing state including no shaking, slight shaking, and severe shaking; An output module is configured to obtain the playing state of the at least one video clip and obtain a video clip with shaking from the input video; The operation module is further configured to: select corresponding pixel points between any two continuous video frames in the plurality of video frames; calculate displacement information of the corresponding pixel points in a horizontal direction and a vertical direction respectively as the optical flow feature information between the two continuous video frames by the first model; obtain the plurality of groups of optical flow feature matrices in the at least one video clip according to the optical flow feature information between the two continuous video frames; convert the plurality of groups of optical flow feature matrices into the optical flow feature vector of the at least one video clip; The analysis module is further configured to: analyze the optical flow feature vector of the at least one video clip by the second model to obtain a multi-dimensional feature of the at least one video clip; obtain probabilities of the at least one video clip being in different playing states respectively according to the multi-dimensional feature; determine the playing state corresponding to the probabilities of the different playing states as the playing state of the at least one video clip; The analysis module is further configured to: obtain optical flow feature information between a plurality of two continuous video frames in the optical flow feature vector; analyze a relationship between any optical flow feature information and a plurality of optical flow feature information other than the any optical flow feature information by the second model; obtain the multi-dimensional feature according to the analysis result.

6. An electronic device, comprising: The method comprises the following steps: one or more processors; a storage device configured to store one or more programs; when the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1-4.

7. A computer program product comprising computer programs or instructions, characterized in that, The computer program or instructions are executed by the processor to implement the method according to any one of claims 1-4.

Images