Video detection method, device, electronic device and storage medium

Abstract

The embodiment of the present invention provides a video detection method, device, electronic device and storage medium, which relate to the field of video processing technology. The method obtains multiple video clips by performing lens segmentation on video data, extracts multiple key frames from each video clip, forms multiple target image sequences, and reduces the time complexity of video detection. Then, each target image sequence is input into a scene classification model for processing, and the scene corresponding to the video clip is identified by the RGB features of each key frame, and then a reverse image sequence opposite to the playback timing of each target image sequence is determined, and each target image sequence and the corresponding reverse image sequence are input into a camera movement classification model for processing, thereby comprehensively identifying the camera movement corresponding to the video clip by combining the RGB features and timing information of multiple key frames, thereby improving the accuracy of camera movement identification, and finally generating a video detection result with high accuracy based on the identified scene and camera movement.

Description

Technical Field

[0001] The present invention relates to the field of video processing technology, and in particular to a video detection method, device, electronic equipment and storage medium. Background Art

[0002] In intelligent video editing and creation scenarios, it is necessary to perform lens detection on each lens corresponding to the video shot by the camera. Among them, lens detection is a combination of lens switching, camera position, camera movement and other information of video shooting. The lens field and camera movement are important and basic information. In existing technologies, face detection is generally used to identify the field and camera movement based on the face. This type of recognition method leads to low accuracy of video detection results and is not suitable for lens detection scenarios other than human shots. Summary of the invention

[0003] The purpose of the embodiments of the present invention is to provide a video detection method, device, electronic device and storage medium to improve the accuracy of video detection results. The specific technical solution is as follows:

[0004] In a first aspect of the present invention, a video detection method is provided, the method comprising:

[0005] Get video data;

[0006] Segmenting the video data into shots to determine a plurality of video segments;

[0007] Extract multiple key frames from each video clip to determine multiple target image sequences;

[0008] Input each target image sequence into the scene classification model for processing, and identify the scene corresponding to each video clip;

[0009] Determine a reverse image sequence having a playback timing opposite to that of each target image sequence;

[0010] Inputting each target image sequence and the corresponding reverse image sequence into a camera movement classification model for processing to identify the camera movement corresponding to the video clip;

[0011] The video detection results are generated based on the scene and camera movement identified in each video clip.

[0012] Optionally, the camera movement classification model includes a target classification network and a weighted network, or includes the target classification network, a reverse classification network and the weighted network;

[0013] The step of inputting each target image sequence and the corresponding reversed image sequence into a camera movement classification model for processing to identify the camera movement corresponding to each video clip includes:

[0014] Input each target image sequence into the target classification network for processing, and output the corresponding target vector;

[0015] Inputting the reverse order image sequence corresponding to each target image sequence into the target classification network / the reverse order classification network for processing, and outputting the corresponding reverse order vector;

[0016] The target vector and the corresponding inverse vector are input into the weighted network for weighting to determine the camera movement corresponding to each video clip.

[0017] Optionally, the target vector and the corresponding inverse vector are input into the weighted network for weighting to determine the camera movement corresponding to each video clip, including:

[0018] Determine a first camera movement by using the target vector, and determine a second camera movement by using the reverse vector;

[0019] If the first camera movement and the second camera movement are consistent, respectively add the probability values ​​corresponding to the same camera movement in the target vector and the corresponding reverse vector to obtain a target probability vector;

[0020] If the first camera movement and the second camera movement are inconsistent, weighting the probability value corresponding to the first camera movement in the target vector to the probability value corresponding to the camera movement opposite to the first camera movement in the reverse vector to obtain a weighted reverse vector, weighting the probability value corresponding to the second camera movement in the reverse vector to the probability value corresponding to the camera movement opposite to the second camera movement in the target vector to obtain a weighted target vector, and adding the probability values ​​corresponding to the same camera movement in the weighted target vector and the weighted reverse vector respectively to obtain a target probability vector;

[0021] The camera movement corresponding to each video clip is determined according to the target probability vector.

[0022] Optionally, the target classification network includes a first classification unit and a second classification unit;

[0023] The step of inputting each target image sequence into the target classification network for processing and outputting a corresponding target vector comprises:

[0024] Inputting the target image sequence into a first classification unit for processing to obtain a first classification vector determined based on the inter-frame optical flow features corresponding to the target image sequence;

[0025] The target image sequence is input into a second classification unit for processing to obtain a second classification vector determined based on RGB features corresponding to the target image sequence.

[0026] Optionally, the step of inputting the reversed image sequence into the target classification network for processing and outputting a corresponding reversed vector includes:

[0027] Inputting the reversed image sequence into a first classification unit for processing to obtain a third classification vector determined based on the inter-frame optical flow features corresponding to the reversed image sequence;

[0028] The reversed image sequence is input into a second classification unit for processing to obtain a fourth classification vector determined based on RGB features corresponding to the reversed image sequence.

[0029] Optionally, the reverse classification network includes a third classification unit and a fourth classification unit;

[0030] The step of inputting the reversed image sequence into the reversed classification network for processing and outputting a corresponding reversed vector comprises:

[0031] Inputting the reversed image sequence into a third classification unit for processing to obtain a third classification vector determined based on the inter-frame optical flow features corresponding to the reversed image sequence;

[0032] The reversed image sequence is input into a fourth classification unit for processing to obtain a fourth classification vector determined based on RGB features corresponding to the reversed image sequence.

[0033] Optionally, the step of inputting the target vector and the corresponding reversed vector into the weighted network for weighting to determine the camera movement corresponding to each video clip includes:

[0034] The first classification vector, the second classification vector, the third classification vector and the fourth classification vector are input into the weighted network for weighting to determine the camera movement corresponding to each video clip.

[0035] Optionally, the step of inputting the first classification vector, the second classification vector, the third classification vector, and the fourth classification vector into the weighted network for weighting to determine the camera movement corresponding to each video clip includes:

[0036] Determine a first camera movement by using the first classification vector, determine a second camera movement by using the second classification vector, determine a third camera movement by using the third classification vector, and determine a fourth camera movement by using the fourth classification vector;

[0037] If the first camera movement, the second camera movement, the third camera movement and the fourth camera movement are all the same, the probability values ​​corresponding to the same camera movement in the first classification vector, the second classification vector, the third classification vector and the fourth classification vector are added together to obtain a target probability vector;

[0038] If the first camera movement, the second camera movement, the third camera movement, and the fourth camera movement are inconsistent, the probability values ​​corresponding to the same camera movement in the first classification vector and the second classification vector are added respectively to obtain a target classification vector, the camera movement corresponding to the maximum probability value in the target classification vector is determined as the fifth camera movement, and the probability values ​​corresponding to the same camera movement in the third classification vector and the fourth classification vector are added respectively to obtain a reverse classification vector, and the camera movement corresponding to the maximum probability value in the reverse classification vector is determined as the sixth camera movement;

[0039] Weighting the probability value corresponding to the fifth camera movement in the target classification vector to the probability value corresponding to the camera movement opposite to the fifth camera movement in the reverse classification vector to obtain a weighted reverse classification vector, weighting the probability value corresponding to the sixth camera movement in the reverse classification vector to the probability value corresponding to the camera movement opposite to the sixth camera movement in the target classification vector to obtain a weighted target classification vector, and adding the probability values ​​corresponding to the same camera movement in the weighted target classification vector and the weighted reverse classification vector respectively to obtain a target probability vector;

[0040] The camera movement corresponding to each video clip is determined according to the target probability vector.

[0041] Optionally, the scene classification model includes a backbone network, a pooling layer and a classification layer;

[0042] The step of inputting each target image sequence into the scene classification model for processing and identifying the scene corresponding to each video clip includes:

[0043] Input each target image sequence into the backbone network to extract features and determine the corresponding feature image;

[0044] Input each feature image into the pooling layer for average pooling to determine the corresponding pooled image;

[0045] Each pooled image is input into the scene classification layer for classification operation to determine the scene corresponding to each video clip.

[0046] Optionally, the method further includes:

[0047] Determine whether to update the scene size according to the attribute information of the camera movement.

[0048] Optionally, determining whether to update the scene scale according to the attribute information of the camera movement includes:

[0049] Determining a scene sequence corresponding to the camera movement according to the attribute information of the camera movement;

[0050] Comparing the scene in the scene sequence with the scene corresponding to the video clip;

[0051] Determine whether to update the scene corresponding to the video clip according to the comparison result.

[0052] Optionally, the method further comprises a training step of training the camera movement classification model, wherein the training step comprises:

[0053] Acquire a sample image sequence and a first image sequence corresponding to all camera movements, wherein the first image sequence is an image sequence having a playback time sequence opposite to that of the sample image sequence, and the sample image sequence includes a plurality of sample images;

[0054] Inputting each of the sample image sequences and the corresponding first image sequence into the camera movement classification model respectively, and identifying the camera movement corresponding to the sample image sequence;

[0055] According to the real camera movements and the identified camera movements of the sample image sequence, the parameters of the camera movement classification model are adjusted to determine a trained camera movement classification model.

[0056] In another aspect of the embodiment of the present invention, a video detection device is provided, the device comprising:

[0057] A video acquisition module, used to acquire video data;

[0058] A video switching module, used to segment the video data into shots to determine a plurality of video segments;

[0059] A key frame extraction module is used to extract multiple key frames from each video clip and determine multiple target image sequences;

[0060] A scene recognition module is used to input each target image sequence into the scene classification model for processing and identify the scene corresponding to each video clip;

[0061] A reverse order determination module, used to determine a reverse order image sequence opposite to the playback time sequence of each target image sequence;

[0062] A camera movement recognition module, used for inputting each target image sequence and the corresponding reverse order image sequence into a camera movement classification model for processing, and identifying the camera movement corresponding to the video clip;

[0063] The detection result generation module is used to generate video detection results based on the scene and camera movement identified in each video clip.

[0064] In another aspect of the present invention, an electronic device is provided, the electronic device comprising a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus;

[0065] Memory, used to store computer programs;

[0066] The processor is used to implement the method steps of any of the above-mentioned video methods when executing the program stored in the memory.

[0067] In another aspect of the present invention, a computer-readable storage medium is provided, wherein instructions are stored in the computer-readable storage medium, and when the computer-readable storage medium is run on a computer, the computer executes any of the above-mentioned video detection methods.

[0068] In another aspect of the present invention, a computer program product comprising instructions is provided, which, when executed on a computer, enables the computer to execute any of the above-mentioned video detection methods.

[0069] In the solution provided by the embodiment of the present invention, a plurality of video clips are obtained by performing lens segmentation on the video data, and a plurality of key frames are extracted from each video clip to form a plurality of target image sequences, thereby reducing the time complexity of video detection. Then, each target image sequence is respectively input into the scene classification model for processing, and the scene corresponding to the video clip is identified by the RGB features of each key frame, and then the reverse image sequence opposite to the playback timing of each target image sequence is determined, and each target image sequence and the corresponding reverse image sequence are input into the camera movement classification model for processing, thereby comprehensively identifying the camera movement corresponding to the video clip by the RGB features and timing information of the plurality of key frames, thereby improving the accuracy of camera movement identification. Finally, based on the identified scene and camera movement, a video detection result with high accuracy is generated. Moreover, the main basis for the recognition of scene and camera movement of the present invention is not the facial features, and it can also be applied to some camera detection scenes other than the human lens. BRIEF DESCRIPTION OF THE DRAWINGS

[0070] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art are briefly introduced below.

[0071] Figure 1 A flowchart of a video detection method provided by an embodiment of the present invention;

[0072] Figure 2 A schematic diagram of a video detection method provided by an embodiment of the present invention;

[0073] Figure 3 A flowchart of another video detection method provided by an embodiment of the present invention;

[0074] Figure 4 A schematic diagram of the structure of a camera movement classification model provided by an embodiment of the present invention;

[0075] Figure 5A schematic diagram of the structure of another camera movement classification model provided by an embodiment of the present invention;

[0076] Figure 6 A schematic diagram of the structure of a video detection device provided by an embodiment of the present invention;

[0077] Figure 7 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0078] The technical solutions in the embodiments of the present invention will be described below in conjunction with the accompanying drawings in the embodiments of the present invention.

[0079] Reference Figure 1 and Figure 2 , a video detection method is shown, which may include:

[0080] S101: Obtain video data.

[0081] S102: Segment the video data into shots to determine a plurality of video segments.

[0082] In the embodiments of the present invention, in some intelligent video editing and creation scenarios, for example, it is necessary to edit the video data, or to perform secondary creation on the video data, such as enriching the shot changes in the video data. The types of video data may include but are not limited to TV series, movies, variety shows, documentaries, music videos, advertisements, etc. Therefore, in order to assist in intelligent creation, it is necessary to perform video detection on the video data, obtain the video detection results, and determine the specific plan for intelligent creation of the video data based on the video detection results.

[0083] In order to better detect the video data, the video data corresponding to a shot is taken as a video segment, thereby the video data can be divided into multiple video segments. In one example, the image similarity can be used to perform the shot segmentation of the video data. For example, the image similarity of two adjacent frames of video images in the video data is calculated, and an image similarity threshold is preset. If the image similarity of the two adjacent frames of video images is greater than or equal to the image similarity threshold, it is determined that the two adjacent frames of video images are video data corresponding to the same shot. If the image similarity of the two adjacent frames of video images is less than the image similarity threshold, it is determined that the two adjacent frames of video images are video data corresponding to different shots, and a shot segmentation is performed between the two adjacent frames of video images. In another example, the histogram statistics method can also be used to perform the shot segmentation of the video data. For example, the color distribution corresponding to each frame of video image is counted by a histogram. Then the color distribution of the two adjacent frames of video images is compared. If the color distribution changes are different, it is determined that the two frames of video images are video data corresponding to different shots, and a shot segmentation is performed between the two adjacent frames of video images. By analogy, multiple video segments can be obtained by segmentation.

[0084] S103 , extracting multiple key frames from each video segment to determine multiple target image sequences.

[0085] S104: Input each target image sequence into a scene classification model for processing, and identify the scene corresponding to each video clip.

[0086] In the embodiment of the present invention, since each video clip includes many frames of video images, considering that each frame of video images in each video clip is subjected to video detection, the complexity of the subsequent detection is greatly improved. Among them, the scene and camera movement of the lens are important and basic information for video detection. When the video clip is re-created or edited, there will not be a large number of different scene and all camera movements in the same shot. Therefore, in order to reduce the time complexity of the subsequent algorithm, multiple key frames can be extracted in each video clip in the order of video playback to form a target image sequence. Therefore, each target image sequence includes multiple key frames arranged in time sequence. Each target image sequence is input into the scene classification model for processing, and the scene corresponding to each video clip can be identified by the scene classification model. Among them, the type of scene can include at least: close-up, close-up, medium shot, panoramic view and long shot. The scene classification model extracts features from the image of each key frame in the target image sequence, and then performs feature recognition based on the extracted feature image to identify the corresponding scene.

[0087] S105: Determine a reverse image sequence that is opposite to the playback timing of each target image sequence.

[0088] S106: Input each target image sequence and the corresponding reverse image sequence into a camera movement classification model for processing, and identify the camera movement corresponding to the video clip.

[0089] In the embodiment of the present invention, when a camera movement effect is applied to a shot, the corresponding multiple frames in the video clip will change. In order to ensure the accuracy of camera movement recognition, after determining the target image sequence, the corresponding reverse image sequence can be determined. The reverse image sequence has a playback timing opposite to that of the target image sequence.

[0090] Each target image sequence and the corresponding reverse image sequence are simultaneously input into a camera movement classification model for processing, and the camera movement corresponding to the video clip can be identified by the camera movement classification model. The camera movement types may include at least: still, moving, close-up, and far-away. The camera movement classification model extracts RGB features corresponding to multiple key frames in the target image sequence and RGB features corresponding to multiple key frames in the reverse image sequence at the same time, and performs feature recognition based on the combination of the two extracted RGB features to identify the corresponding camera movement. Thus, the camera movement corresponding to the video clip is identified by combining the RGB features and timing information of multiple key frames, thereby improving the accuracy of camera movement recognition.

[0091] The above does not limit the execution order of each step. For example, steps S104 and S105 can be executed simultaneously, steps S105 and S106 can be executed before step S104, or steps 105 and 106 can be executed after step S104, etc., which is determined according to actual needs.

[0092] S107: Generate video detection results based on the scene and camera movement identified in each video clip.

[0093] In an embodiment of the present invention, based on the scene and camera movement obtained by identifying each video clip, a video detection result with high accuracy can be generated, and the video detection result can be displayed. The video detection result can be a combination of several scene and camera movement arranged according to the playback sequence of the video data. Since the basis for the recognition of scene and camera movement of the present invention is not limited to facial features, it can be widely used in some shot detection scenarios other than human shots. The embodiment of the present invention has a good detection effect for different shooting subjects such as human beings, objects, and empty shots. Therefore, based on the corresponding video detection results, it can guide relevant staff to determine the precise plan for intelligent creation of video data, thereby improving the viewing quality of video data.

[0094] Reference Figure 3 , another video detection method is shown, which may include:

[0095] S301: Obtain video data.

[0096] S302: Segment the video data into shots to determine a plurality of video segments.

[0097] In the embodiment of the present invention, the description of step S301 and step S302 may refer to the description of step S101 and step S102.

[0098] S303 , extract multiple key frames from each video segment to determine multiple target image sequences.

[0099] S304: Input each target image sequence into a scene classification model for processing, and identify the scene corresponding to each video clip.

[0100] In the embodiment of the present invention, since each video clip includes many frames of video images, considering that each frame of video images in each video clip is subjected to video detection, the complexity of the subsequent detection is greatly improved. Among them, the scene and camera movement of the lens are important and basic information for video detection. When the video clip is re-created or edited, there will not be a large number of different scene and all camera movements in the same lens. Therefore, in order to reduce the time complexity of the subsequent algorithm, multiple key frames can be extracted in each video clip in the order of video playback to form a target image sequence. Therefore, each target image sequence includes multiple key frames arranged in time sequence. When extracting key frames in the video clip, since the duration of each video clip is different, and the number of key frames in different target image sequences needs to be consistent. Therefore, the corresponding key frame extraction method can be determined according to the duration corresponding to each video clip. For example, when the duration of a video clip is 4S, a key frame can be extracted every 1S. At the same time, the change of the shot in the timing sequence needs to be considered in the process of camera movement recognition. Therefore, when extracting the key frames in the video clips, the number of key frames in each video clip is not less than 3. As an example, 3 key frames can be extracted from each video clip, which are recorded as the first key frame, the second key frame and the third key frame.

[0101] Each target image sequence is input into the scene classification model for processing, and the scene corresponding to the video clip can be identified by the scene classification model. The scene types may include at least: close-up, near shot, medium shot, panoramic shot, and distant shot. The scene classification model extracts features from each key frame in the target image sequence, and then performs feature recognition based on the extracted feature images to identify the corresponding scene.

[0102] As an implementation method of an embodiment of the present invention, the scene classification model may include a backbone network, a pooling layer and a classification layer. The backbone network is used to extract features from each key frame in the target image sequence to obtain a corresponding feature image. In the feature extraction process, the features corresponding to the depth of field of the photographed object in the key frame and its pixel ratio in the picture are abstracted. Then, the feature images are averagely pooled at the same time through the pooling layer. The feature image after average pooling is determined as a pooled image. Finally, the pooled image is classified by the scene classification layer to identify the scene corresponding to each video clip, wherein the backbone network may include multiple convolutional layers. In an example, the scene classification model may adopt TSN (Temporal Segment Networks) and the like.

[0103] As an implementation of an embodiment of the present invention, the method further includes the step of training a scene classification model: obtaining a sample image sequence including all scenes, wherein the sample image sequence includes multiple sample images, wherein the multiple sample images are key frames extracted from a video clip of a single shot. Inputting each sample image in the sample image sequence into the scene classification model for feature extraction. Wherein, during the training process, the scene classification layer outputs a scene probability vector corresponding to each scene predicted. The scene probability vector represents the probability distribution corresponding to each scene. And the scene corresponding to the maximum probability value in the scene probability vector is determined as the predicted scene corresponding to the sample image sequence. Thus, the loss function value corresponding to the scene classification model is determined by the difference between the real scene and the predicted scene of the multiple sample image sequences and the corresponding multiple scene probability vectors, wherein the loss function can adopt a multi-classification cross entropy loss function. According to the corresponding loss function value, the relevant parameters in the scene classification model are adjusted. When the loss function value meets the scene preset condition, such as the scene preset condition that the loss function value no longer decreases, or the decrease is very small, for example, the decrease does not exceed 1‰, it is determined that the scene classification model training is completed.

[0104] S305: Determine a reverse image sequence that is opposite to the playback timing of each target image sequence.

[0105] As an implementation of an embodiment of the present invention, the camera movement classification model includes a target classification network, a reverse order classification network and a weighted network. The inputting each target image sequence and the corresponding reverse order image sequence into the camera movement classification model for processing to identify the camera movement corresponding to each video clip may include: step S306-step S308.

[0106] S306: Input each target image sequence into the target classification network for processing, and output a corresponding target vector.

[0107] S307: Input the reversed image sequence corresponding to each target image sequence into the reversed classification network for processing, and output a corresponding reversed vector.

[0108] S308: Input the target vector and the corresponding reverse vector into the weighted network for weighting to determine the camera movement corresponding to each video clip.

[0109] In an embodiment of the present invention, when a camera movement effect is applied to a shot, the corresponding multi-frame images in the video clip will change. In order to ensure the accuracy of camera movement recognition, a reverse image sequence can be determined after the target image sequence is determined. The playback timing of the reverse image sequence is opposite to that of the target image sequence. In one example, the key frames in the target image sequence are arranged in reverse order according to time, and the corresponding reverse image sequence can be obtained. For example, the key frames in the target image sequence are arranged in time sequence as the first key frame, the second key frame, and the third key frame. The above three key frames are arranged in reverse order, and the obtained reverse image sequence is the third key frame, the second key frame, and the first key frame.

[0110] Secondly, the target image sequence and the reverse image sequence are simultaneously input into the camera movement classification model for processing, and the camera movement corresponding to the video clip can be identified by the camera movement classification model. In one example, the camera movement classification model may include a target classification network and a weighted network. Among them, the features corresponding to multiple key frames in the target image sequence are extracted by the target classification network, for example, the RGB features corresponding to each key frame, and the recognition is performed based on the extracted RGB features to determine the probability vector of the camera movement corresponding to the target image sequence, and this probability vector is used as the target vector. The RGB features corresponding to multiple key frames in the reverse image sequence are extracted by the target classification network, and feature recognition is performed based on the extracted RGB features to determine the probability vector of the camera movement corresponding to the reverse image sequence, and this probability vector is used as the reverse vector. Among them, the order in which the target image sequence and the reverse image sequence are input into the target classification network is not limited.

[0111] In another example, the camera movement classification model may include a target classification network, a reverse classification network, and a weighted network. The structures of the target classification network and the reverse classification network may be the same. The target classification network is used to extract RGB features corresponding to multiple key frames in the target image sequence, and feature recognition is performed based on the extracted RGB features to determine the probability vector of the camera movement corresponding to the target image sequence, and this probability vector is used as the target vector. The reverse classification network is used to extract RGB features corresponding to multiple key frames in the reverse image sequence, and feature recognition is performed on the extracted feature images to determine the probability vector of the camera movement corresponding to the reverse image sequence, and this probability vector is used as the reverse vector.

[0112] Before extracting RGB features, the camera movement classification model first extracts features from multiple key frames in the input image sequence to obtain corresponding feature images, and then performs image splicing on multiple feature images corresponding to the image sequence. After image splicing, a multi-dimensional feature vector is obtained. Image splicing refers to superimposing feature images corresponding to key frames along the channel direction of the corresponding image sequence. For example, the feature images corresponding to the first key frame, the second key frame and the third key frame are all length*width*3, and the feature image after image splicing of the first key frame, the second key frame and the third key frame is length*width*9. Then, the weighted operation is performed on the target vector and the reverse sequence vector through the weighted network to obtain a weighted target probability vector. Before performing the weighted operation, the camera movement predicted by the target vector and the camera movement predicted by the reverse sequence vector are pre-determined. The camera movement corresponding to the maximum probability value in the target vector is determined as the first camera movement, and the camera movement corresponding to the maximum probability value in the reverse sequence vector is determined as the second camera movement. The first camera movement and the second camera movement are matched, and the next weighted operation is determined according to the camera movement matching result. The camera movement matching results include consistent and inconsistent results, and different camera movement matching results correspond to different weighting operations.

[0113] In one example, when the first camera movement and the second camera movement are consistent, the weighted operation can be: adding the probabilities of the same camera movement corresponding to the target vector and the reverse vector respectively as the target probability vector, thereby determining the camera movement corresponding to the maximum probability in the target probability vector as the camera movement of the video clip. For example, the camera movement distribution corresponding to the target probability vector is static, moving, close-up and far-away in sequence. The target vector determined according to the first key frame, the second key frame and the third key frame is (0.6, 0.2, 0.1, 0.1), and the first camera movement that can be determined according to the target vector is static, and the reverse vector determined according to the third key frame, the second key frame and the first key frame is (0.7, 0.1, 0.2, 0), and the second camera movement that can be determined according to the reverse vector is also static. Thus, the target probability vector obtained is (1.3, 0.3, 0.3, 0.1). Thus, according to the target probability vector, it can be determined that the camera movement corresponding to the video clip is static.

[0114] In another example, when the camera movement determined by the target vector is inconsistent with the camera movement determined by the reverse vector, for example, when the real camera movement of the video clip is close-up, the camera movement determined by its reverse vector should be far-reaching. When the real camera movement of the video clip is far-reaching, the camera movement determined by its reverse vector should be close-up. The weighted operation can be: weighting the probability value corresponding to the first camera movement in the target vector to the probability value corresponding to the camera movement opposite to the first camera movement in the reverse vector to obtain a weighted reverse vector, and weighting the probability value corresponding to the second camera movement in the reverse vector to the probability value corresponding to the camera movement opposite to the second camera movement in the target vector to obtain a weighted target vector. The probability values ​​corresponding to the same camera movement in the weighted target vector and the weighted reverse vector are added respectively to obtain a target probability vector. And the camera movement corresponding to the maximum probability value in the target probability vector is determined as the camera movement corresponding to the video clip. Converting one of the camera movements to play in time sequence can obtain another camera movement, and it can be determined that one of the camera movements is the opposite of the other camera movement. For example, zoom out and zoom in are two filters that are opposite to each other.

[0115] For example, the target vector determined according to the first key frame, the second key frame and the third key frame is (0, 0.25, 0.7, 0.05), and the corresponding camera movement can be determined to be a close-up according to the target vector. The reverse vector determined according to the third key frame, the second key frame and the first key frame is (0, 0.1, 0.3, 0.6), and the corresponding camera movement can be determined to be a pull-out according to the reverse vector. Thus, the target probability vector obtained is (0, 0.35, 1.6, 1.35). Thus, according to the target probability vector, it can be determined that the camera movement corresponding to the video clip is a close-up. Similarly, the camera movement corresponding to each video clip is determined according to each target probability vector. The camera movement classification model combines the RGB features and timing information of multiple key frames to comprehensively identify the camera movement corresponding to the video clip, thereby improving the accuracy of camera movement recognition.

[0116] S309: Generate video detection results based on the scene and camera movement identified in each video clip.

[0117] In an embodiment of the present invention, based on the scene identified by the scene classification model and the camera movement identified by the camera movement classification model, a video detection result with high accuracy can be generated, and the video detection result can be displayed. The video detection result can be a combination of several scene and camera movement arranged in the playback sequence of the video data. Therefore, based on the corresponding video detection result, relevant staff can be guided to determine an accurate plan for intelligent creation of video data, thereby improving the viewing quality of the video data.

[0118] In one example, when the video data is a movie video to be edited, the video detection result of the entire movie can be determined based on the scene classification model and the camera movement classification model. For example, the video result can be: the movie only uses a small amount of motion and close-up camera movements, and only uses three scene types, such as close-up, medium shot and long shot. Relevant staff can determine the precise plan for intelligent creation of video data based on the video detection results. For example, add close-up, close-up and other scene effects to highlight the details of the movie. Add panoramic or long-range scene effects to highlight the environmental shots in the movie. For another example, add more motion camera movements to switch between different shots, thereby creating atmosphere and rhythm. For another example, some shots in the video data can be deleted. Make the shot conversion of the entire video data appropriate, and the plot context and rhythm clearer, thereby improving the quality of the movie and improving the integrity and appeal of the movie.

[0119] Considering that after the lens is applied with the camera movement effect, the corresponding multiple frames in the video clip will change. The image can be dynamically changed. As an implementation method of the embodiment of the present invention, the camera movement can be recognized not only by RGB features, but also by extracting optical flow features (optical flow features can be used to characterize the image changes between two key frames), thereby further improving the recognition accuracy of the camera movement.

[0120] The step of inputting each target image sequence into the target classification network for processing and outputting a corresponding target vector comprises:

[0121] The target image sequence is input into a first classification unit for processing to obtain a first classification vector determined based on the inter-frame optical flow features corresponding to the target image sequence.

[0122] The target image sequence is input into a second classification unit for processing to obtain a second classification vector determined based on RGB features corresponding to the target image sequence.

[0123] In one example, when the camera movement classification model only includes a target classification network and a weighted network, inputting the reversed image sequence into the target classification network for processing and outputting a corresponding reversed vector includes:

[0124] The reversed image sequence is input into a first classification unit for processing to obtain a third classification vector determined based on the inter-frame optical flow features corresponding to the reversed image sequence.

[0125] The reversed image sequence is input into a second classification unit for processing to obtain a fourth classification vector determined based on RGB features corresponding to the reversed image sequence.

[0126] In an embodiment of the present invention, the RGB features and inter-frame optical flow features of the target image sequence and the reverse image sequence can be combined to comprehensively determine the camera movement corresponding to the video clip. Since the basis for camera movement recognition considers the changes in the RGB features corresponding to all pixels of the entire image, it is not limited to detecting faces, and determining the scene and camera movement by the proportion of faces in the entire image and other factors. The present application also takes into account the inter-frame optical flow features between every two key frames to capture the motion trajectory of objects in the key frames, and is thus also applied to empty lens shots without reference objects. Therefore, the video detection method can be widely used in video detection scenes of different shooting objects such as people, objects, empty lenses, etc., and has a high-precision detection effect.

[0127] Reference Figure 4In one example, the target classification network includes a first classification unit and a second classification unit, wherein the first classification unit and the second classification unit may each include a feature extraction layer, a splicing layer and a classification layer. The first classification unit may also include an optical flow layer, through which an inter-frame optical flow image between two adjacent key frames can be determined. The feature extraction layer of the first classification unit is used to extract corresponding features in the inter-frame optical flow image and determine the feature image corresponding to the inter-frame optical flow image. The feature extraction layer of the second classification unit is used to extract features corresponding to the image sequence and determine the feature image corresponding to the image sequence. The splicing layer of the first classification unit is used to splice multiple feature images corresponding to the inter-frame optical flow image, and the splicing layer of the second classification unit is used to splice multiple feature images of the image sequence. The classification layer is used to extract features from the image spliced ​​by the splicing layer, and to identify and determine the probability vector corresponding to the target image sequence. The optical flow layer may use LiteFlowNet (a lightweight convolutional neural network for optical flow estimation). In one example, the feature extraction layer may use a convolutional network, and the classification layer may use a neural convolutional network.

[0128] The input target image sequence is processed by the first classification unit to obtain the probability vector of the camera movement corresponding to the target image sequence, and this probability vector is used as the first classification vector. The input target image sequence is processed by the second classification unit to obtain the probability vector of the camera movement corresponding to the target image sequence, and this probability vector is used as the second classification vector. The input reverse image sequence is processed by the first classification unit to obtain the probability vector of the camera movement corresponding to the reverse image sequence, and this probability vector is used as the third classification vector. The input reverse image sequence is processed by the second classification unit to obtain the probability vector of the camera movement corresponding to the reverse image sequence, and this probability vector is used as the fourth classification vector.

[0129] In another example, when the camera movement classification model includes a target classification network, a reverse order classification network and a weighted network, the reverse order classification network includes a third classification unit and a fourth classification unit. Inputting the reverse order image sequence into the reverse order classification network for processing and outputting a corresponding reverse order vector includes:

[0130] The reversed image sequence is input into a third classification unit for processing to obtain a third classification vector determined based on the inter-frame optical flow features corresponding to the reversed image sequence.

[0131] The reversed image sequence is input into a fourth classification unit for processing to obtain a fourth classification vector determined based on RGB features corresponding to the reversed image sequence.

[0132] In the embodiment of the present invention, referring to Figure 5As shown, the first classification unit, the second classification unit, the third classification unit and the fourth classification unit can all include a splicing layer and a classification layer. Among them, the function of the third classification unit is the same as that of the first classification unit, and the function of the second classification unit is the same as that of the fourth classification unit. In addition, the structure of the classification layer of the first classification unit is consistent with that of the third classification unit, and the structure of the classification layer of the second classification unit is consistent with that of the fourth classification unit. And when weighting is performed in the weighted network, the model weight parameters corresponding to the target classification network and the reverse classification network can be shared. Setting the target classification network and the reverse classification network at the same time can realize the simultaneous input of the target image sequence and the reverse image sequence, thereby improving the real-time performance of data processing. The first classification unit processes the input target image sequence to obtain the probability vector of the camera movement corresponding to the target image sequence, and the probability vector is used as the first classification vector. The second classification unit processes the input target image sequence to obtain the probability vector of the camera movement corresponding to the target image sequence, and the probability vector is used as the second classification vector. The third classification unit processes the input reverse image sequence to obtain the probability vector of the camera movement corresponding to the reverse image sequence, and the probability vector is used as the third classification vector. The fourth classification unit processes the input reversed image sequence to obtain a probability vector of the camera movement corresponding to the reversed image sequence, and uses the probability vector as the fourth classification vector.

[0133] As an implementation manner of an embodiment of the present invention, the step of inputting the target vector and the corresponding reversed vector into the weighted network for weighting to determine the camera movement corresponding to each video clip includes:

[0134] The first classification vector, the second classification vector, the third classification vector and the fourth classification vector are input into the weighted network for weighting to determine the camera movement corresponding to each video clip.

[0135] In an embodiment of the present invention, a weighted operation is performed on the first classification vector, the second classification vector, the third classification vector and the fourth classification vector through the weighted network to obtain a weighted target probability vector. The camera movement corresponding to the maximum probability in the target probability vector is determined as the camera movement of the video clip. Before performing the weighted operation, the camera movement predicted by each classification vector can be predetermined. For example, the camera movement corresponding to the maximum probability value in the first classification vector is determined as the first camera movement, the camera movement corresponding to the maximum probability value in the second classification vector is determined as the second camera movement, the camera movement corresponding to the maximum probability value in the third classification vector is determined as the third camera movement, and the camera movement corresponding to the maximum probability value in the fourth classification vector is determined as the fourth camera movement. The first camera movement is matched with the third camera movement (or the fourth camera movement), or the second camera movement is matched with the third camera movement (or the fourth camera movement). The weighted operation of the next step is determined according to the camera movement matching result, and the camera movement matching result includes consistency and inconsistency, and different camera movement matching results correspond to different weighted operations.

[0136] In one example, when the first camera movement, the second camera movement, the third camera movement and the fourth camera movement are all consistent, the weighted operation can be: adding the probability values ​​of the first classification vector, the second classification vector, the third classification vector and the fourth classification vector corresponding to the same camera movement respectively to determine the target probability vector.

[0137] For example, the first classification vector is (0.8, 0.05, 0.1, 0.05), and the first camera movement determined therefrom is still. The second classification vector is (0.9, 0.05, 0, 0.05), and the second camera movement determined therefrom is still. The third classification vector is (0.85, 0, 0.1, 0.05), and the third camera movement determined therefrom is still. The fourth classification vector is (0.7, 0, 0.05, 0.25), and the fourth camera movement determined therefrom is still. The first camera movement, the second camera movement, the third camera movement, and the fourth camera movement are all still. Thus, the target probability vector obtained is (3.25, 0.1, 0.25, 0.4). According to the target probability vector, it can be identified that the camera movement corresponding to each video clip is still.

[0138] In another example, when the first camera movement is consistent with the second camera movement, the third camera movement is consistent with the fourth camera movement, but the first camera movement (or the second camera movement) is inconsistent with the third camera movement (or the fourth camera movement), the weighted operation can be: respectively adding the probability values ​​corresponding to the same camera movement of the first classification vector and the second classification vector to obtain the target classification vector, and determining the camera movement corresponding to the maximum probability value in the target classification vector as the fifth camera movement. respectively adding the probability values ​​corresponding to the same camera movement of the third classification vector and the fourth classification vector to obtain the reverse classification vector, and determining the camera movement corresponding to the maximum probability value in the reverse classification vector as the sixth camera movement.

[0139] Next, the probability value corresponding to the fifth camera movement in the target classification is weighted to the probability value corresponding to the camera movement opposite to the fifth camera movement in the reverse classification vector to obtain a weighted reverse classification vector, the probability value corresponding to the sixth camera movement in the reverse classification vector is weighted to the probability value corresponding to the camera movement opposite to the sixth camera movement in the target classification vector to obtain a weighted target classification vector, and the probability values ​​corresponding to the same camera movement in the weighted target classification vector and the weighted reverse classification vector are added respectively to obtain a target probability vector.

[0140] For example, the first classification vector determined according to the first key frame, the second key frame and the third key frame is (0, 0.25, 0.7, 0.05), and the first camera movement determined according to it is a close-up. The second classification vector is (0, 0.2, 0.8, 0), and the second camera movement determined according to it is a close-up. The corresponding target classification vector is determined to be (0, 0.45, 1.5, 0.05). The third classification vector determined according to the third key frame, the second key frame and the first key frame is (0, 0.1, 0.3, 0.6), and the third camera movement determined according to it is a far-away. The fourth classification vector is (0, 0.05, 0.05, 0.9), and the fourth camera movement determined according to it is a far-away. The corresponding reverse classification vector is determined to be (0, 0.15, 0.35, 1.5). Thus, the target probability vector obtained is (0, 0.6, 3.35, 3.05). According to the target probability vector, it can be determined that the camera movement corresponding to the video clip is a close-up. The camera movement classification model combines the RGB features and timing information of multiple key frames to comprehensively identify the camera movement corresponding to the video clip, thereby improving the accuracy of camera movement identification.

[0141] As an implementation mode of an embodiment of the present invention, the method further includes the step of training a camera movement classification model: obtaining a sample image sequence including all camera movements and a corresponding first image sequence, wherein the first image sequence is an image sequence having a timing opposite to that of the sample image sequence, and the sample image sequence includes a plurality of sample images. The sample image is a key frame extracted from a video clip of a single shot. Each sample image sequence and the corresponding first image sequence are respectively input into the camera movement classification model for identification to determine the corresponding camera movement. During the training process, the camera movement classification model can output a camera movement probability vector, wherein the camera movement probability vector represents the probability distribution corresponding to each camera movement. The camera movement corresponding to the maximum probability value in the camera movement probability vector is determined as the camera movement predicted by the camera movement classification model. Therefore, through the differences between the real camera movements and the identified camera movements of multiple sample image sequences and the corresponding multiple camera movement probability vectors, the loss function value corresponding to the camera movement classification model is determined, and the relevant parameters in the camera movement classification model are adjusted according to the loss function value. When the loss function value meets the camera movement preset condition, such as the camera movement preset condition may be that the loss function no longer decreases, or the decrease is very small, for example, the decrease does not exceed 1‰, etc., then it is determined that the camera movement classification model training is completed.

[0142] Considering that when extracting key frames from a video clip, the scene may change within the time interval of extracting multiple key frames. Therefore, in order to improve the recognition accuracy of the scene, as an implementation of an embodiment of the present invention, the method may further include:

[0143] Determine whether to update the scene size according to the attribute information of the camera movement.

[0144] In an embodiment of the present invention, the attribute information can be understood as relevant information obtained based on the analysis of the camera movement, for example, the attribute information can include a distribution rule of the scene. In one example, when the camera movement is static, the scene corresponding to the video clip should be the same. In another example, when the camera movement is moving, the scene corresponding to the video clip may change. In another example, when the camera movement is close-up, the scene corresponding to the video clip will gradually enlarge, for example, from a panoramic view to a medium view, from a medium view to a close view, etc. In another example, when the camera movement is far-away, the scene corresponding to the video clip will gradually shrink, for example, from a close-up to a close view, from a close view to a medium view, etc. Thus, the scene sequence corresponding to the video clip can be determined for the type corresponding to the identified camera movement, wherein the scene sequence includes multiple scenes. For example, the video clip can be divided into target clips in sequence according to a certain time interval. For example, the video clip is divided into 3 target clips. Multiple key frames are extracted from each of the obtained target clips to determine a fragment image sequence. The method for extracting the fragment image sequence can refer to the method for extracting the above-mentioned target image sequence. Thus, the scene classification model is used to identify the scene corresponding to each target segment. The scenes are arranged in time sequence to form a scene sequence corresponding to the video segment. For example, the scene sequence includes 3 scenes.

[0145] In one example, when the camera movement is identified as static, the scene in the scene sequence of the corresponding video clip will not change, and thus, it is determined that the scene corresponding to the video clip is not updated. In another example, when the camera movement is identified as motion, the scene corresponding to the video clip will change. Thus, the scene sequence corresponding to the video clip can be determined. For example, the scene sequence can be (close-up, close-up, close-up), and when the scene corresponding to the previously determined video clip is close-up, it can also be determined that the scene corresponding to the video clip is not updated. If the scene corresponding to the previously determined video clip is close-up, the scene is updated to close-up. In another example, when the camera movement is zoomed out, for example, the scene sequence can be (close-up, medium shot, medium shot), and when the scene corresponding to the previously determined video clip is medium shot, it can be determined that the scene corresponding to the video clip is not updated. In another example, when the camera movement is close-up, for example, the scene sequence can be (medium shot, close-up, close-up), and when the scene corresponding to the previously determined video clip is medium shot, the corresponding scene needs to be updated to close-up. Since the camera movement classification model has a high recognition accuracy, when the scene comparison result is unreasonable, the scene recognized by the video clip is updated according to the scene sequence.

[0146] It should be noted that, for the method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the embodiments of the present application are not limited by the described order of actions, because according to the embodiments of the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of the present application.

[0147] Reference Figure 6 , shows a video detection device, which may include:

[0148] The video acquisition module 601 is used to acquire video data.

[0149] The video segmentation module 602 is used to segment the video data into shots to determine a plurality of video segments.

[0150] The key frame extraction module 603 is used to extract multiple key frames from each video segment to determine multiple target image sequences.

[0151] The scene recognition module 604 is used to input each target image sequence into the scene classification model for processing, and recognize the scene corresponding to each video clip.

[0152] The reverse order determination module 605 is used to determine a reverse order image sequence that is opposite to the playback time sequence of each target image sequence.

[0153] The camera movement identification module 606 is used to input each target image sequence and the corresponding reverse image sequence into the camera movement classification model for processing, so as to identify the camera movement corresponding to the video clip.

[0154] The detection result generation module 607 is used to generate video detection results based on the scene and camera movement identified in each video clip.

[0155] As an implementation manner of an embodiment of the present invention, the camera movement classification model includes a target classification network and a weighted network, or includes the target classification network, a reverse classification network and the weighted network; the camera movement recognition module 606 may include:

[0156] The target vector determination submodule is used to input each target image sequence into the target classification network for processing and output the corresponding target vector.

[0157] The reverse order vector determination submodule is used to input the reverse order image sequence corresponding to each target image sequence into the target classification network / the reverse order classification network for processing, and output the corresponding reverse order vector.

[0158] The camera movement identification submodule is used to input the target vector and the reverse vector into the weighted network for weighting, so as to determine the camera movement corresponding to each video clip.

[0159] As an implementation manner of an embodiment of the present invention, the camera movement recognition submodule may include:

[0160] The first camera movement determining unit determines a first camera movement through the target vector, and determines a second camera movement through the inverse vector.

[0161] The first camera movement determination unit is used to add the probability values ​​corresponding to the same camera movement in the target vector and the corresponding reverse vector respectively to obtain a target probability vector if the first camera movement and the second camera movement are consistent.

[0162] It is also used for, if the first camera movement and the second camera movement are inconsistent, weighting the probability value corresponding to the first camera movement in the target vector to the probability value corresponding to the camera movement opposite to the first camera movement in the inverse vector to obtain a weighted inverse vector, weighting the probability value corresponding to the second camera movement in the inverse vector to the probability value corresponding to the camera movement opposite to the second camera movement in the target vector to obtain a weighted target vector, and adding the probability values ​​corresponding to the same camera movement in the weighted target vector and the weighted inverse vector respectively to obtain a target probability vector.

[0163] The second camera movement determination unit is used to determine the camera movement corresponding to each video clip according to the target probability vector.

[0164] As an implementation of an embodiment of the present invention, the target classification network includes a first classification unit and a second classification unit. The target vector determination submodule may include:

[0165] The first classification vector determination unit is used to input the target image sequence into the first classification unit for processing to obtain a first classification vector determined based on the inter-frame optical flow feature corresponding to the target image sequence.

[0166] The second classification vector determination unit is used to input the target image sequence into the second classification unit for processing to obtain a second classification vector determined based on the RGB features corresponding to the target image sequence.

[0167] As an implementation manner of an embodiment of the present invention, the reverse sequence vector determination submodule may include:

[0168] The third classification vector determination unit is used to input the reversed image sequence into the first classification unit for processing to obtain a third classification vector determined based on the inter-frame optical flow features corresponding to the reversed image sequence.

[0169] The fourth classification vector determination unit is used to input the reversed image sequence into the second classification unit for processing to obtain a fourth classification vector determined based on the RGB features corresponding to the reversed image sequence.

[0170] As an implementation of an embodiment of the present invention, the reverse order classification network includes a third classification unit and a fourth classification unit. The reverse order vector determination submodule may include:

[0171] The third classification vector determination unit is used to input the reversed image sequence into the third classification unit for processing to obtain a third classification vector determined based on the inter-frame optical flow feature corresponding to the reversed image sequence.

[0172] The fourth classification vector determination unit is used to input the reversed image sequence into the fourth classification unit for processing to obtain a fourth classification vector determined based on the RGB features corresponding to the reversed image sequence.

[0173] As an implementation manner of the embodiment of the present invention, the camera movement identification submodule may also be used for:

[0174] The first classification vector, the second classification vector, the third classification vector and the fourth classification vector are input into the weighted network for weighting to determine the camera movement corresponding to each video clip.

[0175] As an implementation manner of the embodiment of the present invention, the camera movement recognition submodule may further include:

[0176] The first camera movement determination unit is used to determine a first camera movement through the first classification vector, determine a second camera movement through the second classification vector, determine a third camera movement through the third classification vector, and determine a fourth camera movement through the fourth classification vector.

[0177] The first camera movement determination unit is used to add the probability values ​​corresponding to the same camera movement in the first classification vector, the second classification vector, the third classification vector and the fourth classification vector respectively to obtain a target probability vector if the first camera movement, the second camera movement, the third camera movement and the fourth camera movement are all consistent.

[0178] It is also used to add the probability values ​​corresponding to the same camera movement of the first classification vector and the second classification vector respectively to obtain a target classification vector if the first camera movement, the second camera movement, the third camera movement and the fourth camera movement are inconsistent, and determine the camera movement corresponding to the maximum probability value in the target classification vector as the fifth camera movement; and add the probability values ​​corresponding to the same camera movement of the third classification vector and the fourth classification vector respectively to obtain a reverse classification vector, and determine the camera movement corresponding to the maximum probability value in the reverse classification vector as the sixth camera movement.

[0179] It is also used to weight the probability value corresponding to the fifth camera movement in the target classification vector to the probability value corresponding to the camera movement opposite to the fifth camera movement in the reverse classification vector to obtain a weighted reverse classification vector, weight the probability value corresponding to the sixth camera movement in the reverse classification vector to the probability value corresponding to the camera movement opposite to the sixth camera movement in the target classification vector to obtain a weighted target classification vector, and add the probability values ​​corresponding to the same camera movement in the weighted target classification vector and the weighted reverse classification vector respectively to obtain a target probability vector.

[0180] The second camera movement determination unit is used to determine the camera movement corresponding to each video clip according to the target probability vector.

[0181] As an implementation of an embodiment of the present invention, the scene classification model includes a backbone network, a pooling layer and a classification layer. The scene recognition module may include:

[0182] The feature extraction submodule is used to input each target image sequence into the backbone network to extract features and determine the corresponding feature image.

[0183] The image pooling submodule is used to input each feature image into the pooling layer for average pooling to determine the corresponding pooled image.

[0184] The scene recognition submodule is used to input each pooled image into the scene classification layer for classification operation to determine the scene corresponding to each video clip.

[0185] As an implementation manner of the embodiment of the present invention, the device may further include:

[0186] The scene updating module is used to determine whether to update the scene according to the attribute information of the camera movement.

[0187] As an implementation manner of the embodiment of the present invention, the scene updating module may include:

[0188] The shot determination submodule is used to determine the shot sequence corresponding to the camera movement based on the attribute information of the camera movement.

[0189] The scene comparison submodule is used to compare the scene in the scene sequence with the scene corresponding to the video clip.

[0190] The scene updating submodule is used to determine whether to update the scene corresponding to the video clip according to the comparison result.

[0191] The embodiment of the present invention further provides an electronic device, such as Figure 7As shown, it includes a processor 701, a communication interface 702, a memory 703 and a communication bus 704, wherein the processor 701, the communication interface 702, and the memory 703 communicate with each other through the communication bus 704.

[0192] Memory 703, used for storing computer programs;

[0193] The processor 701 is used to execute the program stored in the memory 703 to implement the following steps:

[0194] Get video data;

[0195] Segmenting the video data into shots to determine a plurality of video segments;

[0196] Extract multiple key frames from each video clip to determine multiple target image sequences;

[0197] Input each target image sequence into the scene classification model for processing, and identify the scene corresponding to each video clip;

[0198] Determine a reverse image sequence having a playback timing opposite to that of each target image sequence;

[0199] Inputting each target image sequence and the corresponding reverse image sequence into a camera movement classification model for processing to identify the camera movement corresponding to the video clip;

[0200] The video detection results are generated based on the scene and camera movement identified in each video clip.

[0201] The communication bus mentioned in the above terminal can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0202] The communication interface is used for communication between the above terminal and other devices.

[0203] The memory may include a random access memory (RAM) or a non-volatile memory, such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0204] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0205] In another embodiment of the present invention, a computer-readable storage medium is provided. The computer-readable storage medium stores instructions. When the instructions are executed on a computer, the computer executes the video detection method described in any one of the above embodiments.

[0206] In another embodiment of the present invention, a computer program product including instructions is provided. When the computer program product is run on a computer, the computer executes the video detection method described in any one of the above embodiments.

[0207] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented by software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk Solid State Disk (SSD)), etc.

[0208] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0209] Each embodiment in this specification is described in a related manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0210] The above description is only a preferred embodiment of the present invention and is not intended to limit the protection scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention are included in the protection scope of the present invention.

Claims

1. A video detection method, characterized in that: The method comprises: Get video data; Segmenting the video data into shots to determine a plurality of video segments; Extract multiple key frames from each video clip to determine multiple target image sequences; Input each target image sequence into the scene classification model for processing, and identify the scene corresponding to each video clip; Determine a reverse image sequence having a playback timing opposite to that of each target image sequence; Inputting each target image sequence and the corresponding reverse order image sequence into a camera movement classification model for processing, and identifying the camera movement corresponding to each video clip; the camera movement classification model includes a target classification network and a weighted network, or includes the target classification network, the reverse order classification network and the weighted network; Generate video detection results based on the scene and camera movement identified in each video clip; The step of inputting each target image sequence and the corresponding reversed image sequence into a camera movement classification model for processing to identify the camera movement corresponding to each video clip includes: Input each target image sequence into the target classification network for processing, and output the corresponding target vector; Inputting the reverse order image sequence corresponding to each target image sequence into the target classification network or the reverse order classification network for processing, and outputting the corresponding reverse order vector; The target vector and the corresponding inverse vector are input into the weighted network for weighting to determine the camera movement corresponding to each video clip.

2. The method according to claim 1, characterized in that Inputting the target vector and the corresponding reverse vector into the weighted network for weighting to determine the camera movement corresponding to each video clip includes: Determine a first camera movement by using the target vector, and determine a second camera movement by using the reverse vector; If the first camera movement is consistent with the second camera movement, respectively add the probability values ​​corresponding to the same camera movement in the target vector and the corresponding reverse vector to obtain a target probability vector; If the first camera movement and the second camera movement are inconsistent, weighting the probability value corresponding to the first camera movement in the target vector to the probability value corresponding to the camera movement opposite to the first camera movement in the reverse vector to obtain a weighted reverse vector, weighting the probability value corresponding to the second camera movement in the reverse vector to the probability value corresponding to the camera movement opposite to the second camera movement in the target vector to obtain a weighted target vector, and adding the probability values ​​corresponding to the same camera movement in the weighted target vector and the weighted reverse vector respectively to obtain a target probability vector; The camera movement corresponding to each video clip is determined according to the target probability vector.

3. The method according to claim 1, characterized in that The target classification network includes a first classification unit and a second classification unit; The step of inputting each target image sequence into the target classification network for processing and outputting a corresponding target vector comprises: Inputting the target image sequence into a first classification unit for processing to obtain a first classification vector determined based on the inter-frame optical flow features corresponding to the target image sequence; The target image sequence is input into a second classification unit for processing to obtain a second classification vector determined based on the RGB features of the target image sequence.

4. The method according to claim 3, characterized in that The step of inputting the reversed image sequence into the target classification network for processing and outputting a corresponding reversed vector comprises: Inputting the reversed image sequence into a first classification unit for processing to obtain a third classification vector determined based on the inter-frame optical flow features corresponding to the reversed image sequence; The reversed image sequence is input into a second classification unit for processing to obtain a fourth classification vector determined based on RGB features corresponding to the reversed image sequence.

5. The method according to claim 3, characterized in that: The reverse classification network includes a third classification unit and a fourth classification unit; The step of inputting the reversed image sequence into the reversed classification network for processing and outputting a corresponding reversed vector comprises: Inputting the reversed image sequence into a third classification unit for processing to obtain a third classification vector determined based on the inter-frame optical flow features corresponding to the reversed image sequence; The reversed image sequence is input into a fourth classification unit for processing to obtain a fourth classification vector determined based on RGB features corresponding to the reversed image sequence.

6. The method according to claim 4 or 5, characterized in that: The step of inputting the target vector and the corresponding reverse vector into the weighted network for weighting to determine the camera movement corresponding to each video clip includes: The first classification vector, the second classification vector, the third classification vector and the fourth classification vector are input into the weighted network for weighting to determine the camera movement corresponding to each video clip.

7. The method according to claim 6, characterized in that The step of inputting the first classification vector, the second classification vector, the third classification vector, and the fourth classification vector into the weighted network for weighting to determine the camera movement corresponding to each video clip includes: Determine a first camera movement by using the first classification vector, determine a second camera movement by using the second classification vector, determine a third camera movement by using the third classification vector, and determine a fourth camera movement by using the fourth classification vector; If the first camera movement, the second camera movement, the third camera movement and the fourth camera movement are all the same, the probability values ​​corresponding to the same camera movement in the first classification vector, the second classification vector, the third classification vector and the fourth classification vector are added together to obtain a target probability vector; If the first camera movement, the second camera movement, the third camera movement, and the fourth camera movement are inconsistent, the probability values ​​corresponding to the same camera movement in the first classification vector and the second classification vector are added respectively to obtain a target classification vector, the camera movement corresponding to the maximum probability value in the target classification vector is determined as the fifth camera movement, and the probability values ​​corresponding to the same camera movement in the third classification vector and the fourth classification vector are added respectively to obtain a reverse classification vector, and the camera movement corresponding to the maximum probability value in the reverse classification vector is determined as the sixth camera movement; Weighting the probability value corresponding to the fifth camera movement in the target classification vector to the probability value corresponding to the camera movement opposite to the fifth camera movement in the reverse classification vector to obtain a weighted reverse classification vector, weighting the probability value corresponding to the sixth camera movement in the reverse classification vector to the probability value corresponding to the camera movement opposite to the sixth camera movement in the target classification vector to obtain a weighted target classification vector, and adding the probability values ​​corresponding to the same camera movement in the weighted target classification vector and the weighted reverse classification vector respectively to obtain a target probability vector; The camera movement corresponding to each video clip is determined according to the target probability vector.

8. The method according to claim 1, characterized in that The scene classification model includes a backbone network, a pooling layer and a classification layer; The step of inputting each target image sequence into the scene classification model for processing and identifying the scene corresponding to each video clip includes: Input each target image sequence into the backbone network to extract features and determine the corresponding feature image; Input each feature image into the pooling layer for average pooling to determine the corresponding pooled image; Each pooled image is input into the scene classification layer for classification operation to determine the scene corresponding to each video clip.

9. The method according to claim 1, characterized in that: The method further comprises: Determine whether to update the scene size according to the attribute information of the camera movement.

10. The method according to claim 9, characterized in that The step of determining whether to update the scene scale according to the attribute information of the camera movement includes: Determining a scene sequence corresponding to the camera movement according to the attribute information of the camera movement; Comparing the scene in the scene sequence with the scene corresponding to the video clip; Determine whether to update the scene corresponding to the video clip according to the comparison result.

11. The method according to claim 1, characterized in that: The method further comprises a training step of training the camera movement classification model, wherein the training step comprises: Acquire a sample image sequence and a first image sequence corresponding to all camera movements, wherein the first image sequence is an image sequence having a playback time sequence opposite to that of the sample image sequence, and the sample image sequence includes a plurality of sample images; Inputting each of the sample image sequences and the corresponding first image sequence into the camera movement classification model respectively, and identifying the camera movement corresponding to the sample image sequence; According to the real camera movements and the identified camera movements of the sample image sequence, the parameters of the camera movement classification model are adjusted to determine a trained camera movement classification model.

12. A video detection device, characterized in that: The device comprises: A video acquisition module, used to acquire video data; A video switching module, used to segment the video data into shots to determine a plurality of video segments; A key frame extraction module is used to extract multiple key frames from each video clip and determine multiple target image sequences; A scene recognition module is used to input each target image sequence into the scene classification model for processing and identify the scene corresponding to each video clip; A reverse order determination module, used to determine a reverse order image sequence opposite to the playback time sequence of each target image sequence; A camera movement recognition module, used for inputting each target image sequence and the corresponding reverse order image sequence into a camera movement classification model for processing, and identifying the camera movement corresponding to the video clip; the camera movement classification model includes a target classification network and a weighted network, or includes the target classification network, the reverse order classification network and the weighted network; A detection result generation module is used to generate video detection results based on the scene and camera movement identified in each video clip; Wherein, the camera movement recognition module includes: A target vector determination submodule is used to input each target image sequence into the target classification network for processing and output a corresponding target vector; A reverse order vector determination submodule, used for inputting the reverse order image sequence corresponding to each target image sequence into the target classification network or the reverse order classification network for processing, and outputting the corresponding reverse order vector; The camera movement identification submodule is used to input the target vector and the reverse vector into the weighted network for weighting, so as to determine the camera movement corresponding to each video clip.

13. An electronic device, characterized in that: It includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus; Memory, used to store computer programs; A processor, for implementing the method steps described in any one of claims 1-11 when executing a program stored in a memory.

14. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 11 is implemented.

Images