A video key frame extraction method based on an FD-SPnet network

Abstract

The application discloses a video key frame extraction method based on an FD-SPNet network. The method comprises the following steps: establishing the FD-SPNet network; inputting a motion video into the FD-SPNet network for training to obtain the trained FD-SPNet network; inputting a to-be-tested motion video into the trained FD-SPNet network to output feature information of a plurality of to-be-extracted video key frames in the to-be-tested motion video; inputting the feature information of each to-be-extracted video key frame into a Fast-NMS network; and outputting a plurality of final video key frames in the to-be-tested motion video by the Fast-NMS network, so as to realize the extraction of the video key frames. The method realizes the selection of the key frames in combination with the "coarse" and "fine", strengthens the timeliness, and guarantees the accuracy.

Description

TECHNICAL FIELD

[0001] The application relates to a video key frame extraction method, in particular to a video key frame extraction method based on an FD-SPNet network. BACKGROUND

[0002] In recent years, the motion target detection in a video sequence image is applied more and more widely in many fields such as only monitoring, video compression, automatic navigation, human-computer interaction and virtual reality. In order to process the video more conveniently, obtain the main content of the video more quickly and optimize the identification efficiency, a key frame extraction method is proposed. The key frame extraction algorithm commonly includes five types of methods, i.e. a traditional method, a time sampling method, a clustering method, a video segmentation method and a deep learning method.

[0003] The video frame selection method based on the traditional method runs fast, but the error rate is large; the method of inter-frame time average pooling is used at present to extract inter-frame region correlation features and mean features, but the time sampling method ignores the motion information; the clustering method does not consider the time information and is not suitable for the key frame selection of action videos with time information requirements; the video segmentation algorithm based on the video segmentation algorithm divides the shots according to the mutation boundary and is not suitable for action videos in a complex background; a currently proposed video key segment and key frame extraction method based on deep learning mainly extracts features of video frames through a self-encoder, selects key frames by comparing the feature differences between the video frames, and the frame selection method based on deep learning can effectively capture high-dimensional information and space-time information, but relying on a multilayer network to improve the precision will consume the running time. SUMMARY

[0004] In order to solve the problems in the background art, the application provides a video key frame extraction method based on an FD-SPNet network. The improved frame selection network FDNet (Iterative Frame Difference+Sliding Frame Difference) and the SPNet (Smooth Point Net) are combined to realize the key frame selection of the combination of rough and fine, which guarantees the selection accuracy and speeds up the frame selection speed.

[0005] The technical scheme adopted by the application is:

[0006] The video key frame extraction method provided by the application comprises the following steps:

[0007] Step 1) establishing an FD-SPNet network.

[0008] Step 2) inputting a plurality of action videos with action category labels into the FD-SPNet network for training to obtain the trained FD-SPNet network.

[0009] Step 3) input the action video to be tested into the trained FD-SPNet network, the trained FD-SPNet network outputs the feature information of several video key frames to be extracted in the action video to be tested, and the feature information of each video key frame to be extracted is input into the Fast-NMS network, and the Fast-NMS network outputs several final video key frames in the action video to be tested, so as to realize the extraction of the video key frames.

[0010] In the step 1), the established FD-SPNet network comprises a rough frame selection model FDNet and a deep learning frame selection model SPNet connected in sequence; after each action video with an action category label is input into the rough frame selection model FDNet in the FD-SPNet network, the rough frame selection model FDNet performs rough selection of the video key frames and outputs a sequence of the rough selected video key frames, and the rough selected video key frames are input into the deep learning frame selection model SPNet according to the sequence of the rough selected video key frames, and the deep learning frame selection model SPNet outputs the feature information of several video key frames to be extracted in each action video with an action category label.

[0011] In the step 2), several action videos with an action category label are input into the FD-SPNet network for training, and first, each action video with an action category label is input into the rough frame selection model FDNet in the FD-SPNet network, the rough frame selection model FDNet performs rough selection of the video key frames and outputs a sequence of the rough selected video key frames, and for each action video with an action category label, the following steps are performed:

[0012] 2.1) first, the vector values of all video frames in the action video are calculated to obtain the mean value, the maximum value and the minimum value of the vector values, and the video frames with the vector values equal to the maximum value and the minimum value and the vector values closest to the mean value are extracted as the maximum vector max frame, the minimum vector min frame and the average vector avg frame, and the maximum vector max frame, the minimum vector min frame and the average vector avg frame are taken as three fixed frames.

[0013] 2.2) the iterative frame difference method is adopted to sequentially perform frame difference operation on the video frames other than the three fixed frames in step 2.1) based on the fixed frames to obtain a sequence of several video frames.

[0014] 2.3) the sliding frame difference method is adopted to sequentially perform frame difference operation on the video frames other than the three fixed frames in step 2.1) based on the fixed frames to obtain a sequence of several video frames.

[0015] 2.4) performing an or operation on the sequence of video frames obtained in step 2.2) and the sequence of video frames obtained in step 2.3), and finally obtaining a sequence of video frames of the rough frame selection of key frames.

[0016] In step 2.2), the iterative frame difference method is used to perform frame difference operation on the video frames in the action video other than the three fixed frames in step 2.1) based on the three fixed frames. Specifically, a number of video frames before the three fixed frames, a number of video frames between every two fixed frames, and a number of video frames after the three fixed frames are respectively grouped into iterative frame difference sets, i.e. four iterative frame difference sets. For each iterative frame difference set, frame difference operation is performed on every two adjacent video frames in the iterative frame difference set. When performing the frame difference operation for the first time, if the frame difference between the first video frame and the second video frame is greater than a preset initial frame difference threshold, the sequence of the first video frame is retained; if the frame difference between the first video frame and the second video frame is less than or equal to the preset initial frame difference threshold, the sequence of the first video frame is discarded. Then, the same frame difference operation as that between the first video frame and the second video frame is continued to be performed on the second video frame and the next video frame of the second video frame. The frame difference threshold in the last frame difference operation is reduced by a preset difference value to be used as the frame difference threshold in the current frame difference operation. The frame difference operation is iterated until the number of video frames in the sequence is greater than a preset number of frames, and the sequence of video frames is output.

[0017] When performing the frame difference operation on the next iterative frame difference set, the last frame difference threshold of the previous iterative frame difference set is reduced by a preset difference value to be used as the first frame difference threshold of the next iterative frame difference set.

[0018] In step 2.3), the other video frames in the action video except the three fixed frames in step 2.1) are sequentially subjected to frame difference operation based on the fixed frames by using the sliding frame difference method. Specifically, a plurality of video frames before the three fixed frames, a plurality of video frames between every two fixed frames, and a plurality of video frames after the three fixed frames are respectively grouped into sliding frame difference sets, i.e., four sliding frame difference sets. For each fixed frame and the two sliding frame difference sets before and after the fixed frame, first, each video frame in the sliding frame difference set before the fixed frame is sequentially subjected to frame difference operation with the fixed frame. When the frame difference between the first video frame in the sliding frame difference set before the fixed frame and the fixed frame is greater than a preset initial frame difference threshold in the first frame difference operation, the sequence of the first video frame is retained. When the frame difference between the first video frame in the sliding frame difference set before the fixed frame and the fixed frame is less than or equal to the preset initial frame difference threshold, the sequence of the first video frame is discarded. Then, the same frame difference operation as that of the first video frame and the fixed frame is continued from the second video frame in the sliding frame difference set before the fixed frame and the fixed frame. The frame difference threshold in the last frame difference operation is reduced by a preset difference value to serve as the frame difference threshold in the current frame difference operation in each loop of the frame difference operation. The fixed frame and each video frame in the sliding frame difference set after the fixed frame are subjected to frame difference operation. For each video frame in the sliding frame difference set after the fixed frame, when the frame difference between the video frame and the fixed frame is greater than the current frame difference threshold, the sequence of the video frame is retained. When the frame difference between the video frame and the fixed frame is less than or equal to the current frame difference threshold, the sequence of the video frame is discarded. The frame difference threshold in the last frame difference operation is reduced by a preset difference value to serve as the frame difference threshold in the current frame difference operation in each loop of the frame difference operation. The sliding frame difference is stopped and the retained sequence of video frames is outputted until the number of the retained sequence of video frames is greater than a preset number of frames.

[0019] In the frame difference operation of the next sliding frame difference set, the last frame difference threshold of the previous sliding frame difference set is reduced by a preset difference value to serve as the first frame difference threshold of the next sliding frame difference set.

[0020] The deep learning frame selection model SPNet includes 12 3*3 convolution layers, a channel attention module and a feature fusion module MSCA, the input of the deep learning frame selection model SPNet is input into the channel attention module and the second 3*3 convolution layer for processing after being input into the first 3*3 convolution layer, the output of the channel attention module and the output of the second 3*3 convolution layer are jointly input into the feature fusion module MSCA for processing, the output of the first 3*3 convolution layer and the output of the feature fusion module MSCA are multiplied through a residual operation to obtain a multiplication result, the output of the second 3*3 convolution layer is sequentially input into the third to eleventh 3*3 convolution layers for processing, the output of the eleventh 3*3 convolution layer and the multiplication result are jointly input into the twelfth 3*3 convolution layer for processing and output as the output of the deep learning frame selection model SPNet.

[0021] The output of the eleventh 3*3 convolution layer is high-level feature, and the multiplication result is low-level feature, for the feature information required by the descriptor, the low-level feature of the model is fused with the high-level information through the channel attention module, the feature information is strengthened, and the Fast-NMS network extracts more accurate points and descriptors.

[0022] The channel attention module includes a global average pooling layer GAP, a depth separable convolution layer PWC and a fully connected layer Dense connected in sequence.

[0023] In the step 3), the feature information of each video key frame to be extracted is input into the Fast-NMS network, and the Fast-NMS network outputs a plurality of final video key frames in the action video to be detected, specifically, the feature information of each video key frame to be extracted is input into the Fast-NMS network to extract points and descriptors to obtain matrix information of each video key frame to be extracted, the distance between the matrix information of every two adjacent video key frames is calculated, when the calculated distance is greater than the descriptor matching threshold, the previous frame in the two adjacent video key frames is extracted, when the calculated distance is less than or equal to the descriptor matching threshold, the video key frame is not extracted, and finally a plurality of extracted video key frames are output as the final video key frames.

[0024] The beneficial effects of the present application are:

[0025] 1) The present application proposes a rough frame selection model FDNet, proposes a sliding frame difference and an iterative frame difference method, and combines the two methods to realize adaptive selection of frames and improve the accuracy of frame selection.

[0026] 2) The present application proposes a deep learning frame selection model SPNet, designs a new deep learning method to extract feature information, and combines the Fast-NMS network to extract key frames, so that the key frame selection is more accurate.

[0027] 3) The application combines the coarse frame selection model FDNet and the deep learning frame selection model SPNet, realizes the key frame selection of "coarse and fine" combination, strengthens the timeliness, and guarantees the accuracy. BRIEF DESCRIPTION OF DRAWINGS

[0028] Figure 1 The technical scheme flow chart of the method of the application. DETAILED DESCRIPTION

[0029] The application will be further described in detail below in combination with the drawings and specific embodiments.

[0030] The data set used in the specific implementation of the application is the largest open-source sign language data set CSL in China. Figure 1 As shown in the figure, the action video to be tested in the data set is input into the trained FD-SPNet network, the trained FD-SPNet network outputs the feature information of the several video key frames to be extracted in the action video to be tested, the feature information of each video key frame to be extracted is input into the Fast-NMS network, and the Fast-NMS network outputs the several final video key frames in the action video to be tested, so as to realize the extraction of the video key frames.

[0031] The video key frame extraction method of the application comprises the following steps:

[0032] Step 1) establishing an FD-SPNet network.

[0033] In step 1), the established FD-SPNet network comprises a coarse frame selection model FDNet and a deep learning frame selection model SPNet connected in sequence; each action video with an action category label is input into the coarse frame selection model FDNet in the FD-SPNet network, the video key frames are coarsely selected, and the sequence of the coarsely selected video key frames is output; according to the sequence of the coarsely selected video key frames, the coarsely selected video key frames are input into the deep learning frame selection model SPNet, and the deep learning frame selection model SPNet outputs the feature information of the several video key frames to be extracted in each action video with an action category label.

[0034] The deep learning frame selection model SPNet includes 12 3x3 convolutional layers, a channel attention module, and a feature fusion module MSCA. The input of the deep learning frame selection model SPNet is input into the channel attention module and the second 3x3 convolutional layer after being input into the first 3x3 convolutional layer. The output of the channel attention module and the output of the second 3x3 convolutional layer are jointly input into the feature fusion module MSCA for processing. The output of the first 3x3 convolutional layer and the output of the feature fusion module MSCA are multiplied by a residual operation to obtain a multiplication result. The output of the second 3x3 convolutional layer is sequentially input into the third to eleventh 3x3 convolutional layers for processing. The output of the eleventh 3x3 convolutional layer and the multiplication result are jointly input into the twelfth 3x3 convolutional layer for processing and output as the output of the deep learning frame selection model SPNet.

[0035] The output of the eleventh 3x3 convolutional layer is a high-level feature, and the multiplication result is a low-level feature. For the feature information required by the descriptor, the low-level feature of the model is fused with the high-level information through the channel attention module, which strengthens the feature information and enables the Fast-NMS network to extract more accurate points and descriptors.

[0036] The channel attention module includes a global average pooling layer GAP, a depth separable convolutional layer PWC, and a fully connected layer Dense connected in sequence.

[0037] Step 2) input several action videos with action category labels into the FD-SPNet network for training to obtain the trained FD-SPNet network.

[0038] In step 2), several action videos with action category labels are input into the FD-SPNet network for training. First, each action video with an action category label is input into the coarse frame selection model FDNet in the FD-SPNet network to perform coarse selection of video key frames and output a sequence of coarsely selected video key frames. For each action video with an action category label, the following steps are performed:

[0039] 2.1) First, the vector values of all video frames in the action video are calculated to obtain the mean, maximum, and minimum of the vector values. The video frames with vector values equal to the maximum and minimum and the vector value closest to the mean are extracted as the maximum vector max frame, the minimum vector min frame, and the average vector avg frame. The maximum vector max frame, the minimum vector min frame, and the average vector avg frame are used as three fixed frames.

[0040] 2.2) Using the iterative frame difference method, the other video frames in the action video except the three fixed frames in step 2.1) are sequentially operated on the basis of the fixed frames to obtain a sequence of video frames.

[0041] 2.3) Using the sliding frame difference method, the other video frames in the action video except the three fixed frames in step 2.1) are sequentially operated on the basis of the fixed frames to obtain a sequence of video frames.

[0042] 2.4) The sequence of video frames obtained in step 2.2) and the sequence of video frames obtained in step 2.3) are operated to obtain a sequence of video frames for rough key frame selection.

[0043] In step 2.2), the iterative frame difference method is used to sequentially operate on the basis of the three fixed frames to obtain a sequence of video frames. Specifically, a number of video frames before the three fixed frames, a number of video frames between each two fixed frames, and a number of video frames after the three fixed frames in the action video are respectively grouped into an iterative frame difference set, i.e., four iterative frame difference sets. For each iterative frame difference set, frame difference operation is performed on every two adjacent video frames in the iterative frame difference set. When performing frame difference operation for the first time, if the frame difference between the first video frame and the second video frame is greater than a preset initial frame difference threshold, the sequence of the first video frame is retained; if the frame difference between the first video frame and the second video frame is less than or equal to the preset initial frame difference threshold, the sequence of the first video frame is discarded. Then, the same frame difference operation as that of the first video frame and the second video frame is continued from the second video frame and the next video frame thereof. In each loop of frame difference operation, the frame difference threshold in the last frame difference operation is reduced by a preset difference value to serve as the frame difference threshold in the current frame difference operation. The iteration of frame difference is stopped when the number of video frames in the sequence is greater than a preset number of frames, and the sequence of retained video frames is output.

[0044] When performing frame difference operation on the next iterative frame difference set, the last frame difference threshold of the previous iterative frame difference set is reduced by a preset difference value to serve as the first frame difference threshold of the next iterative frame difference set.

[0045] The output result formula of the iterative frame difference method is as follows:

[0046] z = IFD(x)

[0047] Where IFD(*) is an iterative frame difference function, x is the input video frame, and z is the frame sequence extracted by the iterative frame difference method.

[0048] In step 2.3), the sliding frame difference method is used to sequentially perform frame difference operation on the video frames other than the three fixed frames in step 2.1) based on the fixed frames. Specifically, a plurality of video frames before the three fixed frames, a plurality of video frames between every two fixed frames, and a plurality of video frames after the three fixed frames are grouped into sliding frame difference sets, i.e., four sliding frame difference sets. For each fixed frame and the two sliding frame difference sets before and after the fixed frame, first, frame difference operation is performed on each video frame in the sliding frame difference set before the fixed frame and the fixed frame. When the frame difference between the first video frame in the sliding frame difference set before the fixed frame and the fixed frame is greater than a preset initial frame difference threshold in the first frame difference operation, the sequence of the first video frame is retained. When the frame difference between the first video frame in the sliding frame difference set before the fixed frame and the fixed frame is less than or equal to the preset initial frame difference threshold, the sequence of the first video frame is discarded. Then, the same frame difference operation as that of the first video frame and the fixed frame is performed on the second video frame in the sliding frame difference set before the fixed frame and the fixed frame. In each loop of the frame difference operation, the frame difference threshold in the last frame difference operation is reduced by a preset difference value to serve as the frame difference threshold in the current frame difference operation. Frame difference operation is performed on each video frame in the sliding frame difference set after the fixed frame and the fixed frame. For each video frame in the sliding frame difference set after the fixed frame, when the frame difference between the video frame and the fixed frame is greater than the current frame difference threshold, the sequence of the video frame is retained. When the frame difference between the video frame and the fixed frame is less than or equal to the current frame difference threshold, the sequence of the video frame is discarded. In each loop of the frame difference operation, the frame difference threshold in the last frame difference operation is reduced by a preset difference value to serve as the frame difference threshold in the current frame difference operation. The sliding frame difference is stopped when the number of the video frames with the retained sequences reaches a preset frame number, and the sequence of the retained video frames is output.

[0049] In the frame difference operation of the next sliding frame difference set, the last frame difference threshold of the previous sliding frame difference set is reduced by a preset difference value to serve as the first frame difference threshold of the next sliding frame difference set.

[0050] The output result formula of the sliding frame difference method is as follows:

[0051] y = sFD(x)

[0052] wherein, SFD(*) is the sliding frame difference; x is the input video frame; and y is the frame sequence extracted by the sliding frame difference method.

[0053] In the specific implementation, the preset initial frame difference threshold is 0.5, the preset difference value is 0.1, and the preset frame number is one fourth of the total number of video frames.

[0054] The frame sequences extracted by the sliding frame difference method and the iterative frame difference method have a large number of same frame sequences, and the two groups of frame sequences are subjected to or operation to remove redundant frame sequences, and the output result is the finally obtained roughly selected key frame sequence, as follows:

[0055] output=y||z

[0056] Wherein, output is the finally obtained roughly selected key frame sequence; || is a logical or operation.

[0057] Step 3) input the to-be-tested action video into the trained FD-SPNet network, the trained FD-SPNet network outputs the feature information of a plurality of to-be-extracted video key frames in the to-be-tested action video, and the feature information of each to-be-extracted video key frame is input into the Fast-NMS network, and the Fast-NMS network outputs a plurality of final video key frames in the to-be-tested action video, thereby realizing extraction of the video key frame.

[0058] In step 3), the feature information of each to-be-extracted video key frame is input into the Fast-NMS network, and the Fast-NMS network outputs a plurality of final video key frames in the to-be-tested action video, specifically, the feature information of each to-be-extracted video key frame is input into the Fast-NMS network to extract the point and descriptor to obtain a 3xN matrix information of each to-be-extracted video key frame, the distance between the matrix information of each two adjacent video key frames is calculated, when the calculated distance is greater than the descriptor matching threshold, the former one of the two adjacent video key frames is extracted, when the calculated distance is less than or equal to the descriptor matching threshold, the video key frame is not extracted, and finally a plurality of extracted video key frames are output as the final video key frames.

[0059] The to-be-tested action video in the data set is input into the trained FD-SPNet network, the original video frame information is used as the input, the attention channel is subjected to GlobalAveragePooling2D to reduce the parameter quantity of fc and is attached with global information; by means of the point-by-point convolution and the point-by-point convolution, the correlation between the spatial dimension and the channel dimension is split, the extraction of the spatial information is completed, the activation function is sigmoid, the feature weight on each channel is generated, and the specific process is as follows:

[0060] F G =GAP(conv_1)

[0061] F D =DWC(F G )

[0062] Wherein, GAP(*) is a global average pooling function GlobalAveragePooling; DWC(*) is a channel convolution and pointwise convolution; conv_1 is the feature information output by the first 3*3 convolution layer; F D is the feature information output by the attention channel module.

[0063] The output of the second 3*3 convolution layer and the output of step 3-1) are taken as inputs and input into the MSCA feature fusion module; the feature information of the first 3*3 convolution layer and the MSCA output feature are multiplied by residual operation; the output information after multiplication is fused with the feature information output by the eleventh 3*3 convolution layer, as follows:

[0064] F M = MSCA(conv_2, F D )

[0065] output = Multiply(couv_1, F M )

[0066] Wherein, MSCA(*) is the MSCA feature fusion module; couv_1 is the feature information output by the first 3*3 convolution layer; conv_2 is the feature information output by the second 3*3 convolution layer; output is the final output of the trained FD-SPNet network.

[0067] The obtained feature information is input into the Fast-NMS network to extract points and descriptors, and a 3*N matrix is obtained; the distance between the matrix information of each two frames is calculated, the descriptor matching threshold is set, the distance difference calculated is greater than the threshold, and the frame is extracted, and finally the key frame of the deep learning frame selection network is obtained, as follows:

[0068]

[0069] Wherein, x k is the descriptor parameter of the previous frame; y k is the descriptor parameter of the next frame; k is the number of descriptors; dist is the matching threshold.

[0070] The method of the present application firstly reads a data set video action to obtain a video label; the video is input into a key frame selection network FDNet for rough selection, average vector frame (avg frame), minimum value vector frame (min frame) and maximum value vector frame (max frame) are obtained through vector value calculation of the video frame, and the three frames are taken as fixed frames; based on the fixed frames, the iterative frame difference and sliding frame difference method are used for key frame sequence selection, and the key frame sequence selected roughly is obtained through or operation; the sequence frame is extracted according to the video frame and input into a deep learning key frame selection network SPNet to extract feature information, the feature information obtained is input into a Fast-NMS network to extract points and descriptors, the distance of the descriptors is calculated and a distance threshold is set, the frames greater than the threshold are saved to obtain the final key frames, and the extraction of the video key frames is realized. In the case of "rough" selection, there is still a large amount of redundancy in the selected frames, and after "fine" selection, the final key frames are obtained. The method can express the video semantics completely with less key frames, and the running time loss compared with other methods can be ignored.

Claims

1.A method for extracting key frames from a video based on an FD-SPNet network, characterized in that: The method comprises the following steps: Step 1) establishing an FD-SPNet network; Step 2) inputting a plurality of action videos with action category labels into the FD-SPNet network for training to obtain a trained FD-SPNet network; Step 3) inputting a to-be-tested action video into the trained FD-SPNet network, the trained FD-SPNet network outputs feature information of a plurality of to-be-extracted video key frames in the to-be-tested action video, the feature information of each to-be-extracted video key frame is inputted into a Fast-NMS network, and the Fast-NMS network outputs a plurality of final video key frames in the to-be-tested action video, thereby realizing extraction of the video key frames; in the step 1), the established FD-SPNet network comprises a rough frame selection model FDNet and a deep learning frame selection model SPNet connected in sequence; after each action video with an action category label is inputted into the rough frame selection model FDNet in the FD-SPNet network, rough selection of the video key frames is performed, and a sequence of the roughly selected video key frames is outputted; the roughly selected video key frames are inputted into the deep learning frame selection model SPNet according to the sequence of the roughly selected video key frames, and the deep learning frame selection model SPNet outputs feature information of a plurality of to-be-extracted video key frames in each action video with an action category label; In the step 2), the plurality of action videos with action category labels are inputted into the FD-SPNet network for training, and first, each action video with an action category label is inputted into the rough frame selection model FDNet in the FD-SPNet network, rough selection of the video key frames is performed, and a sequence of the roughly selected video key frames is outputted; for each action video with an action category label, the following steps are specifically performed: Step 2.1) first, vector values of all video frames in the action video are calculated to obtain a mean value, a maximum value and a minimum value of the vector values, video frames with the maximum value and the minimum value of the vector values and the vector value closest to the mean value are extracted as a maximum value vector max frame, a minimum value vector min frame and an average vector avg frame, and the maximum value vector max frame, the minimum value vector min frame and the average vector avg frame are taken as three fixed frames; Step 2.2) an iterative frame difference method is adopted to sequentially perform frame difference operations on other video frames in the action video except the three fixed frames in the step 2.1) based on the fixed frames to obtain a plurality of sequences of video frames; Step 2.3) a sliding frame difference method is adopted to sequentially perform frame difference operations on other video frames in the action video except the three fixed frames in the step 2.1) based on the fixed frames to obtain a plurality of sequences of video frames; Step 2.4) the sequences of video frames obtained in the step 2.2) and the sequences of video frames obtained in the step 2.3) are subjected to an or operation, and finally a sequence of key frame rough frame selection video frames is obtained. 2.The video key frame extraction method based on FD-SPNet network according to claim 1, characterized in that: In step 2.2), the other video frames in the action video except the three fixed frames in step 2.1) are sequentially subjected to frame difference operation based on the three fixed frames by using the iterative frame difference method. Specifically, a number of video frames before the three fixed frames, a number of video frames between every two fixed frames, and a number of video frames after the three fixed frames are respectively grouped into iterative frame difference sets, i.e., four iterative frame difference sets. For each iterative frame difference set, frame difference operation is performed on every two adjacent video frames in the iterative frame difference set. When performing frame difference operation for the first time, when the frame difference between the first video frame and the second video frame is greater than a preset initial frame difference threshold, the sequence of the first video frame is retained; when the frame difference between the first video frame and the second video frame is less than or equal to the preset initial frame difference threshold, the sequence of the first video frame is discarded. Then, the same frame difference operation as that of the first video frame and the second video frame is continuously performed from the second video frame and the next video frame thereof. In each loop of frame difference operation, the frame difference threshold in the last frame difference operation is reduced by a preset difference value to serve as the frame difference threshold in the current frame difference operation. The iteration of frame difference is stopped and the sequence of the retained video frames is outputted when the number of the sequence of the video frames is greater than a preset number of frames. When performing frame difference operation on the next iterative frame difference set, the last frame difference threshold of the previous iterative frame difference set is reduced by a preset difference value to serve as the first frame difference threshold of the next iterative frame difference set. 3.The method of claim 1, wherein the method is based on a FD-SPNet network. In step 2.3), the sliding frame difference method is used to sequentially perform frame difference operation on the video frames other than the three fixed frames in step 2.1) based on the fixed frames, specifically, a plurality of video frames before the three fixed frames, a plurality of video frames between every two fixed frames, and a plurality of video frames after the three fixed frames are respectively grouped into sliding frame difference sets, that is, four sliding frame difference sets, for each fixed frame and the two sliding frame difference sets before and after the fixed frame, first, each video frame in the sliding frame difference set before the fixed frame is sequentially subjected to frame difference operation with the fixed frame, when the frame difference between the first video frame in the sliding frame difference set before the fixed frame and the fixed frame is greater than a preset initial frame difference threshold in the first frame difference operation, the sequence of the first video frame in the sliding frame difference set before the fixed frame is retained, when the frame difference between the first video frame in the sliding frame difference set before the fixed frame and the fixed frame is less than or equal to the preset initial frame difference threshold, the sequence of the first video frame in the sliding frame difference set before the fixed frame is discarded, then the same frame difference operation as that of the first video frame and the fixed frame is continued to be performed on the second video frame in the sliding frame difference set before the fixed frame and the fixed frame, and the frame difference threshold in the last frame difference operation is reduced by a preset difference value to serve as the frame difference threshold in the current frame difference operation; each video frame in the sliding frame difference set after the fixed frame is subjected to frame difference operation with the fixed frame, for each video frame in the sliding frame difference set after the fixed frame, when the frame difference between the video frame and the fixed frame is greater than the current frame difference threshold, the sequence of the video frame is retained, when the frame difference between the video frame and the fixed frame is less than or equal to the current frame difference threshold, the sequence of the video frame is discarded, and the frame difference threshold in the last frame difference operation is reduced by a preset difference value to serve as the frame difference threshold in the current frame difference operation; the frame difference operation is stopped until the number of the video frames with the sequence is greater than a preset number of frames, and the sequence of the retained video frames is output. In the frame difference operation of the next sliding frame difference set, the last frame difference threshold of the previous sliding frame difference set is reduced by a preset difference value to serve as the first frame difference threshold of the next sliding frame difference set. 4.The method of claim 1, wherein the method is based on a FD-SPNet network. The deep learning frame selection model SPNet includes 12 3×3 convolutional layers, a channel attention module, and a feature fusion module MSCA, the input of the deep learning frame selection model SPNet is input into the channel attention module and the second 3×3 convolutional layer after being input into the first 3×3 convolutional layer, the output of the channel attention module and the output of the second 3×3 convolutional layer are jointly input into the feature fusion module MSCA for processing, the output of the first 3×3 convolutional layer and the output of the feature fusion module MSCA are multiplied to obtain a multiplication result, the output of the second 3×3 convolutional layer is sequentially input into the third to eleventh 3×3 convolutional layers for processing, the output of the eleventh 3×3 convolutional layer and the multiplication result are jointly input into the twelfth 3×3 convolutional layer for processing and output as the output of the deep learning frame selection model SPNet. 5.The method of claim 4, wherein the method is based on a FD-SPNet network. The channel attention module comprises, in sequence, a global average pooling layer GAP, a depth separable convolution layer PWC and a fully connected layer Dense. 6.The method of claim 1, wherein the method is based on a FD-SPNet network. In the step 3), the feature information of each video key frame to be extracted is input into the Fast-NMS network, and the Fast-NMS network outputs a plurality of final video key frames in the action video to be detected, specifically, the feature information of each video key frame to be extracted is input into the Fast-NMS network to obtain matrix information of each video key frame to be extracted by extraction points and descriptors, distance calculation is performed on the matrix information between two adjacent video key frames, when the calculated distance is greater than a descriptor matching threshold, the previous frame of the two adjacent video key frames is extracted, when the calculated distance is less than or equal to the descriptor matching threshold, the video key frame is not extracted, and finally a plurality of extracted video key frames are output as the final video key frames.

Images