Video processing method and device and storage medium

Abstract

The invention relates to a video processing method and device and a storage medium. The video processing method comprises the steps that image features of N frames of images in a video sequence are extracted, N is a positive integer, and the N frames of images comprise hand posture information of a target object; based on the image features, n frames of key frame images are sequentially selected from the N frames of images according to a time sequence, N is greater than or equal to n, the time sequence of the n frames of key frame images is consistent with that of the N frames of images, n is a preset number value of image frames contained in the video abstract to be generated, and the image features of the key frame images are target hand action features; and generating a video abstract based on the key frame image. According to the invention, the key frames in the video can be captured more accurately, and redundant frames in the video abstract can be reduced.

Description

Technical Field

[0001] The present disclosure relates to the field of video processing technology, and in particular to a video processing method, device and storage medium. Background Art

[0002] In the related art, a video is composed of a sequence of pictures, and classifying a video requires a large number of picture frames to be calculated to determine the video content. However, the relevance of a particular video to a user is not always obvious to the user, especially when the video duration is long. Summary of the invention

[0003] In order to overcome the problems existing in the related art, the present disclosure provides a video processing method, device and storage medium.

[0004] According to a first aspect of an embodiment of the present disclosure, a video processing method is provided, comprising: extracting image features of N frames of images in a video sequence, wherein N is a positive integer, and the N frames of images include hand gesture information of a target object; based on the image features, sequentially selecting n frames of key frame images from the N frames of images in chronological order, wherein N is greater than or equal to n, wherein the n frames of key frame images are consistent with the chronological order of the N frames of images, n is a preset value of the number of image frames included in a video summary to be generated, and the image features of the key frame images are target hand movement features; and generating a video summary based on the key frame images.

[0005] In one embodiment, based on the image features, n key frame images are selected sequentially in chronological order from the N frames of images, including: based on the difference between N and n, and the value of n, the size of the dynamic temporal pooling area of ​​the image feature sequence of the N frames of images is adjusted to obtain a subsequence whose cumulative cost is less than a preset threshold, the cumulative cost is used to represent the alignment path of the feature sequence and the subsequence, the dynamic temporal pooling is used to segment the image feature sequence in chronological order to extract the subsequence; and the frame image in the subsequence is used as the key frame image.

[0006] In one implementation, adjusting the size of the dynamic temporal pooling region of the image feature sequence of the N frames of images based on the difference between the N and the n, and the value of the n, to obtain a subsequence whose cumulative cost is less than a preset threshold, includes:

[0007] Determine the alignment cost of timing alignment between the i-th frame and the j-th frame in the image feature sequence frame by frame, the number of frames corresponding to the i-th frame and the j-th frame is less than or equal to the number of image frames; determine the minimum cumulative cost among the cumulative cost of the frame before the i-th frame and the j-th frame, the cumulative cost of the i-th frame and the frame before the j-th frame, and the cumulative cost of the frame before the i-th frame and the frame before the j-th frame after frame extraction; determine the sum of the alignment cost and the minimum cumulative cost as the cumulative cost of the current frame; and determine a subsequence whose cumulative cost is less than a preset threshold based on the cumulative cost.

[0008] In one embodiment, the method further includes: generating an image feature sequence with a length of N based on the key frame image, and the image feature sequence is used for sign language recognition.

[0009] In one embodiment, the key frame image is extracted using a spatiotemporal convolutional network; generating an image feature sequence with a length of N based on the key frame image includes: using a convolutional neural network that is correlated with the spatiotemporal convolutional neural network to restore image features of the key frame image to obtain an image feature sequence with a length of N.

[0010] In one embodiment, the method further includes: upsampling the image feature sequence to obtain an upsampled sequence; the upsampled sequence is used to constrain a first model, the first model is used to train a second model using a distillation method, the second model is used to extract image features of N frames of images in the video sequence, and based on the image features, n key frame images are selected in sequence in chronological order from the N frames of images to generate a video summary.

[0011] In one implementation, the second model is trained based on the first model in the following manner:

[0012] Input the video sequence training samples into the first model to obtain a first image feature sequence; input the first image feature sequence into the second model, and use the second model to downsample the first image feature sequence to obtain a key frame image; use upsampling to restore the image features of the key frame image to obtain a second image feature sequence with the same length as the first image feature sequence; determine the distillation loss based on the first image feature sequence and the second feature sequence, use the distillation loss to constrain the first model and the second model, and train to obtain the second model.

[0013] According to a second aspect of an embodiment of the present disclosure, a video processing device is provided, the device comprising: an extraction unit, configured to extract image features of N frames of images in a video sequence, wherein N is a positive integer, and the N frames of images include hand gesture information of a target object; a processing unit, configured to sequentially select n frames of key frame images from the N frames of images in chronological order based on the image features, wherein N is greater than or equal to n, wherein the n frames of key frame images are consistent with the chronological order of the N frames of images, n is a preset value of the number of image frames included in a video summary to be generated, and the image features of the key frame images are target hand movement features; and a generation unit, configured to generate a video summary based on the key frame images.

[0014] In one embodiment, the processing unit selects n key frame images in sequence in chronological order from the N frames of images based on the image features in the following manner: based on the difference between N and n, and the value of n, the size of the dynamic temporal pooling area of ​​the image feature sequence of the N frames of images is adjusted to obtain a subsequence whose cumulative cost is less than a preset threshold, the cumulative cost is used to represent the alignment path of the feature sequence and the subsequence, and the dynamic temporal pooling is used to segment the image feature sequence in chronological order to extract the subsequence; and the frame image in the subsequence is used as the key frame image.

[0015] In one embodiment, the processing unit further adjusts the size of the dynamic temporal pooling area of ​​the image feature sequence of the N frames of images based on the difference between the N and the n, and the value of the n, in the following manner to obtain a subsequence whose cumulative cost is less than a preset threshold: determining the alignment cost for temporal alignment between the i-th frame and the j-th frame frame by frame in the image feature sequence, the number of frames corresponding to the i-th frame and the j-th frame being less than or equal to the number of image frames; determining the minimum cumulative cost among the cumulative cost of the frame before the i-th frame and the j-th frame, the cumulative cost of the i-th frame and the frame before the j-th frame, the cumulative cost of the frame before the i-th frame and the frame before the j-th frame, and the cumulative cost of the frame before the i-th frame and the frame before the j-th frame after frame extraction; determining the sum of the alignment cost and the minimum cumulative cost as the cumulative cost of the current frame; and determining a subsequence whose cumulative cost is less than a preset threshold based on the cumulative cost.

[0016] In one implementation, the generating unit is further used to generate an image feature sequence with a length of N based on the key frame image, and the image feature sequence is used for sign language recognition.

[0017] In one embodiment, the key frame image is extracted using a spatiotemporal convolutional network;

[0018] The generating unit generates an image feature sequence having a length of N based on the key frame image in the following manner: a convolutional neural network having correlation with the spatiotemporal convolutional neural network is used to restore image features of the key frame image to obtain an image feature sequence having a length of N.

[0019] In one embodiment, the processing unit is also used to: upsample the image feature sequence to obtain an upsampled sequence; the upsampled sequence is used to constrain the first model, the first model is used to train the second model using a distillation method, the second model is used to extract image features of N frames of images in a sign language video sequence, and based on the image features, n key frame images are selected in sequence in chronological order from the N frames of images to generate a sign language video summary.

[0020] In one embodiment, the processing unit is further configured to obtain the second model based on the first model training in the following manner:

[0021] Input the video sequence training sample into the first model to obtain the first image feature sequence; input the first image feature sequence into the prediction model of the second model, and use the prediction model of the second model to downsample the first image feature sequence to obtain the key frame image; use the upsampling method to restore the image features of the key frame image to obtain the second image feature sequence with the same length as the first image feature sequence; determine the distillation loss based on the first image feature sequence and the second image feature sequence, use the distillation loss to constrain the first model, and train the prediction model of the second model to obtain the second model.

[0022] According to a third aspect of an embodiment of the present disclosure, a video processing device is provided, comprising: a processor; a memory for storing processor executable instructions; wherein the processor is configured to: execute the method described in the first aspect or any one of the first aspect.

[0023] According to a fourth aspect of an embodiment of the present disclosure, a storage medium is provided, in which instructions are stored. When the instructions in the storage medium are executed by a processor of a terminal, the terminal is enabled to execute the method described in the first aspect or any one of the first aspects.

[0024] The technical solution provided by the embodiments of the present disclosure may include the following beneficial effects: by extracting the image features of N frames of images including the hand gesture information of the target object in the video sequence frame by frame, an image feature sequence can be obtained. According to the image features, n frames of key frame images are selected in sequence in chronological order from the N frames of images, N is greater than or equal to n, wherein the n frames of key frame images are consistent with the chronological order of the N frames of images, n is a preset value of the number of image frames included in the sign language video summary to be generated, and the image features of the key frame images are the target hand movement features. Based on the key frame images, a video summary is generated. In this way, the key frames in the video can be captured more accurately, and the redundant frames in the video summary can be reduced, thereby improving the accuracy of the sign language video summary generation and reducing the computational overhead of the recognition model.

[0025] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0027] Figure 1 The specific network structure used by VAC shown in an exemplary embodiment of the present disclosure is shown.

[0028] Figure 2 A schematic diagram of implementing time series classification based on dynamic time series pooling according to an exemplary embodiment of the present disclosure is shown.

[0029] Figure 3 A scene graph showing the application of the video summarization technology shown in an exemplary embodiment of the present disclosure to extract a video summary of a sign language video is shown.

[0030] Figure 4 The figure is a flowchart of a method for generating a video summary according to an exemplary embodiment.

[0031] Figure 5 A flowchart of selecting n key frame images in sequence in chronological order from N frames of images based on image features is shown in an exemplary embodiment of the present disclosure.

[0032] Figure 6 A flowchart is shown of an exemplary embodiment of the present disclosure, which adjusts the size of the dynamic temporal pooling area of ​​the image feature sequence of N frames of images based on the difference between N and n and the value of n to obtain a subsequence whose cumulative cost is less than a preset threshold.

[0033] Figure 7A flowchart of determining a second model according to an exemplary embodiment of the present disclosure is shown.

[0034] Figure 8 A schematic diagram of accelerating a sign language video recognition process based on using adaptive dynamic temporal pooling is shown according to an exemplary embodiment of the present disclosure.

[0035] Fig. 9 A schematic diagram showing comparison results of sign language video summaries generated using adaptive dynamic temporal pooling and using uniform sampling according to an exemplary embodiment of the present disclosure is shown.

[0036] Fig.10 A schematic diagram of comparison results between using VAC as a baseline model and using ADTP is shown in an exemplary embodiment of the present disclosure.

[0037] Fig.11 The figure is a block diagram of a video processing device according to an exemplary embodiment.

[0038] Fig.12 The invention is a block diagram of a device for video processing according to an exemplary embodiment. DETAILED DESCRIPTION

[0039] Here, exemplary embodiments will be described in detail, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present disclosure.

[0040] The video summary generation method provided in the embodiments of the present disclosure is applied to the scenario of generating a video summary, for example, it is applied to the scenario of generating a sign language video summary.

[0041] At present, continuous sign language recognition aims to sequentially identify the sign language words appearing in sign language videos through computer vision technology, providing a convenient communication tool for the deaf and dumb and the healthy. The method recently adopted for continuous sign language recognition is Visual Alignment Constraint (VAC). This method uses a two-dimensional convolutional neural network (CNN) to extract visual features frame by frame. Then, 1D-CNN and bidirectional long short-term memory network (BiLSTM) are used to extract local short-term and long-term features in sequence. Among them, in order to promote the optimization of the model, this method introduces an auxiliary classifier after 1D-CNN, and uses connectionist temporal classification (CTC) to provide supervision signals for the main classifier and auxiliary classifier. However, since the main goal of this method is continuous sign language recognition and a frame-by-frame feature extractor is used, the structural information of the sign language itself is not considered, which brings a large computational overhead.

[0042] Figure 1 FIG. 4 shows a network structure used by a VAC according to an exemplary embodiment of the present disclosure. Figure 1 As shown in the figure, the sign language video is first processed by 2D-CNN to obtain image features, and then by 1D-CNN and BiLSTM to obtain image feature sequences. p The final prediction result is obtained. In order to better optimize the model, an auxiliary classifier F is additionally introduced in the training phase. a , both of which are supervised by CTC. For the prediction result w, the loss function is: L CTC (w, X) = -lnp(wX; θ), where X is the input video sequence, w is the corresponding label sequence, and θ is the parameter of the model.

[0043] Furthermore, in order to achieve more accurate time series classification, dynamic temporal pooling (DTP) is used to segment the time series data and summarize the features of each segment to achieve more accurate time series classification. However, in order to classify time series, it is necessary to train a template feature for each segment of each category in combination with high-level semantic features. At the same time, the number of segments needs to be set in advance, and the number of segment template features is fixed, which cannot be applied to video sequences of different lengths.

[0044] Figure 2 FIG. 1 shows a schematic diagram of implementing time series classification based on dynamic time series pooling according to an exemplary embodiment of the present disclosure. Figure 2As shown, the sequence feature H = [h1, ..., h T ]∈R K×T , and designed a learnable prototype sequence feature P = [p1, ..., p L ]∈R K×L , through Figure 2 The dynamic time warping (DTW) process shown in Figure aligns H and P based on cosine similarity, thereby dividing H into L segments. The features of segment l are calculated by pooling: The final classifier uses different weights for each segment and predicts the probability of belonging to class c as:

[0045] In view of this, the present disclosure provides a method for generating a video summary. In this method, according to an image feature sequence extracted from a video, a preset number of key frame images are selected based on a temporal downsampling method to generate a video summary, which can improve the computational efficiency of a video recognition algorithm. According to the temporal downsampling, the number of key frames can be dynamically selected according to actual needs and video length to generate a more accurate video summary.

[0046] For example, Figure 3 FIG. 1 shows a scene diagram of a video summarization technique shown in an exemplary embodiment of the present disclosure being applied to extracting a video summary of a sign language video. Figure 3 As shown, the meaning of the sign language video is that many people went to the restaurant today. The original state sign language video is shown in 3a. Terminal 3e performs feature extraction to form an image feature sequence, as shown in 3b. After the feature sequence is temporally downsampled, key frames are extracted, as shown in 3c. A video summary is generated through the terminal, as shown in 3d. Through the sign language video summary technology, video information can be converted into the form of an image sequence to help sign language users quickly browse the content of the sign language video. At the same time, for some important content, you can jump to the corresponding video clip by clicking the key frame, which is convenient for the sign language user to confirm the relevant information in the sign language video again.

[0047] The embodiment of the present disclosure provides a video processing method, through which a method suitable for generating a sign language video summary is provided. In the method, a video summary of a suitable length is adaptively generated according to the video content in a video sequence, and the video summary can accurately reflect the hand movement characteristics.

[0048] Figure 4 is a flowchart of a method for generating a video summary according to an exemplary embodiment. Figure 4 As shown, the video summary production method includes the following steps.

[0049] In step S11, image features of N frames of images in a video sequence are extracted.

[0050] In one example of the present disclosure, N is a positive integer. N frames of images include hand gesture information of the target object.

[0051] The video involved in the embodiments of the present disclosure may be a video including sign language movements.

[0052] In the disclosed embodiment, image features of each frame of N frames of images including hand gesture information in a sign language video sequence are extracted frame by frame to obtain an image feature sequence including hand gesture information of the target object.

[0053] In step S12, based on the image features, n key frame images are selected from the N frames of images in chronological order.

[0054] In the disclosed embodiment, based on the image features, n key frame images are selected from N frames of images in chronological order. It can be understood that the key frame images are selected from the image feature sequence by using a time-series downsampling method. The number of key frame images is a preset number.

[0055] In the disclosed embodiment, a temporal downsampling method is adopted, and the number of key frames can be dynamically selected according to actual needs and video length.

[0056] In the disclosed embodiment, time series downsampling is to downsample an image feature sequence composed of N image features to obtain n key frame images arranged in chronological order, where N is greater than or equal to n, and the order of the n key frame images is consistent with the order of the N frame images.

[0057] In the disclosed embodiment, the key frame image includes an image corresponding to the target hand movement feature in the sign language video.

[0058] For example, in a sign language video, a key frame image can be understood as an image with hand movements corresponding to the main content to be expressed in the video.

[0059] For example, the word "full" corresponds to a sign language action. When the sign language speaker performs the action corresponding to "full", there will be redundant actions (such as raising the arm). In this case, the key frame image can be understood as the image corresponding to "full" as the key frame image, and the "raising arm action" as the redundant image.

[0060] In the disclosed embodiment, the number of key frame images is the number of pre-set images corresponding to the target hand movement.

[0061] In step S13, a video summary is generated based on the key frame images.

[0062] In the disclosed embodiment, a video summary is generated based on key frame images selected by using a time-series downsampling method.

[0063] In the disclosed embodiment, according to the video content, the time-series downsampling method is adopted to dynamically select the number of key frames, and then, a video summary of a suitable length is adaptively generated, and the summary video can accurately reflect the content of the video.

[0064] A sign language video processing method provided by the embodiment of the present disclosure can obtain an image feature sequence based on the image features of N frames of images. According to the image features, n frames of key frame images are selected in sequence in chronological order from the N frames of images, where N is greater than or equal to n, wherein the n frames of key frame images are in the same chronological order as the N frames of images, and n is a preset value of the number of image frames included in the sign language video summary to be generated, and the image features of the key frame images are target sign language action features. And based on the key frame images, a video summary is generated. Through this method, a video summary of a suitable length can be accurately and adaptively generated, and the video recognition algorithm can be accelerated. Furthermore, redundant frames in the video summary can be reduced.

[0065] Embodiments of the present disclosure The following describes a method for generating a video summary according to an embodiment of the present disclosure.

[0066] Figure 5 FIG. 1 shows a flowchart of selecting n key frame images in chronological order from N frames of images based on image features according to an exemplary embodiment of the present disclosure. Figure 5 As shown, the following steps are included.

[0067] In step S21, based on the difference between N and n, and the value of n, the size of the dynamic temporal pooling area of ​​the image feature sequence of N frames of images is adjusted to obtain a subsequence whose cumulative cost is less than a preset threshold. The cumulative cost is used to represent the alignment path of the feature sequence and the subsequence.

[0068] In the disclosed embodiment, the number of time-series down-sampling frames is obtained by subtracting the total number of image frames in the image feature sequence from the number of key frame images.

[0069] In the disclosed embodiment, a subsequence whose cumulative cost is less than a preset threshold is obtained based on adjusting the size of the dynamic temporal pooling region of the image feature sequence of N frames of images.

[0070] It can be understood that in the embodiment of the present disclosure, based on the number of temporal downsampling frames and the number of key frame images, the size of the dynamic temporal pooling area of ​​the image feature sequence is automatically adjusted to obtain a subsequence whose cumulative cost is less than a preset threshold.

[0071] In an example of the present disclosure, dynamic temporal pooling is used to segment an image feature sequence and extract subsequences.

[0072] In an example of the present disclosure, the cumulative cost is used to represent the alignment path of the feature sequence and the subsequence.

[0073] In the disclosed embodiment, the cumulative cost can be represented by ADTW[i, j, k], where ADTW[i, j, k] represents aligning the feature sequence from 1 to i and the feature sequence from 1 to jk, and iterating the feature sequence from 1 to jk with the minimum cumulative cost of the previous end.

[0074] In step S22, the frame images in the subsequence are used as key frame images.

[0075] In the embodiment of the present disclosure, the frame corresponding to the subsequence whose cumulative cost is less than a preset threshold is a key frame image, wherein the preset threshold is a preset value.

[0076] In the embodiment of the present disclosure, the frames corresponding to the subsequences whose cumulative cost is less than the preset threshold are key frames. It can be understood that the sequence corresponding to the shortest path is selected as the subsequence, and the frames corresponding to the subsequences are key frames.

[0077] In the disclosed embodiment, by adopting the method of temporal downsampling, based on the number of temporal downsampling frames and the number of key frame images, the image feature sequence is adaptively and dynamically temporally pooled to obtain a subsequence whose cumulative cost is less than a preset threshold, and the frame image corresponding to the subsequence is the key frame image. In this way, it can be used at any layer of the model without targeted training, and the features of the sign language recognition network can be directly used to generate video summaries.

[0078] Figure 6 A flowchart of an exemplary embodiment of the present disclosure is shown, which shows a method of adjusting the size of the dynamic temporal pooling region of the image feature sequence of N frames of images based on the difference between N and n and the value of n, to obtain a subsequence whose cumulative cost is less than a preset threshold. Figure 6 As shown, the following steps are included.

[0079] In step S31, the alignment cost for temporal alignment between the i-th frame and the j-th frame is determined frame by frame in the image feature sequence.

[0080] In the disclosed embodiment, the alignment cost between the i-th frame and the j-th frame is represented by D[i, j], where the number of frames corresponding to the i-th frame and the j-th frame is less than or equal to the total number of image frames corresponding to the video.

[0081] In step S32, the minimum cumulative cost is determined.

[0082] In the disclosed embodiment, determining the minimum cumulative cost is to determine the minimum cumulative cost among the cumulative costs of the frame before the i-th frame and the j-th frame, the cumulative costs of the i-th frame and the frame before the j-th frame, the cumulative costs of the frame before the i-th frame and the frame before the j-th frame, and the cumulative costs of the frame before the i-th frame and the frame before the j-th frame after frame extraction.

[0083] In an example of the present disclosure, the minimum cumulative cost is determined using the following formula:

[0084]

[0085] Where Δk is the number of frames, i.e., redundant frames, and η is the maximum continuous frame loss interval of the sampling.

[0086] In step S33, the sum of the alignment cost and the minimum cumulative cost is determined as the cumulative cost of the current frame.

[0087] In the embodiment of the present disclosure, the cumulative cost of the current frame may be determined using the following formula:

[0088]

[0089] In step S34, based on the cumulative cost, a subsequence whose cumulative cost is less than a preset threshold is determined.

[0090] In the disclosed embodiment, an iterative algorithm is used to determine a subsequence corresponding to a minimum cumulative cost based on each calculated cumulative cost.

[0091] In the disclosed embodiment, by performing adaptive dynamic temporal pooling on the image feature sequence, a subsequence whose cumulative cost is less than a preset threshold is obtained. The number of key frames can be dynamically selected according to actual needs and video length, thereby generating a more accurate video summary.

[0092] Among them, the above-mentioned temporal downsampling implementation process of "selecting n frames of key frame images in N frames of images in chronological order based on image features" involved in the embodiment of the present disclosure can be understood as an adaptive dynamic temporal pooling (ADTP) process.

[0093] In another embodiment of the present disclosure, in order to further perform sign language recognition, an image feature sequence with a length of N may be generated based on the key frame image. The generated image feature sequence is used for sign language recognition.

[0094] In the disclosed embodiment, based on the key frame image, an image feature sequence with a length of N is generated. The generated image feature sequence with a length of N can be understood as a sign language video, and the sign language video includes an image feature sequence with a length of N.

[0095] The video processing method provided by the embodiments of the present disclosure may be implemented based on a pre-trained model.

[0096] In one implementation, in the disclosed embodiment, the model for performing video processing can be trained in two stages. In the first stage, a general continuous sign language recognition model (such as [4]) is trained; in the second stage, a small amount of spatiotemporal convolution is introduced in the frame-by-frame feature extraction process and temporal downsampling is performed to reduce the amount of computation in the feature extraction stage. In order to reduce the performance loss caused by temporal downsampling, the original feature sequence is restored based on the similarity of shallow features at the end of the visual feature extractor, and the model trained in the first stage is used to provide additional sequence supervision signals.

[0097] In the embodiment of the present disclosure, for the convenience of description, the model trained in the first stage is referred to as the first model, and the model trained in the second stage is referred to as the second model. In the embodiment of the present disclosure, the second model can be obtained by training based on the first model using knowledge distillation.

[0098] The first model is used to train the second model by distillation. The second model is used to extract image features of N frames in the sign language video sequence, and based on the image features, select n key frame images in the N frames in chronological order to generate a sign language video summary.

[0099] Among them, the first model can be understood as a teacher model, and the second model can be understood as a student model. The student model can be trained by using the knowledge distillation method of the teacher model and the student model.

[0100] In the disclosed embodiment, when the trained second model is used to extract key frame images, it is a scheme for extracting key frame images based on the ADTP layer, which can be understood as downsampling the timing information. This method will cause irreversible information loss, so the spatiotemporal convolutional neural network is used in the first layer of convolution of the previous convolution module using the ADTP layer. The above-mentioned process of extracting key frames in the disclosed embodiment is obtained by extracting frame by frame based on the spatiotemporal convolutional neural network. Among them, a small amount of spatiotemporal convolution is introduced in the frame-by-frame extraction process, thereby reducing the amount of calculation in the feature extraction stage.

[0101] In the embodiment of the present disclosure, in order to guide the first model to train the second model, more comprehensive sequence information can be captured. After extracting the frame-by-frame sequence features after downsampling, the feature sequence of the original length is restored through the correlation of shallow features. The feature sequence of the original length is restored through the correlation of shallow features. The image features of the key frame image can be restored by using a convolutional neural network that is correlated with the spatiotemporal convolutional neural network to obtain an image feature sequence of length N. The convolutional neural network that is correlated with the spatiotemporal convolutional neural network can be, for example, Figure 8 In the figure, there are three layers of convolutional neural networks: C3, C4 and C5.

[0102] In the embodiment of the present disclosure, the image feature sequence is restored to its original length by upsampling, that is, the image feature sequence including N frames of images. By determining the loss function of the restored image feature sequence and the original image feature sequence, the second model training process can be guided and constraints can be enforced to train a more accurate second model.

[0103] Figure 7 FIG. 4 shows a flow chart of determining the second model according to an exemplary embodiment of the present disclosure. Figure 7 As shown, the following steps are included.

[0104] In step S41, the video sequence training sample is input into the first model to obtain a first image feature sequence.

[0105] In the disclosed embodiment, the video for which a video summary needs to be extracted is subjected to a two-dimensional convolutional neural network and a one-dimensional convolutional neural network for feature extraction to obtain an image feature sequence.

[0106] In an example of the present disclosure, the first convolution layer and the second convolution layer are used to extract image features.

[0107] In an example of the present disclosure, the first convolution layer and the second convolution layer are convolution layers included in the second model.

[0108] In step S42, the first image feature sequence is input into the prediction model of the second model, and the prediction model of the second model downsamples the first image feature sequence to obtain a key frame image.

[0109] In the disclosed embodiment, the image feature sequence is temporally downsampled through an adaptive dynamic temporal pooling layer to obtain a key frame image.

[0110] In an example of the present disclosure, the adaptive dynamic timing pooling layer is the pooling layer included in the second model.

[0111] In an example of the present disclosure, the adaptive dynamic temporal pooling layer is used to perform adaptive dynamic temporal pooling on the image feature sequence.

[0112] In an example of the present disclosure, the adaptive dynamic pooling layer can be understood as an automatic dynamic adjustment pooling layer.

[0113] In step S43, an upsampling method is adopted to restore the image features of the key frame image, and a second image feature sequence having the same length as the first image feature sequence is obtained.

[0114] In the disclosed embodiment, after extracting the frame-by-frame feature sequence after time-series downsampling, the image feature sequence, that is, the feature sequence of the original length, is restored by upsampling through the correlation of the features, which can guide the network to capture more comprehensive sequence information.

[0115] In step S44, a distillation loss is determined based on the first image feature sequence and the second feature sequence, the distillation loss is used to constrain the first model, and the prediction model of the second model is trained to obtain the second model.

[0116] In the disclosed embodiment, the loss function is determined based on the second image feature sequence output by the second model and the first image feature sequence output by the first model, which can be understood as determining the distillation loss between the first model and the second model. That is, the constraint information is determined by distillation. In the disclosed embodiment, the second model network is trained based on the distillation loss constraint.

[0117] In an example of the present disclosure, it can be understood that during the training of the second model, the constraint loss can also be determined based on the prediction results of the auxiliary classifier and the main classifier. That is, the constraint loss involved in the training of the second model in the embodiment of the present disclosure includes the loss of the main classifier, the loss of the auxiliary classifier, and the distillation loss.

[0118] In the disclosed embodiment, the image feature sequence is temporally downsampled through an adaptive dynamic temporal pooling layer to obtain a key frame image, which can be used in any layer of the model without targeted training, and can directly use the features of the sign language recognition network to generate a video summary. After extracting the frame-by-frame feature sequence after temporal downsampling, the image feature sequence is restored, which can guide the network to capture more comprehensive sequence information.

[0119] The present disclosure below uses the generation of a sign language video summary and the recognition of a sign language video as examples to schematically illustrate the training and use process of the first model and the second model involved above.

[0120] Figure 8 FIG. 1 shows a schematic diagram of an exemplary embodiment of the present disclosure showing a process of accelerating sign language video recognition based on using adaptive dynamic temporal pooling. Figure 8 As shown in Figure 2, when the ADTP layer is used to generate video summaries. Given a video sequence X∈R T ×H×W×3 , by extracting visual features frame by frame, if based on the output feature v of the second module (2) To generate video summary, the corresponding video frame subscript is: τ = ADTP (v (2) , TK).

[0121] like Figure 8As shown in , when the ADTP layer is used for continuous sign language recognition acceleration. After the second module, the temporal downsampling is performed, and the feature correlation of this layer is used to restore the feature sequence of the original length. The whole process is as follows Figure 5 As shown. The input is a sign language video X∈R T×H×W×3 After two layers of feature extraction, the sequence feature v (2) Perform global pooling to obtain , and use the ADTP layer to get the corresponding downsampling subscript Based on the subscript τ, the original feature sequence v (2) Downsample to get v′ (2) , after passing through the remaining CNN modules, we get the feature sequence v′ (5) , and use the correlation between shallow features to restore the original feature sequence: Where θ(x) = W θ (x) and φ(x) = W φ (x) represents two linear mapping layers, which are used to enhance the expressiveness of the model. γ is a hyperparameter that controls the degree of weighted sparsity.

[0122] In one example of the present disclosure, in order to better capture the timing information, the feature sequence obtained by the first stage training is used. By distillation, a distillation loss constraint is provided for the upsampling process:

[0123]

[0124] Where, CE(p,q)=-E p [log q] calculates the cross entropy of probability distributions p and q, while f and g are composed of linear mapping layers and softmax functions to map features to category probability distributions.

[0125] Based on the above, the constraints involved in the training phase are: Among them, v and z are the prediction results of the auxiliary classifier and the main classifier respectively. CTC (v,l) is the loss of the auxiliary classifier. L CTC (z,l) is the loss of the main classifier. is the distillation loss constraint.

[0126] The concepts of the main classifier, auxiliary classifier, and distillation loss can be found in the description of the related art, and will not be described in the embodiments of the present disclosure. It should be noted that the distillation loss involved in the embodiments of the present disclosure is determined based on the second image feature sequence output by the second model and the first image feature sequence output by the first model, and the rest is consistent with the related art.

[0127] It can be understood that the present disclosure Figure 8This is just a schematic illustration. Figure 8 Any one of the examples may also be any other example except the present disclosure, and the present disclosure is not limited to this.

[0128] The video processing method provided by the embodiment of the present disclosure reduces the computational overhead of the sign language recognition model by retaining key frames and reducing redundant frames. Unlike the downsampling method proposed in [5], this method does not need to introduce additional proxy features, nor does it need to set a fixed number of segments. It can be dynamically adjusted according to the sequence length and can be used in any stage of feature extraction. In the use stage, when used for sign language video summary generation, this technology is independent of the specific use model. Given a video length T and a video summary length K, this technology can be used to select key frames based on the similarity of the intermediate layer features to generate the corresponding video summary; when used to accelerate the sign language recognition model, the training model obtained in the second stage is used for inference.

[0129] Fig. 9 FIG. 1 is a schematic diagram showing a comparison result of a sign language video summary generated by using adaptive dynamic temporal pooling and using uniform sampling according to an exemplary embodiment of the present disclosure. Fig. 9 As shown, the total length of the video is T. Fig. 9 The A part in is the image corresponding to the key frame, that is, the action expressing the main meaning in the sign language video. Based on dynamic adaptive temporal sampling, redundant frames are significantly reduced and key frames are retained, such as Fig. 9 As shown in part B. Fig. 9 B1 in the B part is a key frame. Based on uniform sampling, there are redundant frames, such as Fig. 9 As shown in part C. Fig. 9 The C1 in the C part is a redundant frame.

[0130] In the disclosed embodiment, Fig.10 FIG. 4 shows a comparison table of the results of using VAC as a reference model and using ADTP according to an exemplary embodiment of the present disclosure. Fig.10As shown in the figure, the frame rate has a great impact on the recognition performance of the baseline model, and the direct use of the maximum pooling method can bring a small performance improvement, but it will also bring a certain amount of computational overhead (for example, when the maximum pooling method is used to downsample at C3, the feature frame rate is reduced to 3FPS, and the word error rate is reduced by 6.1% / 6.3% compared to the Baseline (3FPS), but the computational amount is increased by 33.1GFLOPs). Using ADTP can effectively reduce the word error rate under the same amount of computation (for example, when ADTP is used to downsample at C3, the word error rate is reduced by 11.0% / 11.1%); at the same time, it can better balance the accuracy and computational overhead. When using an input frame rate of 8FPS and a downsampling rate of 0.375, using ADTP at C3 can achieve a lower word error rate (22.0% / 21.8% vs. 23.1% / 23.7%) than the Baseline (6FPS) at a smaller computational cost (89.2 vs. 91.5GFLOPs).

[0131] Based on the same concept, an embodiment of the present disclosure also provides a video processing device.

[0132] It is understandable that, in order to realize the above functions, the video processing device provided by the embodiment of the present disclosure includes hardware structures and / or software modules corresponding to the execution of each function. In combination with the units and algorithm steps of each example disclosed in the embodiment of the present disclosure, the embodiment of the present disclosure can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art may use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the technical solution of the embodiment of the present disclosure.

[0133] Fig.11 is a block diagram of a video processing device according to an exemplary embodiment. Fig.11 The device 100 includes an extracting unit 101, a processing unit 102, and a generating unit 103.

[0134] The extraction unit 101 is used to extract image features of N frames of images in a video sequence, where N is a positive integer and the N frames of images include hand gesture information of a target object. The processing unit 102 is used to select n key frame images in chronological order from the N frames of images based on the image features, where N is greater than or equal to n, wherein the n key frame images have the same chronological order as the N frames of images, and n is a preset value of the number of image frames included in the video summary to be generated, and the image features of the key frame images are the target hand motion features. The generation unit 103 is used to generate a video summary based on the key frame images.

[0135] In one embodiment, the processing unit 102 selects n key frame images in sequence in chronological order from N frames of images based on image features in the following manner: based on the difference between N and n, and the value of n, the size of the dynamic temporal pooling region of the image feature sequence of the N frames of images is adjusted to obtain a subsequence whose cumulative cost is less than a preset threshold, the cumulative cost is used to represent the alignment path of the feature sequence and the subsequence, and the dynamic temporal pooling is used to segment the image feature sequence in chronological order to extract the subsequence. The frame image in the subsequence is used as the key frame image.

[0136] In one embodiment, the processing unit 102 also uses the following method to adjust the size of the dynamic temporal pooling area of ​​the image feature sequence of N frames of images based on the difference between N and n, and the value of n, to obtain a subsequence whose cumulative cost is less than a preset threshold: determine the alignment cost of temporal alignment between the i-th frame and the j-th frame frame by frame in the image feature sequence, and the number of frames corresponding to the i-th frame and the j-th frame is less than or equal to the number of image frames. Determine the minimum cumulative cost of the cumulative cost of the frame before the i-th frame and the j-th frame, the cumulative cost of the i-th frame and the frame before the j-th frame, and the cumulative cost of the frame before the i-th frame and the frame before the j-th frame after frame extraction. Determine the sum of the alignment cost and the minimum cumulative cost as the cumulative cost of the current frame. Based on the cumulative cost, determine a subsequence whose cumulative cost is less than a preset threshold.

[0137] In one implementation, the generating unit 103 is further configured to: generate an image feature sequence with a length of N based on the key frame image, and the image feature sequence is used for sign language recognition.

[0138] In one implementation, the key frame image is extracted using a spatiotemporal convolutional network.

[0139] The generating unit 103 generates an image feature sequence with a length of N based on the key frame image in the following manner: using a convolutional neural network correlated with the spatiotemporal convolutional neural network to restore the image features of the key frame image and obtain an image feature sequence with a length of N.

[0140] In one embodiment, the processing unit 102 is further used to: upsample the image feature sequence to obtain an upsampled sequence. The upsampled sequence is used to constrain the first model, the first model is used to train the second model using a distillation method, the second model is used to extract image features of N frames of images in the sign language video sequence, and based on the image features, select n key frame images in the N frames in chronological order to generate a sign language video summary.

[0141] In one implementation, the processing unit 102 is further configured to obtain the second model based on the first model training in the following manner:

[0142] The video sequence training sample is input into the first model to obtain the first image feature sequence. The first image feature sequence is input into the prediction model of the second model, and the prediction model of the second model downsamples the first image feature sequence to obtain the key frame image. The image features of the key frame image are restored by upsampling to obtain a second image feature sequence with the same length as the first image feature sequence. The distillation loss is determined based on the first image feature sequence and the second image feature sequence, and the distillation loss is used to constrain the first model, and the prediction model of the second model is trained to obtain the second model.

[0143] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0144] Fig.12 2 is a block diagram of a device for video processing according to an exemplary embodiment. For example, the device 200 may be a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, etc.

[0145] Reference Fig.12 , the device 200 may include one or more of the following components: a processing component 202 , a memory 204 , a power component 206 , a multimedia component 208 , an audio component 210 , an input / output (I / O) interface 212 , a sensor component 214 , and a communication component 216 .

[0146] The processing component 202 generally controls the overall operation of the device 200, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component 202 may include one or more processors 220 to execute instructions to perform all or part of the steps of the above-described method. In addition, the processing component 202 may include one or more modules to facilitate interaction between the processing component 202 and other components. For example, the processing component 202 may include a multimedia module to facilitate interaction between the multimedia component 208 and the processing component 202.

[0147] The memory 204 is configured to store various types of data to support operations on the device 200. Examples of such data include instructions for any application or method operating on the device 200, contact data, phone book data, messages, pictures, videos, etc. The memory 204 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0148] The power component 206 provides power to the various components of the device 200. The power component 206 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power for the device 200.

[0149] The multimedia component 208 includes a screen that provides an output interface between the device 200 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 208 includes a front camera and / or a rear camera. When the device 200 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera may receive external multimedia data. Each front camera and rear camera may be a fixed optical lens system or have a focal length and optical zoom capability.

[0150] The audio component 210 is configured to output and / or input audio signals. For example, the audio component 210 includes a microphone (MIC), and when the device 200 is in an operation mode, such as a call mode, a recording mode, and a speech recognition mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in the memory 204 or sent via the communication component 216. In some embodiments, the audio component 210 also includes a speaker for outputting audio signals.

[0151] I / O interface 212 provides an interface between processing component 202 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include but are not limited to: a home button, a volume button, a start button, and a lock button.

[0152] The sensor assembly 214 includes one or more sensors for providing various aspects of the status assessment of the device 200. For example, the sensor assembly 214 can detect the open / closed state of the device 200, the relative positioning of components, such as the display and keypad of the device 200, the sensor assembly 214 can also detect the position change of the device 200 or a component of the device 200, the presence or absence of user contact with the device 200, the orientation or acceleration / deceleration of the device 200 and the temperature change of the device 200. The sensor assembly 214 can include a proximity sensor configured to detect the presence of a nearby object without any physical contact. The sensor assembly 214 can also include an optical sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 214 can also include an accelerometer, a gyroscope sensor, a magnetic sensor, a pressure sensor or a temperature sensor.

[0153] The communication component 216 is configured to facilitate wired or wireless communication between the device 200 and other devices. The device 200 can access a wireless network based on a communication standard, such as WiFi, 2G or 3G, or a combination thereof. In an exemplary embodiment, the communication component 216 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 216 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0154] In an exemplary embodiment, the apparatus 200 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors or other electronic components to perform the above method.

[0155] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 204 including instructions, and the instructions can be executed by the processor 220 of the device 200 to perform the above method. For example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0156] It is to be understood that in the present disclosure, "plurality" refers to two or more than two, and other quantifiers are similar. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the associated objects before and after are in an "or" relationship. The singular forms "a", "the" and "the" are also intended to include plural forms, unless the context clearly indicates other meanings.

[0157] It is further understood that the terms "first", "second", etc. are used to describe various information, but such information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other, and do not indicate a specific order or degree of importance. In fact, the expressions "first", "second", etc. can be used interchangeably. For example, without departing from the scope of the present disclosure, the first information can also be referred to as the second information, and similarly, the second information can also be referred to as the first information.

[0158] It can be further understood that, unless otherwise specified, “connection” includes a direct connection without other components between the two, and also includes an indirect connection with other components between the two.

[0159] It is further understood that, although the operations are described in a specific order in the drawings in the embodiments of the present disclosure, it should not be understood as requiring the operations to be performed in the specific order shown or in a serial order, or requiring the execution of all the operations shown to obtain the desired results. In certain environments, multitasking and parallel processing may be advantageous.

[0160] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. This application is intended to cover any modifications, uses or adaptations of the present disclosure, which follow the general principles of the present disclosure and include common knowledge or customary technical means in the art that are not disclosed in the present disclosure.

[0161] It should be understood that the present disclosure is not limited to the precise structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the scope of the appended claims.

Claims

1. A video processing method, characterized in that: include: Extracting image features of N frames of images in a video sequence, where N is a positive integer, and the N frames of images include hand gesture information of a target object; Based on the image features, n key frame images are selected in sequence from the N frames of images in chronological order, where N is greater than or equal to n, wherein the n key frame images are in the same chronological order as the N frames of images, n is a preset number of image frames included in the video summary to be generated, and the image features of the key frame images are target hand movement features; A video summary is generated based on the key frame images.

2. The method according to claim 1, characterized in that The selecting, based on the image features, n key frame images in sequence from the N frames of images in chronological order, comprises: Based on the difference between N and n, and the value of n, the size of the dynamic temporal pooling region of the image feature sequence of the N frames of images is adjusted to obtain a subsequence whose cumulative cost is less than a preset threshold, wherein the cumulative cost is used to represent an alignment path between the feature sequence and the subsequence, and the dynamic temporal pooling is used to segment the image feature sequence in chronological order to extract the subsequence; The frame images in the subsequence are used as the key frame images.

3. The method according to claim 2, characterized in that The adjusting the size of the dynamic temporal pooling region of the image feature sequence of the N frames of images based on the difference between the N and the n, and the value of the n, to obtain a subsequence whose cumulative cost is less than a preset threshold, includes: Determining, frame by frame in the image feature sequence, an alignment cost for performing temporal alignment between an i-th frame and a j-th frame, wherein the number of frames corresponding to the i-th frame and the j-th frame is less than or equal to the number of image frames; Determine the minimum cumulative cost among the cumulative cost of the frame before the i-th frame and the j-th frame, the cumulative cost of the frame before the i-th frame and the j-th frame, the cumulative cost of the frame before the i-th frame and the j-th frame, and the cumulative cost of the frame before the i-th frame and the j-th frame after frame extraction; Determine the sum of the alignment cost and the minimum cumulative cost as the cumulative cost of the current frame; Based on the cumulative cost, a subsequence whose cumulative cost is less than a preset threshold is determined.

4. The method according to any one of claims 1 to 3, characterized in that The method further comprises: Based on the key frame image, an image feature sequence with a length of N is generated, and the image feature sequence is used for sign language recognition.

5. The method according to claim 4, characterized in that The key frame image is extracted by using a spatiotemporal convolutional network; The step of generating an image feature sequence having a length of N based on the key frame image comprises: The image features of the key frame image are restored by using a convolutional neural network that is correlated with the spatiotemporal convolutional neural network to obtain an image feature sequence with a length of N.

6. The method according to claim 4 or 5, characterized in that: The method further comprises: Upsampling the image feature sequence to obtain an upsampled sequence; The upsampling sequence is used to constrain the first model, the first model is used to train the second model by distillation, the second model is used to extract image features of N frames in the video sequence, and based on the image features, n key frame images are selected in chronological order from the N frames to generate a video summary.

7. The method according to claim 6, characterized in that The second model is trained based on the first model in the following manner: Inputting the video sequence training sample into the first model to obtain a first image feature sequence; Inputting the first image feature sequence into a prediction model of a second model, and using the prediction model of the second model to downsample the first image feature sequence to obtain a key frame image; Restore the image features of the key frame image by upsampling to obtain a second image feature sequence with the same length as the first image feature sequence; A distillation loss is determined based on the first image feature sequence and the second image feature sequence, the first model is constrained by using the distillation loss, and a prediction model of the second model is trained to obtain the second model.

8. A video processing device, characterized in that: The device comprises: An extraction unit, used to extract image features of N frames of images in a video sequence, where N is a positive integer, and the N frames of images include hand gesture information of a target object; a processing unit, configured to select n key frame images in sequence from the N frames of images in chronological order based on the image features, wherein N is greater than or equal to n, wherein the n key frame images are in the same chronological order as the N frames of images, n is a preset number of image frames included in the video summary to be generated, and the image features of the key frame images are target hand motion features; A generating unit is used to generate a video summary based on the key frame image.

9. A video processing device, characterized in that: include: processor; a memory for storing processor-executable instructions; The processor is configured to execute the method described in any one of claims 1 to 7.

10. A storage medium, characterized in that: The storage medium stores instructions, and when the instructions in the storage medium are executed by a processor of the terminal, the terminal is enabled to execute the method described in any one of claims 1 to 7.

Images