A video classification method, apparatus, device, and computer-readable storage medium

By using a video classification model with a convolutional attention module, the problems of audio modal data interference and target object size in existing technologies are solved, thereby improving the accuracy and precision of video classification.

CN117152649BActive Publication Date: 2026-11-17CHINA MOBILE CHENGDU INFORMATION & TELECOMM TECH CO LTD +1
View PDF 4 Cites 0 Cited by

Patent Information

Application Number
CN202210547401.3
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-05-18
Publication Date
2026-11-17
Estimated Expiration
2042-05-18

AI Technical Summary

Technical Problem

Existing video classification methods can cause interference when processing audio modal data, reducing classification accuracy. They also have difficulty handling target objects of different sizes, which affects the accuracy of video classification.

Method used

A pre-trained video classification model is used, which includes a convolutional attention module. The weights of the convolutional kernels that match the size of the target object are greater than the weights of other convolutional kernels. This module is used to classify the pre-processed target image frames.

Benefits of technology

It improves the accuracy of video classification, can handle target objects of different sizes, and enhances the precision of video classification.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN117152649B_ABST
    Figure CN117152649B_ABST
Patent Text Reader

Abstract

This application discloses a video classification method, apparatus, device, and computer-readable storage medium. The method includes: acquiring at least one target image frame corresponding to a video file to be processed; preprocessing each target image frame to obtain at least one preprocessed target image frame and a trained video classification model; using the trained video classification model to classify the at least one preprocessed target image frame to obtain a classification result of the video file to be processed, and outputting the classification result. The trained video classification model includes at least a convolutional attention module with multiple convolutional kernels. The weight of the convolutional kernel matching the size of a target object in the preprocessed target image frame is greater than the weight of other convolutional kernels, thereby enabling the acquisition of different target objects matching each convolutional kernel, realizing the processing of target objects of different sizes in the preprocessed target image frame, and improving the accuracy of video classification.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a video method, apparatus, device, and computer-readable storage medium. Background Technology

[0002] With the ever-increasing volume of online video data, video data processing technology has been widely adopted. Understanding the content represented by video data and classifying videos has become a fundamental issue in video data processing. Video classification technology primarily assigns a video category based on its content. This technology can be widely applied in scenarios such as short video recommendation, autonomous driving assistance, and intelligent security. Accurately and quickly understanding video content and assigning corresponding categories can greatly improve and enhance the user experience.

[0003] In related technologies, one video classification method is based on an Inflated 3D ConvNet (I3D) network, which divides video data into audio modal data and visual modal data for separate processing. This method processes the audio modal data in the video, which can cause some interference to video classification in some scenarios and reduce the accuracy of video classification. Another video classification method extracts optical flow image frames and RGB image frames from the video and uses an I3D network with channel attention mechanism to classify the RGB image frames and optical flow image frames. However, this method has difficulty handling target objects of different sizes in the video, thus failing to guarantee the accuracy of video classification. Summary of the Invention

[0004] To address the aforementioned technical problems, this application aims to provide a video classification method that can improve the accuracy of video classification.

[0005] The technical solution of the application embodiment is implemented as follows:

[0006] This application provides a video classification method, including:

[0007] Obtain at least one target image frame corresponding to the video file to be processed, and preprocess the at least one target image frame to obtain at least one preprocessed target image frame;

[0008] A trained video classification model is obtained, wherein the trained video classification model includes at least a convolutional attention module, wherein the convolutional attention module includes multiple convolutional kernels, and the weight of the convolutional kernel that matches the size of the target object in the preprocessed target image frame is greater than the weight of the other convolutional kernels.

[0009] The trained video classification model is used to classify the at least one preprocessed target image frame to obtain the classification result of the video file to be processed.

[0010] Output the classification results.

[0011] This application provides a video classification device, including:

[0012] The first acquisition module is used to acquire at least one target image frame corresponding to the video file to be processed, and to preprocess the at least one target image frame to obtain at least one preprocessed target image frame.

[0013] The second acquisition module is used to acquire a trained video classification model. The trained video classification model includes at least a convolutional attention module. The convolutional attention module includes multiple convolutional kernels. The weight of the convolutional kernel that matches the size of the target object in the preprocessed target image frame is greater than the weight of the other convolutional kernels.

[0014] The processing module is used to classify the at least one preprocessed target image frame using the trained video classification model to obtain the classification result of the video file to be processed.

[0015] The output module is used to output the classification results.

[0016] This application provides a video classification device, including:

[0017] Memory, used to store executable video classification instructions;

[0018] The processor, when executing executable video classification instructions stored in the memory, implements the video classification method provided in the embodiments of this application.

[0019] This application provides a computer-readable storage medium storing computer-executable video classification instructions configured to execute the video classification method provided in this application.

[0020] This application provides a video classification method, apparatus, device, and computer-readable storage medium. Using this technical solution, firstly, at least one target image frame corresponding to the video file to be processed is acquired; each target image frame is preprocessed to obtain at least one preprocessed target image frame; then, a trained video classification model is acquired; the trained video classification model is used to classify the at least one preprocessed target image frame to obtain the classification result of the video file to be processed, and the classification result is output. In this application embodiment, the trained video classification model includes at least a convolutional attention module with multiple convolutional kernels. Among the multiple convolutional kernels, the weight of the convolutional kernel that matches the size of the target object in the preprocessed target image frame is greater than the weight of the other convolutional kernels, thereby enabling the acquisition of different target objects matching each convolutional kernel, realizing the processing of target objects of different sizes in the preprocessed target image frame, and improving the accuracy of video classification. Attached Figure Description

[0021] Figure 1 A flowchart illustrating a video classification method provided in an embodiment of this application;

[0022] Figure 2 A schematic flowchart of a method for acquiring a target image frame provided in the application embodiment;

[0023] Figure 3 A schematic diagram of a video classification method based on the SK-I3D network model provided in this application embodiment;

[0024] Figure 4 A schematic diagram of the structure of an SK-I3D video classification model provided in an embodiment of this application;

[0025] Figure 5 A schematic diagram of the structure of an SK-Inc module provided in an embodiment of this application;

[0026] Figure 6 This is a schematic diagram of the composition structure of a video classification device provided in an embodiment of this application;

[0027] Figure 7 This is a schematic diagram of the composition structure of a video classification device provided in an embodiment of this application. Detailed Implementation

[0028] The technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings.

[0029] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0030] In the following description, references to “some embodiments” or “other embodiments” describe a subset of all possible embodiments. However, it is understood that “some embodiments” or “other embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0031] In the following description, the terms "first" and "second" are used merely to distinguish similar objects and do not represent a specific ordering of objects. It is understood that "first" and "second" may be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0032] Unless otherwise defined, all technical and scientific terms used in this application have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains. The terminology used in this application is for the purpose of describing embodiments of this application only and is not intended to be limiting of this application.

[0033] To better understand the video classification method provided in the embodiments of this application, the video classification methods in related technologies and their shortcomings are first explained.

[0034] Chinese patent application CN202010117185.X proposes a single-label classification method for short videos based on multimodal feature fusion using deep networks. This method selects three I3D networks to process audio and visual modal data. The model constructed with three I3D networks significantly increases computational cost, greatly reducing the prediction speed of video classification. Furthermore, processing audio modal data in the video can interfere with video classification in some scenarios, such as classifying videos with background music in live streaming. Therefore, this method has a slow prediction speed in video classification, making it unsuitable for scenarios requiring real-time or near-real-time processing. Moreover, introducing audio modal data can reduce video classification accuracy in certain scenarios.

[0035] Chinese patent application CN202110116862.0 proposes an action recognition method based on dual-stream convolutional attention. This method first divides the video into K segments, then randomly extracts RGB image frames from each segment, and simultaneously extracts optical flow image frames. The RGB and optical flow image frames are then classified using I3D with an attention mechanism. However, this approach suffers from several drawbacks. The random sampling method misses crucial information segments in the video, and dividing it into K sub-videos leads to a loss of correlations between multiple sub-videos. Furthermore, extracting optical flow image frames from the video requires significant computational power, making it unsuitable for real-time or near-real-time scenarios. Although it uses an attention mechanism to improve I3D, this method only uses channel attention and cannot adaptively focus on key targets of different sizes within the video, thus compromising the accuracy of video classification.

[0036] To address the aforementioned issues, this application provides a video classification method that can guarantee the prediction speed of video classification and improve the accuracy of video classification.

[0037] The following will describe the video classification method provided in the embodiments of this application. This method can be applied to a video classification device, which can be a terminal such as a smartphone, desktop computer, laptop, tablet computer, or server. Figure 1 The diagram shown is a flowchart of a video classification method provided in an embodiment of this application. The method includes the following steps:

[0038] S101. Obtain at least one target image frame corresponding to the video file to be processed, and preprocess the at least one target image frame to obtain at least one preprocessed target image frame.

[0039] It should be noted that an image frame can be multiple frames corresponding to the video file to be processed, and a target image frame can be a key image frame in the video file to be processed. Key image frames can reflect changes in video content and can completely restore the original video. In some embodiments, only RGB image frames can be extracted from the video file to be processed, that is, all obtained target image frames can be RGB image frames.

[0040] In some embodiments, preprocessing of the target image frame may involve cropping at least one obtained target image frame so that the cropped target image frame meets the input requirements of the video classification model. Preprocessing at least one target image frame may involve cropping the target image frame to a preset size. The preset size may be a pre-defined image size, representing the size of the image frame. If the length and width of the image frame are a and b respectively, then the image size can be represented by a*b. The preset size can be any suitable value, such as 224*224, as long as it meets the image size requirements of the classification model. For example, if the classification model requires an image size of 224*224, the preset size can be 224*224. If the image size of the target image frame obtained from the video file to be processed is 800*800, then the target image frame with an image size of 800*800 can be cropped to an image frame with an image size of 224*224.

[0041] S102. Obtain the trained video classification model.

[0042] In some embodiments, the trained video classification model includes at least a convolutional attention module, which includes multiple convolutional kernels. The weight of the convolutional kernel that matches the size of the target object in the preprocessed target image frame is greater than the weight of the other convolutional kernels.

[0043] In some embodiments, the convolutional kernels in the convolutional attention module may include ordinary convolutional kernels and dilated convolutional kernels, and the number of ordinary convolutional kernels and dilated convolutional kernels may be one or more. Multiple convolutional kernels have their own corresponding weights, and different convolutional kernels are matched with target objects of different sizes. When feature extraction of a target object in a target image frame is performed based on the convolutional kernels, the weight of the convolutional kernel that matches the size of the target object in the target image frame is greater than the weight of other convolutional kernels. This allows each convolutional kernel to extract all features of the target object it matches, thereby obtaining a more accurate and comprehensive view of the video file to be processed.

[0044] S103. Use the trained video classification model to classify at least one preprocessed target image frame to obtain the classification result of the video file to be processed.

[0045] In some embodiments, after obtaining the preprocessed target image frame and the trained video classification model, the preprocessed target image frame can be input into the trained video classification model, and the trained video classification model can be used to classify the preprocessed target image frame to obtain the classification result of the video file to be processed corresponding to the preprocessed target image frame.

[0046] S104. Output the classification results.

[0047] After obtaining the classification result corresponding to the video file to be processed, the classification result can be output from the trained video classification model. The classification result can represent the category of the video file to be processed, such as technology, food, travel, sports, etc.

[0048] In some embodiments, when step S104 is implemented by a terminal device such as a smartphone or desktop computer, the output classification result may be presented through the display device of the terminal device; when the step is implemented by a server, the output classification result may be sent to the terminal requesting video classification, and then the terminal requesting video classification will present the classification result through its own display device after receiving the classification result.

[0049] In the video classification method provided in this application embodiment, firstly, at least one target image frame corresponding to the video file to be processed is obtained; each target image frame is preprocessed to obtain at least one preprocessed target image frame; then, a trained video classification model is obtained; the trained video classification model is used to classify the at least one preprocessed target image frame to obtain the classification result of the video file to be processed, and the classification result is output. In this application embodiment, the trained video classification model includes at least a convolutional attention module with multiple convolutional kernels. Among the multiple convolutional kernels, the weight of the convolutional kernel that matches the size of the target object in the preprocessed target image frame is greater than the weight of other convolutional kernels, thereby obtaining different target objects that match each convolutional kernel, realizing the processing of target objects of different sizes in the preprocessed target image frame, and improving the accuracy of video classification.

[0050] like Figure 2 The diagram shown is a flowchart of a method for obtaining a target image frame according to an embodiment of this application. In some embodiments of this application, obtaining at least one target image frame corresponding to the video file to be processed, i.e., step S101, can also be implemented by the following steps S1011 to S1018. Each step is described in detail below.

[0051] S1011. Adjust the resolution of the video file to be processed based on the preset resolution to obtain the first video file.

[0052] It should be noted that the resolution of a video file can be used to measure the amount of data within a corresponding image frame, representing the effective pixels in the horizontal and vertical directions of the image frame. Since there may be multiple video files to be processed, and these files will have different resolutions, these differences can make it difficult for video classification models to handle different video files. Therefore, it is necessary to uniformly process all video files to ensure that their resolutions are consistent, facilitating the video classification model's processing of each file.

[0053] In some embodiments, the preset resolution can be any pre-set value, such as 720×480, 1920×1080, etc. After adjusting the resolution of the video file to be processed based on the preset resolution, the resolution of the first video file obtained is the preset resolution.

[0054] S1012. Adjust the frame rate of the first video file based on the first preset frame rate to obtain the second video file.

[0055] It should be noted that frame rate can represent the rate or frequency at which images appear continuously on the display in units of frames, expressed in frames per second (fps). The first preset frame rate can be a preset frame rate, which can be any pre-set positive integer, such as 5, 8, 10, etc.

[0056] In some embodiments, when there are multiple video files to be processed, the frame rates of the various first video files may differ after resolution adjustment, making it difficult for the video classification model to process different video files. Therefore, it is necessary to adjust the frame rate of each first video file so that the various video files, i.e., the second video files, have the same frame rate after frame rate adjustment. In some embodiments, the video duration of the second video file may be different from the video duration of the first video file before frame rate adjustment. For example, if the video duration of a first video file is 6 minutes, the video duration of the second video file obtained after frame rate adjustment may be 10 minutes.

[0057] S1013. Extract image frames from the second video file at intervals of a first preset duration to obtain multiple extracted image frames.

[0058] It should be noted that the first preset duration can be any preset duration, which can be used as the sampling period for extracting image frames from the second video file. Image frames are extracted once every first preset duration until the end time of the video is reached, thereby obtaining multiple image frames corresponding to the second video file.

[0059] S1014. Obtain the number of image frames S extracted from the image frames.

[0060] S1015. Determine whether the number of image frames S is less than the preset frame number threshold Q.

[0061] In some embodiments, after obtaining the image frames corresponding to the second video file, the number of image frames can be counted to obtain the number of image frames S. Then, it can be determined whether the obtained number of image frames S is less than a preset frame count threshold Q. The preset frame count threshold Q can be a pre-defined number of image frames, used to represent the number of image frames that meet the requirements of the video classification model. The frame count threshold Q can be any positive integer, such as 7, 10, 15, etc.

[0062] In some embodiments, if the number of image frames S is less than the frame number threshold Q, then steps S1016 to S1017 are executed; otherwise, if the number of image frames S is greater than or equal to the preset frame number threshold Q, then step S1018 is executed.

[0063] S1016. Obtain the last image frame extracted from the second video file.

[0064] It should be noted that the multiple image frames extracted from the second video file have timestamps, and the last image frame can be the image frame that is arranged last after being sorted by timestamp. In some embodiments, when the number S of the multiple extracted image frames is less than the frame number threshold Q, it means that the number of image frames extracted from the second video file does not meet the number of image frames required by the classification model. In this case, the last image frame can be obtained from the multiple extracted image frames.

[0065] S1017. Determine the extracted multiple image frames and (QS) last image frames as at least one target image frame corresponding to the second video file.

[0066] In some embodiments, after obtaining multiple image frames corresponding to the second video file and the last image frame among the multiple image frames, the difference (QS) between the number of image frames S and the frame number threshold Q can be determined first, and then the multiple image frames and the last image frame of (QS) can be determined as the target image frame corresponding to the second video file.

[0067] S1018. The extracted multiple image frames are identified as at least one target image frame corresponding to the second video file.

[0068] In other embodiments, if the number of image frames S is greater than or equal to a preset frame number threshold Q, it means that the number of image frames extracted from the second video file meets the number of image frames required by the video classification model. In this case, the extracted multiple image frames can be directly used as the target image frames corresponding to the second video file.

[0069] It is understandable that adjusting the resolution and frame rate of the video file to be processed, and determining the target image frame corresponding to the video file to be processed based on multiple image frames and the last image frame when the number of extracted multiple image frames does not meet the requirements of the video classification model, can avoid the problem that the video classification model has difficulty processing different video files due to differences in resolution, frame rate, video duration, etc., and provide the prerequisites for the classification processing of the video classification model.

[0070] In some embodiments of this application, the trained video classification model further includes a convolution module, a pooling module, and a classification module. Based on this, the trained video classification model is used to classify at least one preprocessed target image frame to obtain the classification result of the video file to be processed, i.e., step S103. This can also be achieved through the following steps S1031 to S1034.

[0071] S1031. Using the convolution module in the trained video classification model, perform convolution processing on at least one preprocessed target image frame to obtain at least one first feature map after convolution processing.

[0072] In some embodiments, the convolutional modules in a trained video classification model may include one or more convolutional modules, each corresponding to one or more convolutional kernels. All convolutional kernels may be three-dimensional kernels, and the size of a convolutional kernel can be represented by a*b*t, where a, b, and t represent the length, width, and duration of the kernel, respectively. For example, a convolutional module may include one 7*7*7 kernel, two 1*1*1 kernels, and one 3*3*3 kernel, with the stride of each kernel set to 2.

[0073] In some embodiments, when classifying preprocessed target image frames using a trained video classification model, a convolution module can first be used to convolve the preprocessed target image frames to obtain one or more first feature maps after convolution. The first feature map can represent the feature map obtained after convolving the processed target image frame using the corresponding convolution kernel in the convolution module. The number of first feature maps can correspond to the number of convolution kernels. When multiple first feature maps are obtained, each feature map may be different.

[0074] S1032. Use the pooling module to perform pooling processing on at least one first feature map to obtain at least one second feature map after pooling processing.

[0075] It should be noted that pooling modules can include one or more, and the size of the sliding window corresponding to different pooling modules can be different, as can the stride of the sliding window. In some embodiments, the size of the sliding window can be represented by m*n*l, where m, n, and l represent the length, width, and duration, respectively, and the stride can be represented by stride = a, b, c, where a, b, and c represent the stride corresponding to the sliding window sliding in the length, width, and duration dimensions, respectively.

[0076] In some embodiments, after obtaining at least one first feature map after convolution processing, the first feature map can be pooled using a pooling module to obtain a pooled second feature map. When there are multiple pooling modules, different pooling modules can perform max pooling or average pooling on the first feature map. Hereinafter, the pooling module that performs max pooling is referred to as the max pooling module, and the pooling module that performs average pooling is referred to as the average pooling module.

[0077] For example, in some embodiments, the trained video classification model may include 5 pooling modules, wherein the sliding window size corresponding to two max pooling modules is 1*3*3 with stride=1,2,2; the sliding window size corresponding to one max pooling module is 1*3*3 with stride=2,2,2; the sliding window size corresponding to one max pooling module is 2*2*2 with stride=2,2,2; and the sliding window size corresponding to one average pooling module is 2*7*7 with stride=2,2,2.

[0078] S1033. Use the convolutional attention module to perform convolution and aggregation processing on at least one second feature map to obtain at least one third feature map.

[0079] It should be noted that the convolutional attention module corresponds to one or more convolutional kernels. These kernels can include ordinary convolutional kernels and dilated convolutional kernels, and the number of ordinary and dilated kernels can be one or more. In some embodiments, after obtaining the second feature map after pooling by the pooling module, the second feature map can be convolved separately based on the multiple convolutional kernels corresponding to the convolutional attention module to obtain multiple convolutional processing results. Then, the multiple convolutional processing results are aggregated to obtain the processed third feature map.

[0080] S1034. Use the classification module to determine the classification result of at least one video file corresponding to the third feature map.

[0081] It should be noted that the classification module can determine the classification result of the video file to be processed based on at least one third feature map. The classification result can be the category to which the video file to be processed belongs. In some embodiments, the classification module may include a softmax classifier, which can be used to determine the probability of the current video file to be processed belonging to different categories. Then, the category with the highest probability is selected as the category to which the current video file to be processed belongs.

[0082] In some embodiments of this application, the convolutional attention module may include M first convolutional kernels and N second convolutional kernels, wherein the second convolutional kernels are dilated convolutional kernels, and M and N are both integers greater than 0. Based on this, the convolutional attention module is used to perform convolution and aggregation processing on at least one second feature map to obtain at least one third feature map. That is, step S1033 can also be implemented by the following steps S201 to S204.

[0083] S201. Using each of the first convolution kernels in the convolutional attention module, perform convolution processing on the j-th second feature map to obtain M first convolution results corresponding to the j-th second feature map.

[0084] It should be noted that, where j = 1, 2, ..., P, and P is the total number of feature maps. For example, if the number of first convolution kernels M is 2 and the number of second feature maps is 3, i.e., j = 1, 2, 3, then when using the first convolution kernels to perform convolution processing on the second feature maps, when j = 1, the first second feature map is convolved using two first convolution kernels respectively, resulting in two first convolution results; when j = 2, the second second feature map is convolved using two first convolution kernels respectively, resulting in two first convolution results; and when j = 3, the third second feature map is convolved using two first convolution kernels respectively, resulting in two first convolution results.

[0085] S202. Perform convolution processing on the j-th second feature map using each second convolution kernel to obtain N second convolution results corresponding to the j-th second feature map.

[0086] In some embodiments, after performing convolution processing on each second feature map using each first convolution kernel, each second convolution kernel can also be used to perform convolution processing on each second feature map. For example, if the number of first convolution kernels N is 2 and the number of second feature maps is 3, i.e., j = 1, 2, 3, then when performing convolution processing on the second feature maps using the second convolution kernels, when j = 1, the first second feature map is convolved using two second convolution kernels respectively, resulting in two second convolution results; when j = 2, the second second feature map is convolved using two second convolution kernels respectively, resulting in two second convolution results; and when j = 3, the third second feature map is convolved using two second convolution kernels respectively, resulting in two second convolution results.

[0087] S203. Based on the first weights corresponding to each first convolution kernel and the second weights corresponding to each second convolution kernel, the M first convolution results corresponding to the j-th second feature map and the N second convolution results corresponding to the j-th second feature map are weighted and summed to obtain the j-th third feature map.

[0088] In some embodiments, for example, if the number M of the first convolution kernels is 2, the first weights corresponding to the two first convolution kernels are a and b, respectively; the number N of the second convolution kernels is also 2, and the second weights corresponding to the two second convolution kernels are c and d, respectively. If the number of second feature maps is 3, i.e., j = 1, 2, 3, when j = 1, if the two first convolution results corresponding to the first second feature map are U11 and U12, and the two second convolution results are U13 and U14, then the first third feature map can be represented as: A1 = a*U11 + b*U12 + c*U13 + d*U14; when j = 2, if the two first convolution results corresponding to the second second feature map are U21 and U22, 2 If the results of the second convolution are U23 and U24, then the second third feature map can be represented as: A2 = a*U21 + b*U22 + c*U23 + d*U24; when j = 3, if the two first convolution results corresponding to the third second feature map are U31 and U32, and the two second convolution results are U33 and U34, then the third third feature map can be represented as: A3 = a*U31 + b*U32 + c*U33 + d*U34.

[0089] In some embodiments of this application, before obtaining the trained video classification model, i.e., step S102, the following steps S301 to S304 can also be performed. Each step is described in detail below.

[0090] S301. Obtain the preset video classification model and obtain the training image frame set corresponding to each training video file.

[0091] It should be noted that each training video file corresponds to a category label, which indicates the category to which each training video file belongs. The preset video classification model can be an untrained video classification model. In addition to obtaining the preset video classification model and the category labels corresponding to the training video files, we can also obtain the set of image frames corresponding to the training video files. Since there are multiple training video files, each training video file can include multiple image frames. Therefore, by extracting image frames from multiple training video files, we can obtain the set of image frames.

[0092] In some embodiments, the process of obtaining image frames from training video files is similar to the process of obtaining target image frames from video files to be processed, and can also be implemented through steps S1011 to S1018. The process of obtaining image frames from training video files will not be described in detail in this application.

[0093] S302. Use a video classification model to predict the image frame set corresponding to each training video file to obtain the prediction result corresponding to each training video file.

[0094] In some embodiments, after obtaining the set of training image frames corresponding to each training video file, multiple training image frames can be input into a video classification model. The video classification model is then used to perform prediction processing on the set of image frames corresponding to each training video file, thereby obtaining the prediction result corresponding to each training video file. This prediction result can represent the category corresponding to each training video file.

[0095] S303. Based on the prediction results and category labels of each training video file, backpropagation training is performed on the video classification model to obtain the trained video classification model.

[0096] In some embodiments, after obtaining the prediction results corresponding to each training video file, the prediction results can be compared with the category labels corresponding to the training video files to determine the prediction error of the video classification model. Based on the prediction error, the video classification model can be backpropagated to obtain a trained video classification model.

[0097] In some embodiments of this application, the video classification model is backpropagated and trained based on the prediction results and category labels corresponding to each training video file to obtain the trained video classification model, i.e., step S303. It can also be implemented by the following steps S3031 to S3032. Each step is described in detail below.

[0098] S3031. Feed back the prediction results and category labels of each training video file to the video classification model and adjust the network parameters in the video classification model.

[0099] It should be noted that the network parameters include at least the weights corresponding to each first convolutional kernel and each second convolutional kernel in the convolutional attention module. In some embodiments, when training the video classification model through backpropagation, the prediction results and class labels corresponding to each training video file can be fed back to the video classification model to adjust the network parameters in the video classification model. These network parameters may include the weights corresponding to each first convolutional kernel and each second convolutional kernel, and may also include the network parameters in the convolutional and pooling modules, such as the size of the convolutional kernels and the sliding window.

[0100] In some embodiments, when prediction processing begins on the set of image frames corresponding to each training video file, each first convolutional kernel and each second convolutional kernel can be set with initial weights. Subsequently, based on the difference between the prediction results of the video classification model for each training video file and the category labels for each training video file, the initial weights corresponding to the first convolutional kernel and the initial weights corresponding to the second convolutional kernel can be adjusted.

[0101] It is understood that, in the embodiments of this application, when training the video classification model through backpropagation, the weights corresponding to each first convolutional kernel and each second convolutional kernel are adjusted so that the video classification model can adjust the convolutional kernels that need to be focused on according to the target objects of different sizes in the image frame, thereby ensuring that the video classification model can acquire target objects of different sizes in the image frame and thus improve the accuracy of video classification.

[0102] S3032. When the conditions for ending training are met, the trained video classification model is obtained.

[0103] It should be noted that the training can end when the prediction error of the video classification model is less than a preset threshold. For example, the prediction error of the video classification model can be represented by a loss function. For instance, cross-entropy can be used as the loss function Loss of the video classification model, as shown in equation (1) below:

[0104]

[0105] Where M represents the number of categories corresponding to the training video files, Q represents the number of training video files, and y ic For the conformance function, if the class label (true class) of sample i is class c, then the value is 1; otherwise, it is 0. ic This represents the probability that training video file i belongs to category c in the prediction results.

[0106] In some embodiments, when the loss value corresponding to the loss function is less than a preset threshold, for example, the preset threshold is 0.3 and the loss value corresponding to the loss function is 0.18, it can be determined that the video classification model meets the conditions for the end of training, thereby obtaining a well-trained video classification model.

[0107] In this embodiment, at least one target image frame corresponding to the video file to be processed is obtained. Each target image frame is preprocessed to obtain at least one preprocessed target image frame and a trained video classification model. The trained video classification model is then used to classify the at least one preprocessed target image frame to obtain the classification result of the video file to be processed, and the classification result is output. In this embodiment, the trained video classification model includes at least a convolutional attention module with multiple convolutional kernels. Among the multiple convolutional kernels, the weight of the convolutional kernel that matches the size of the target object in the preprocessed target image frame is greater than the weight of the other convolutional kernels. This allows for the acquisition of different target objects matching each convolutional kernel, enabling the processing of target objects of different sizes in the preprocessed target image frame and improving the accuracy of video classification.

[0108] The implementation process of the embodiments of this application in a practical application scenario will be described below.

[0109] In some embodiments, such as Figure 3 The diagram shown is a flowchart illustrating a video classification method based on the SK-I3D (Selective Kernel Inflated 3D ConvNet, SK-I3D) network model provided in this application embodiment. This method, based on the SK-I3D video classification model, can classify different video files. The video classification method provided in this application embodiment can be implemented through the following steps S401 to S408, each of which will be described in detail below.

[0110] S401. Obtain at least one original video file and at least one predicted video file, as well as the category label corresponding to each original video file.

[0111] It should be noted that the original video files (training video files) can be training samples, and each original video file corresponds to a category label. The predicted video files (video files to be processed) can be test samples.

[0112] S402. Preprocess at least one original video file and obtain at least one key image frame corresponding to each original video file.

[0113] In some embodiments, the original video files may differ in resolution, frame rate, and video duration. These differences make it difficult for the SK-I3D video classification model (video classification model) to handle different types of input. Therefore, it is necessary to perform uniform preprocessing on the original video files.

[0114] For example, in some embodiments, each collected original video file can be processed using the ffmpeg library to extract I keyframes at a rate of 1 frame per second, where I is an integer greater than 0, and then uniformly cropped to a size of 224x224. If the keyframes extracted from the original video file do not meet the minimum frame count requirement of the SK-I3D video classification model (if the number of image frames S is less than the frame count threshold Q), the last keyframe is used to repeat the process n times until the minimum frame count requirement is met. The processing commands for modifying resolution, cropping video, and obtaining keyframes are shown in the following steps:

[0115] (1) To modify the resolution and frame rate, use the following commands and parameters:

[0116] ffmpeg-i<input_file> -ss<start_time> -t<end_time> -s 224x224-r <fps>-favi out.avi, where input_file represents the input video file, start_time and end_time represent the start and end times of the video, and fps represents the frame rate.

[0117] (2) To crop a video, use the following commands and parameters:

[0118] ffmpeg-i<input_file> -ss<start_time> -t<end_time> -avoid_negative_ts1out.avi, where input_file represents the input video file, and start_time and end_time represent the start and end times of the video.

[0119] (3) To obtain keyframes, use the following commands and parameters:

[0120] ffmpeg-i<input_file> -vf select='eq(pict_type\,I)',setpts='N / (25*TB)' <out>_%05d.jpg, where input_file represents the input video file and out represents the prefix for the extracted frame file name.

[0121] S403. Use the SK-I3D video classification model to predict at least one key image frame and determine the prediction results corresponding to each original video file.

[0122] In some embodiments, in order to improve the prediction speed of the video classification model, optical flow features are not extracted from the video. Therefore, the optical flow branch network of the network is removed, and only the RGB flow branch network is retained. Based on this, an improved Inception module, namely the SK-Inc module (convolutional attention module), is introduced.

[0123] In this embodiment, the improved SK-I3D video classification model mainly consists of 9 SK-Inc modules, 4 convolutional modules, 4 max pooling modules, and 1 average pooling module (pooling module). A schematic diagram of the SK-I3D video classification model is shown below. Figure 4 As shown, the four convolutional modules consist of one 7x7x7 convolutional kernel, one 1x1x1 convolutional kernel, one 3x3x3 convolutional kernel, and one 1x1x1 convolutional kernel, with each kernel having a stride of 2. The four max pooling modules are as follows: one with a sliding window size of 1x3x3 and stride of 1, 2, 2; one with a sliding window size of 1x3x3 and stride of 1, 2, 2; one with a sliding window size of 1x3x3 and stride of 2, 2, 2; one with a sliding window size of 2x2x2 and stride of 2, 2, 2; and one with a sliding window size of 2x7x7 and stride of 2, 2, 2. The SK-I3D video classification model can be used to predict multiple key image frames corresponding to the original video file, and determine the prediction result corresponding to each original video file (the prediction result corresponding to each training video file). This prediction result can represent the category to which each original video file belongs.

[0124] In some embodiments, after preprocessing the original video file, some key target objects in the original video file will be scaled to varying degrees. In the original I3D Inception module, although convolutional kernels of different sizes are used to capture information of target objects in the video, the features extracted by convolutional kernels of different sizes are simply concatenated, that is, the importance of convolutional kernels for target objects of different sizes is not considered. Based on this, this application improves the Inception module in I3D by introducing the SK-Inc module, such as... Figure 5 As shown, the SK-Inc module introduces four paths and uses four different convolutional kernels with different receptive field sizes. The SK-Inc module includes one 1*1*1 convolutional kernel, one 3*3*3 convolutional kernel, one 3*3*3 dilated convolution with a dilation rate D of 2, and one 3*3*3 dilated convolution with a dilation rate D of 3.

[0125] Understandably, using a 1*1*1 convolutional kernel can introduce non-linear excitation to the input feature information, improve the expressive power of the classification network, and can also reduce or increase the dimensionality of the convolutional kernel channel number; using a 3*3*3 ordinary convolutional kernel can capture some key actions and other information that change over time in the video, and can also express information about some small target objects in the video.

[0126] In some embodiments, considering that changes in target objects in some videos only become noticeable after an interval of 1 to 2 seconds, a 3*3*3 dilated convolution with a dilation rate of 2 is used to extract short-interval information, such as actions like falling.

[0127] It is understandable that using a 3*3*3 dilated convolution with a dilation rate of 2 is equivalent to using a 5*5*5 ordinary convolution kernel, that is, it has the same receptive field size. Moreover, the 3*3*3 dilated convolution not only reduces the number of network parameters, but also reduces the network computation. Since the target objects in the video are at least larger than a single pixel, there is no problem of losing information continuity. Therefore, dilated convolution is very suitable for video classification tasks.

[0128] In some embodiments, a dilated convolution with a large receptive field, namely a 3*3*3 dilated convolution with a scaling factor of 3, is used. This is done to capture some relatively large target objects passing through the video, and also to capture long-range dependency information in the video, such as a long-duration action, like exercising or dancing in the video.

[0129] Understandably, using a 3*3*3 dilated convolution with a scaling factor of 3 is equivalent to a regular convolution kernel with a receptive field of 7, and effectively reduces the dramatic increase in parameters and computation caused by large convolution kernels.

[0130] In some embodiments, such as Figure 5 As shown, the input of the SK-Inc module is denoted as x. l Let x be the feature map obtained from the previous layer module, and the feature map output after passing through the convolutional attention module. l+1 Input x l After passing through 4 different convolution kernels, we obtain U1, U2, U3, and U4, and the calculation formula is as follows (2):

[0131] U i =Conv i (x l i = 1,...,4 (2);

[0132] Then, the convolution results of multiple convolution kernels are aggregated to obtain the result U. The aggregation method is element-wise summation, and the calculation formula is as follows (3):

[0133] U = U1 + U2 + U3 + U4 (3);

[0134] In order to adaptively aggregate weights for convolutional kernels of different sizes, global average pooling is first applied to U to obtain global information s, that is, to reduce U to a size of H×W. Taking the i-th element as an example, the calculation formula is as shown in equation (4):

[0135]

[0136] Among them, F gp This is for global pooling. Then, feature compression is performed on the global information s to obtain z, and the calculation formula is as shown in equation (5):

[0137] z = F fc (s)=δ(β(Ws)) (5);

[0138] Among them, F fc Let W represent the fully connected layer, W be the network parameters that the video classification model needs to learn, δ represent the ReLU activation function, and β represent the normalization operation. Then, cross-channel soft attention is applied to the compressed feature z, i.e., the softmax operator is applied to each channel, and its calculation formula is shown in equation (6):

[0139]

[0140] Among them, a c ,b c ,c c ,d c These are the weights corresponding to the four convolution kernels, and equation (6) satisfies a c +b c +c c +d c =1, A, B, C, D are the parameters that the network needs to learn, and the final third feature map is obtained by assigning attention weights to each convolutional kernel. The value at position c in the third feature map is... It can be expressed by the following formula (7):

[0141]

[0142] Among them, U 1c This represents the value at the c-th position of U1.

[0143] It is understood that, in the embodiments of this application, by setting different convolution kernels in the SK-Inc module, the SK-I3D video classification model can combine convolution kernels of different sizes to extract features from target objects of different scales in the original video. This makes the SK-I3D video classification model insensitive to the size scaling after video preprocessing, and also reduces the noise interference caused by scaling of the same target object at different time points in the video, thereby improving the robustness and accuracy of the SK-I3D video model.

[0144] S404. Based on the error loss values ​​corresponding to each original video file, perform backpropagation training on the SK-I3D video classification model to obtain the trained SK-I3D video classification model.

[0145] In some embodiments, the batch size during training can be set to 64 frames. After training with 2 epochs of data, the error loss value can be calculated using the cross-entropy loss function, i.e., formula (1). After obtaining the prediction results corresponding to each original video file, the error loss value can be calculated based on the prediction results corresponding to each original video file, the category labels corresponding to each original video file, and the cross-entropy loss function. Then, the prediction result is fed back to the SK-I3D video classification model to adjust the network parameters in the SK-I3D video classification model. When the error loss value is less than the preset threshold, the training of the network parameters is stopped, and the trained SK-I3D video classification model is obtained.

[0146] S405. Preprocess at least one predicted video file and obtain at least one key image frame (preprocessed target image frame) corresponding to each predicted video file.

[0147] In some embodiments, the processing of the predicted video file in step S405 is similar to the processing of the original video file in step S402, and will not be described again here.

[0148] S406. Use the trained SK-I3D video classification model to classify at least one key image frame corresponding to each predicted video file to obtain the classification result corresponding to each predicted video file.

[0149] In one embodiment, after obtaining the trained SK-I3D video classification model and at least one key image frame corresponding to each predicted video file, the at least one key image frame corresponding to each predicted video file can be input into the trained video classification model. The trained SK-I3D video classification model is used to classify the key image frames corresponding to each predicted video file to obtain the classification probability of each predicted video file belonging to different categories. The category with the highest probability is taken as the classification category of the video file to be processed, thereby obtaining the classification result of each predicted video file.

[0150] It is understood that the video classification method based on the SK-I3D model provided in this application is an improvement on the I3D network, removing the optical flow branch and retaining only the RGB branch, which greatly shortens the video classification time. Simultaneously, this application proposes an SK-Inc module, which can simultaneously perform attention mechanisms on both channels and convolutional kernels. The SK-Inc block is an improvement on SKNet (Selective Kernel Network), introducing four different sized convolutional kernels to capture key targets at different scales in the video. To avoid the dramatic increase in computation and parameters caused by large-scale convolutional kernels, and to extract long-range dependency information from the video, dilated convolutions are used instead of large convolutional kernels. By introducing SKNet to perform attention mechanisms on each channel and convolutional kernel, it can better handle issues such as scale variations of key targets in the video, inconsistent video resolution, and the duration of actions in the video. Using the SK-Inc block, it can better adapt to feature extraction from videos in more scenarios, improving the accuracy and speed of video classification.

[0151] This application also provides a video classification device. Figure 6 This is a schematic diagram of the composition structure of a video classification device provided in an embodiment of this application, as shown below. Figure 6 As shown, the video classification device 500 includes:

[0152] The first acquisition module 501 is used to acquire at least one target image frame corresponding to the video file to be processed, and to preprocess the at least one target image frame to obtain at least one preprocessed target image frame.

[0153] The second acquisition module 502 is used to acquire a trained video classification model. The trained video classification model includes at least a convolutional attention module. The convolutional attention module includes multiple convolutional kernels. The weight of the convolutional kernel that matches the size of the target object in the preprocessed target image frame is greater than the weight of the other convolutional kernels.

[0154] Processing module 503 is used to classify the at least one preprocessed target image frame using the trained video classification model to obtain the classification result of the video file to be processed.

[0155] Output module 504 is used to output the classification result.

[0156] It should be noted that the description of the video classification device in this application is similar to the description of the method embodiment described above, and has similar beneficial effects as the method embodiment; therefore, it will not be repeated. For technical details not disclosed in this device embodiment, please refer to the description of the method embodiment in this application for understanding.

[0157] It should be noted that, in the embodiments of this application, if the above-described control method is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiments of this application, or the part that contributes to the related technology, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), magnetic disks, or optical disks. Thus, the embodiments of this application are not limited to any specific hardware and software combination.

[0158] Accordingly, embodiments of this application provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the video classification method provided in the above embodiments.

[0159] This application also provides a video classification device. Figure 7 This is a schematic diagram of the composition structure of a video classification device provided in an embodiment of this application, as shown below. Figure 7 As shown, the video classification device 600 includes: a memory 601, a processor 602, a communication interface 603, and a communication bus 604. The memory 601 stores executable video classification instructions; the processor 602, when executing the executable video classification instructions stored in the memory, implements the video classification method provided in the above embodiments.

[0160] The descriptions of the video classification device and storage medium embodiments above are similar to those of the method embodiments above, and have similar beneficial effects. For technical details not disclosed in the video classification device and storage medium embodiments of this application, please refer to the descriptions of the method embodiments of this application for understanding.

[0161] It should be understood that the phrases "some embodiments" or "one embodiment" throughout the specification mean that a specific feature, structure, or characteristic related to an embodiment is included in at least one embodiment of this application. Therefore, "some embodiments" or "one embodiment" appearing throughout the specification do not necessarily refer to the same embodiment. Furthermore, these specific features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. It should be understood that in the various embodiments of this application, the sequence numbers of the above-described processes do not imply a sequential order of execution; the execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application. The sequence numbers of the above-described embodiments are merely descriptive and do not represent the superiority or inferiority of the embodiments.

[0162] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0163] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods, such as: multiple units or components can be combined, or integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the various components shown or discussed can be through some interfaces, and the indirect coupling or communication connection between devices or units can be electrical, mechanical, or other forms.

[0164] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units. They may be located in one place or distributed across multiple network units. Some or all of the units may be selected to achieve the purpose of this embodiment according to actual needs.

[0165] In addition, each functional unit in the various embodiments of this application can be integrated into one processing unit, or each unit can be a separate unit, or two or more units can be integrated into one unit; the integrated unit can be implemented in hardware or in the form of hardware plus software functional units.

[0166] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various media that can store program code, such as mobile storage devices, ROMs, magnetic disks, or optical disks.

[0167] Alternatively, if the integrated units described above are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the embodiments of this application, or the parts that contribute to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a product to execute all or part of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, ROMs, magnetic disks, or optical disks.

[0168] The above description is merely an embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.< / out> < / fps>

Claims

1. A video classification method, characterized in that, include: Obtain at least one target image frame corresponding to the video file to be processed, and preprocess the at least one target image frame to obtain at least one preprocessed target image frame; A trained video classification model is obtained, which includes at least a convolutional attention module. This convolutional attention module includes one 1×1×1 convolutional kernel, one 3×3×3 convolutional kernel, one 3×3×3 dilated convolution with a scaling factor D of 2, and one 3×3×3 dilated convolution with a scaling factor D of 3. The weight of the convolutional kernel matching the size of the target object in the preprocessed target image frame is greater than the weight of the other convolutional kernels. The video classification model includes an SK-I3D video classification model, which is a model that removes the optical flow branch from the I3D network and retains only the RGB branch. The 3×3×3 dilated convolution with a scaling factor D of 3 is used to capture long-range dependency information in the video. The SK-I3D video classification model includes nine of the aforementioned convolutional attention modules, four convolutional modules, four max-pooling modules, and one average pooling module. The trained video classification model is used to classify the at least one preprocessed target image frame to obtain the classification result of the video file to be processed. Output the classification results.

2. The method according to claim 1, characterized in that, The trained video classification model also includes a convolution module, a pooling module, and a classification module; The step of classifying the at least one preprocessed target image frame using the trained video classification model to obtain the classification result of the video file to be processed includes: Using the convolution module in the trained video classification model, the at least one preprocessed target image frame is convolved to obtain at least one first feature map after convolution processing. The pooling module is used to perform pooling processing on the at least one first feature map to obtain at least one second feature map after pooling processing; The convolutional attention module is used to perform convolution and aggregation processing on the at least one second feature map to obtain at least one third feature map; The classification module is used to determine the classification result of the video file corresponding to the at least one third feature map.

3. The method according to claim 2, characterized in that, The convolutional attention module includes M first convolutional kernels and N second convolutional kernels, wherein the second convolutional kernels are dilated convolutional kernels; The step of using the convolutional attention module to perform convolutional and aggregation processing on the at least one second feature map to obtain at least one third feature map includes: The first convolution kernel in the convolutional attention module is used to perform convolution processing on the j-th second feature map to obtain M first convolution results corresponding to the j-th second feature map; j=1,2,…,P; P is the total number of feature maps; The j-th second feature map is convolved using each of the second convolution kernels to obtain N second convolution results corresponding to the j-th second feature map; Based on the first weights corresponding to each first convolution kernel and the second weights corresponding to each second convolution kernel, the M first convolution results corresponding to the j-th second feature map and the N second convolution results corresponding to the j-th second feature map are weighted and summed to obtain the j-th third feature map.

4. The method according to claim 1, characterized in that, The method further includes: Obtain a preset video classification model and obtain a set of training image frames corresponding to each training video file; wherein, each training video file corresponds to a category label; The video classification model is used to predict the image frame set corresponding to each training video file to obtain the prediction result corresponding to each training video file. Based on the prediction results and category labels corresponding to each training video file, the video classification model is trained by backpropagation to obtain the trained video classification model.

5. The method according to claim 4, characterized in that, The process of backpropagating the video classification model based on the prediction results and category labels of each training video file to obtain the trained video classification model includes: The prediction results and category labels corresponding to each training video file are fed back to the video classification model to adjust the network parameters in the video classification model. The network parameters include at least the weights corresponding to each first convolutional kernel and each second convolutional kernel in the convolutional attention module. When the conditions for ending training are met, the trained video classification model is obtained.

6. The method according to claim 1, characterized in that, The step of obtaining at least one target image frame corresponding to the video file to be processed includes: The resolution of the video file to be processed is adjusted based on the preset resolution to obtain the first video file; The frame rate of the first video file is adjusted based on the first preset frame rate to obtain the second video file; Image frames are extracted from the second video file at first preset intervals to obtain multiple extracted image frames; Obtain the number S of the extracted image frames; If the number of image frames S is greater than or equal to a preset frame number threshold Q, the extracted multiple image frames are determined as at least one target image frame corresponding to the second video file.

7. The method according to claim 6, characterized in that, The method further includes: If the number of image frames S is less than the frame number threshold Q, obtain the last image frame extracted from the second video file; The extracted multiple image frames and (QS) last image frames are determined as at least one target image frame corresponding to the second video file.

8. A video classification device, characterized in that, include: The first acquisition module is used to acquire at least one target image frame corresponding to the video file to be processed, and to preprocess the at least one target image frame to obtain at least one preprocessed target image frame. The second acquisition module is used to acquire a trained video classification model. The trained video classification model includes at least a convolutional attention module, which comprises one 1×1×1 convolutional kernel, one 3×3×3 convolutional kernel, one 3×3×3 dilated convolution with a scaling factor D of 2, and one 3×3×3 dilated convolution with a scaling factor D of 3. The weight of the convolutional kernel matching the size of the target object in the preprocessed target image frame is greater than the weight of the other convolutional kernels. The video classification model includes an SK-I3D video classification model, which is a model that removes the optical flow branch from the I3D network and retains only the RGB branch. The 3×3×3 dilated convolution with a scaling factor D of 3 is used to capture long-range dependency information in the video. The SK-I3D video classification model includes nine of the aforementioned convolutional attention modules, four convolutional modules, four max-pooling modules, and one average pooling module. The processing module is used to classify the at least one preprocessed target image frame using the trained video classification model to obtain the classification result of the video file to be processed. The output module is used to output the classification results.

9. A video classification device, characterized in that, include: Memory, used to store executable video classification instructions; A processor, when executing an executable video classification stored in the memory, implements the method of any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The system stores video classification instructions for inducing the processor to execute the method as described in any one of claims 1 to 7.

Citation Information

Patent Citations

  • A Single-Label Classification Method for Short Videos Based on Multimodal Feature Fusion Using Deep Networks

    CN111460223B

  • Action recognition method based on double-flow convolution attention

    CN112926396A

  • Action recognition method for high-time-sequence 3D neural network based on cavity convolution

    CN110334589A

  • Orthopedic rehabilitation information processing system based on CEST orthopedic rehabilitation clinical system

    CN113345578A