A video scene multi-dimensional classification method, device, equipment and storage medium

Abstract

The application discloses a video scene multi-dimension classification method and device, equipment and a storage medium. The technical scheme provided by the application comprises: obtaining a video to be classified, extracting a plurality of video frames from the video to be classified to form a frame sequence; inputting the frame sequence into a pre-trained multi-dimension scene classification model to obtain a plurality of scene categories output by the multi-dimension scene classification model; wherein the multi-dimension scene classification model is obtained by training a multi-task network based on a training sample set, and the loss weight of each task is adaptively adjusted in the training process of the multi-task network; and the plurality of scene categories output by the multi-dimension scene classification model are determined as the scene categories of the video to be classified. Through the above technical scheme, the resource cost of running the multi-dimension scene classification model is saved, the application of the multi-dimension scene classification model in actual business is promoted, and the problem that the multi-dimension scene classification model is difficult to be applied to actual business in the related art is solved.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a method, apparatus, device and storage medium for multi-dimensional classification of video scenes. Background Technology

[0002] The mobile internet era has made video a crucial medium for interaction, and video information has become an indispensable part of daily life. With the rapid development of big data technology, some video platforms can analyze the video content people watch to determine their preferences for various video types, and then recommend videos in categories with high user engagement. Video content is diverse and complex, and video scene classification is fundamental to video content analysis. However, classifying video scenes from only one dimension often fails to fully express the semantic content of the video. For example, a video can be categorized as indoor or outdoor, as well as as dancing or non-dancing scenes based on the subject's behavior, or as bright or dark scenes based on lighting intensity, and so on. Therefore, classifying video scenes from multiple dimensions can fully express the semantic content of the video.

[0003] Current multi-dimensional video scene classification methods often use a single deep learning model to identify one dimension of the video scene. Therefore, multiple deep learning models need to be deployed to identify multiple dimensions of video scenes. However, due to the large computational cost of multiple deep learning models, the resource cost of multi-dimensional scene classification models is too high during operation, making it difficult to successfully apply them to real-world business applications. Summary of the Invention

[0004] This application provides a method, apparatus, device, and storage medium for multi-dimensional classification of video scenes. It deploys a lightweight model based on multi-task learning to save model computing power, reduce the resource cost of running the multi-dimensional scene classification model, promote the application of the multi-dimensional scene classification model in actual business, and solve the problem that multi-dimensional scene classification models in related technologies are difficult to apply to actual business.

[0005] Firstly, this application provides a multi-dimensional classification method for video scenes, including:

[0006] Obtain the video to be classified, and extract multiple video frames from the video to be classified to form a frame sequence;

[0007] The frame sequence is input into a pre-trained multi-dimensional scene classification model to obtain multiple scene categories output by the multi-dimensional scene classification model; wherein, the multi-dimensional scene classification model is trained on a multi-task network based on a training sample set, and the loss weights of each task are adaptively adjusted during the training of the multi-task network.

[0008] The multiple scene categories output by the multi-dimensional scene classification model are determined as the scene categories of the video to be classified.

[0009] Secondly, this application provides a video scene multi-dimensional classification device, including:

[0010] The frame sequence acquisition module is configured to acquire the video to be classified and extract multiple video frames from the video to be classified to form a frame sequence.

[0011] The classification model prediction module is configured to input the frame sequence into a pre-trained multi-dimensional scene classification model to obtain multiple scene categories output by the multi-dimensional scene classification model; wherein, the multi-dimensional scene classification model is trained on a multi-task network based on a training sample set, and the loss weights of each task are adaptively adjusted during the training of the multi-task network.

[0012] The scene category determination module is configured to determine the scene categories of the video to be classified from the multiple scene categories output by the multi-dimensional scene classification model.

[0013] Thirdly, this application provides a video scene multi-dimensional classification device, including: a memory and one or more processors;

[0014] The memory is used to store one or more programs;

[0015] When the one or more programs are executed by the one or more processors, the one or more processors implement the video scene multi-dimensional classification method as described in the first aspect.

[0016] Fourthly, this application provides a computer-readable storage medium storing a program that, when executed by a processor, implements the video scene multi-dimensional classification method as described in the first aspect.

[0017] Fifthly, this application provides a computer program product comprising a computer program stored in a computer-readable storage medium, wherein at least one processor of the device reads from the computer-readable storage medium and executes the computer program, thereby enabling the device to implement the video scene multi-dimensional classification method as described in the first aspect.

[0018] This application pre-trains a multi-task network using a training sample set to obtain a multi-dimensional scene classification model, and adaptively adjusts the loss weights of each task during the training process. Multiple video frames are extracted from the video to be classified and assembled into a frame sequence. This frame sequence is input into the multi-dimensional scene classification model. The backbone network of the multi-dimensional scene classification model extracts spatial and temporal feature information from the frame sequence. Based on the spatial and temporal feature information in the frame sequence, multiple classification heads in the multi-dimensional scene classification model predict the scene category of the video to be classified. Through the above techniques, the multi-task network combines multiple scene classification tasks together, using a multi-task network as the multi-dimensional scene classification model to perform multi-dimensional classification of video scenes using a single model. This saves the resource cost of running the multi-dimensional scene classification model and promotes its application in practical business. During the training of the multi-task network, multiple scene classification tasks are trained together, and the feature representations learned by each task are shared. The tasks mutually promote learning, improving the generalization effect and training efficiency of the multi-dimensional scene classification model and ensuring its classification accuracy. During training, the loss weights for each task are adaptively adjusted based on the learning progress of each task, balancing the learning speed of each task and improving the training effect of the multi-task network. Video frame sequences include spatial feature information of video frames and temporal feature information between video frames. Using video frame sequences as input to the multi-dimensional scene classification model enriches the video feature information that the model can extract, thereby improving the classification accuracy of the multi-dimensional scene classification model. Attached Figure Description

[0019] Figure 1 This is a flowchart of a video scene multi-dimensional classification method provided in an embodiment of this application;

[0020] Figure 2 This is a schematic diagram of the structure of the multi-task network provided in the embodiments of this application;

[0021] Figure 3 This is a flowchart of the model training process for adaptively adjusting loss weights provided in the embodiments of this application;

[0022] Figure 4 This is a flowchart illustrating scene classification of sample frame sequences using a multi-task network provided in this application embodiment;

[0023] Figure 5 This is a flowchart of the training process of the multi-task network provided in the embodiments of this application, which involves targeted and then unified training.

[0024] Figure 6 This is a flowchart of training a basic scene classification task provided in an embodiment of this application;

[0025] Figure 7 This is a schematic diagram of any video frame in the frame sequence provided in the embodiments of this application;

[0026] Figure 8 This is a schematic diagram of another video scene multi-dimensional classification device provided in an embodiment of this application;

[0027] Figure 9 This is a structural schematic diagram of a video scene multi-dimensional classification device provided in an embodiment of this application. Detailed Implementation

[0028] To make the objectives, technical solutions, and advantages of this application clearer, specific embodiments of this application will be described in further detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are merely for explaining this application and not for limiting it. It should also be noted that, for ease of description, only the parts relevant to this application are shown in the drawings, not all of them. Before discussing exemplary embodiments in more detail, it should be mentioned that some exemplary embodiments are described as processes or methods depicted as flowcharts. Although the flowcharts describe operations (or steps) as sequential processes, many of these operations can be performed in parallel, concurrently, or simultaneously. Furthermore, the order of the operations can be rearranged. The process can be terminated when its operation is completed, but may also have additional steps not included in the drawings. The process can correspond to a method, function, procedure, subroutine, subprogram, etc.

[0029] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such use of data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class and the number of objects is not limited; for example, a first object can be one or more. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0030] The video scene multi-dimensional classification method provided in this embodiment can be executed by a video scene multi-dimensional classification device. This device can be implemented through software and / or hardware, and can consist of two or more physical entities, or a single physical entity. In this embodiment, the video scene multi-dimensional classification device can be a computer.

[0031] In one embodiment, video scenes can be classified from more granular dimensions such as basic environment, lighting environment, motion level, text environment, and broadcaster behavior to fully express the semantic content of the video. To classify video scenes from multiple dimensions, a deep learning model can be used to identify a video scene in one dimension, and then multiple deep learning models can be deployed and combined to achieve multi-dimensional classification of video scenes. Although multiple deep learning models can achieve multi-dimensional scene classification, the computational cost of multiple deep learning models is large, and their runtime resource consumption is high, making them difficult to deploy on computers and thus difficult to successfully apply in actual business applications.

[0032] To address the aforementioned issues, this application provides a method for multi-dimensional classification of video scenes, thereby saving resource costs associated with running multi-dimensional scene classification models and promoting their application in practical business scenarios.

[0033] Figure 1 This is a flowchart illustrating a multi-dimensional video scene classification method provided in an embodiment of this application. The following description uses a live streaming platform as an example to illustrate the implementation of this method. (Reference) Figure 1 The multi-dimensional classification method for this video scene includes:

[0034] S110. Obtain the video to be classified, and extract multiple video frames from the video to be classified to form a frame sequence.

[0035] The video to be classified refers to the video used for multi-dimensional scene classification. The video to be classified consists of multiple video frames arranged in timestamp order. Multiple video frames can be extracted from the video to be classified and arranged in timestamp order to form a frame sequence, so that the frame sequence includes not only the spatial feature information of the video frames, but also the temporal feature information between the video frames.

[0036] S120. Input the frame sequence into the pre-trained multi-dimensional scene classification model to obtain multiple scene categories output by the multi-dimensional scene classification model; wherein, the multi-dimensional scene classification model is trained on the multi-task network based on the training sample set, and the loss weights of each task are adaptively adjusted during the training process of the multi-task network.

[0037] The multi-dimensional scene classification model refers to a model used to classify videos into multiple scenes. In this embodiment, a multi-task network is used as the multi-dimensional scene classification model. A multi-task network is a network model capable of multi-task learning. For example, Figure 2 This is a schematic diagram of the structure of the multi-task network provided in an embodiment of this application. For example... Figure 2As shown, the multi-task network consists of a backbone network and at least two classification heads. The output of the backbone network serves as the input to each classification head. The backbone network is a convolutional layer used to extract feature information from the input data, and the classification heads are used to predict the category of the corresponding scene based on the feature information output by the backbone network. Different classification heads classify scenes of different dimensions. For example, one classification head is used to classify a video as a dark scene or a bright scene, while another classification head is used to classify a video as a dancing scene or a non-dancing scene. In the multi-task network, multiple classification heads share a single backbone network; that is, a classification head and the backbone network can form a network for classifying one-dimensional scenes. Therefore, the multi-task network can combine classification tasks for multiple scenes into a single network model, thereby enabling a single network model to classify multi-dimensional scenes. Compared to existing technologies that use multiple single-task networks to achieve multi-dimensional scene classification of videos, this embodiment uses a single multi-task network to achieve multi-dimensional scene classification of videos, saving resource costs for running multi-dimensional scene classification models and promoting the application of multi-dimensional scene classification models in practical business applications.

[0038] When training a multi-task network based on a training sample set, multiple classification tasks within the network are combined for training. By sharing the feature representations learned from different classification tasks, these tasks mutually promote each other's learning. These multiple classification tasks exhibit both correlation and difference. Correlation allows the model's classification head to focus on common feature representations across the tasks, improving learning efficiency. Difference allows the classification head to learn feature representations that are difficult to learn in a single-task network from other classification heads. Furthermore, other classification heads can be used to determine the effectiveness of the feature representations learned by the current classification head, effectively improving the model's generalization and performance. Based on the rich feature representations learned, the multi-dimensional scene classification model can accurately predict the multi-dimensional scene categories of videos, ensuring high classification accuracy.

[0039] refer to Figure 2This embodiment describes a multi-task network comprising five classification heads. The third classification head and the backbone network form a network for basic scene classification tasks. The third classification head can output the category of the basic scene. In this embodiment, the basic scene includes more than ten categories. The fourth classification head and the backbone network form a network for classifying whether a video is a dark scene. The fourth classification head can output two scene categories: the video is a dark scene or the video is not a dark scene. The fifth classification head and the backbone network form a network for classifying whether a video is a static scene. The fifth classification head can output two scene categories: the video is a static scene or the video is not a static scene. The sixth classification head and the backbone network form a network for classifying whether a video is a scene with a lot of text coverage. The sixth classification head can output two scene categories: the video is a scene with a lot of text coverage or the video is not a scene with a lot of text coverage. The seventh classification head and the backbone network form a network for classifying broadcaster behavior scenes. The seventh classification head can output the category of broadcaster behavior scenes. In this embodiment, broadcaster behavior scenes include several categories. In this embodiment, the training sample set includes multiple sample frame sequences, each sample frame sequence is labeled with the scene labels of these five dimensions. The sample frame sequence is input into the multi-task network, and the category of the corresponding scene is obtained from the output of each classification head in the multi-task network. Based on the category output by the classification head and the label of the scene corresponding to the sample frame sequence, the loss value of the classification task for the corresponding scene can be calculated, and the model parameters of the multi-task network can be optimized based on the loss value.

[0040] In one embodiment, the loss functions of all classification tasks can be summed to obtain the total loss function of the multi-task network, which is then used to optimize the model parameters of the multi-task network. However, the magnitudes of the loss values ​​for different classification tasks may differ. Directly summing the loss functions of classification tasks can easily lead to the learning of multiple classification tasks being dominated by one particular task, causing the model to tend to fit the dominant task and negatively impacting the learning of other classification tasks, resulting in poorer learning performance. Therefore, to make simple adjustments to the loss functions of classification tasks, a weight coefficient is configured for the loss function of each classification task. Then, the total loss function is equal to the weighted sum of the loss functions of each classification task and their corresponding weight coefficients. The weight coefficients of each classification task can be pre-set according to their importance, so that the loss functions are weighted and summed with fixed weights throughout the training cycle. However, the learning difficulty of different classification tasks varies, and the convergence speed of different classification heads also differs. At the same time during training, different classification tasks may be at different learning stages; for example, the fourth classification head may be close to convergence while the third classification head is not yet trained. Imbalance in the learning phases of a classification task can hinder model training. Therefore, a weight strategy with a fixed loss function may limit the learning performance of the classification task at a certain training phase, resulting in a deterioration in the learning performance of the classification task.

[0041] To address this, this embodiment proposes that during the training process of a multi-task network, the weight coefficients of each classification task are adaptively adjusted based on the learning difficulty, progress, and effectiveness of each task. This balances the learning stages of each task, improves the learning effectiveness of each task, and enhances the training efficiency of the model. For example, if the loss value of a certain classification task is large (large gradient magnitude) or the training speed of a certain task is too fast, the weight coefficient of that task can be appropriately reduced. This allows classification tasks with smaller loss values ​​(smaller gradient magnitude) or slower training speeds to have a greater impact on model optimization, thereby improving the learning effectiveness of those tasks.

[0042] In one embodiment, Figure 3 This is a flowchart of the model training process for adaptively adjusting loss weights provided in an embodiment of this application. Figure 3 As shown, the specific steps of the model training process that adaptively adjusts the loss weights include S210-S240:

[0043] S210. Input the sample frame sequence from the training sample set into the multi-task network to obtain the scene category information output by each classification head in the multi-task network.

[0044] For example, Figure 4 This is a flowchart illustrating scene classification of sample frame sequences using a multi-task network provided in this application embodiment. For example... Figure 4 As shown, the steps for scene classification of the sample frame sequence by the multi-task network specifically include S2101-S2102:

[0045] S2101. Input the sample frame sequence into the backbone network to obtain the spatial feature information and temporal feature information extracted by the backbone network from the sample frame sequence.

[0046] S2102. Input spatial feature information and temporal feature information into each classification head to obtain scene category information of the sample frame sequence output by each classification head.

[0047] Scene category information refers to the category of the scene corresponding to the sample frame sequence predicted by the classification head. For example, the scene category information output by the fourth classification head is "dark scene" or "not a dark scene." The fourth classification head determines the category of the scene corresponding to the video frame sequence as "dark scene" or "not a dark scene" based on the features of the video frame sequence. (Reference) Figure 2 The sample frame sequence is input into the backbone network, which extracts spatial and temporal feature information from the sample frame sequence. This spatial and temporal feature information is then input together into the third, fourth, fifth, sixth, and seventh classification heads to obtain the corresponding scene categories output by these five classification heads. In one embodiment, the backbone network can be a convolutional neural network, such as MobileNet V3, and the classification heads can be fully connected layers.

[0048] S220. Determine the loss value of the corresponding classification head based on the scene category information and the scene label information of the corresponding sample frame sequence.

[0049] The scene label information refers to the labels for each dimension of the scene marked on the sample frame sequence. For example, the label for whether a sample frame sequence is a dark scene is "dark scene". For instance, the category of "dark scene" output by the fourth classification head is compared with the label of "dark scene" in the sample frame sequence to obtain the loss value of the fourth classification head. The process for determining the loss values ​​of other classification heads is the same.

[0050] S230. Determine the weight coefficient of the classification head based on the classification accuracy of the classification head, and determine the total loss value based on the loss value and weight coefficient of each classification head.

[0051] For example, a higher classification accuracy of a classification head indicates that the learning difficulty of the corresponding classification task is lower and the training speed is faster in the current training stage. Therefore, the weight coefficient of the classification head can be reduced to decrease the impact of the classification task on model optimization. Conversely, a lower classification accuracy of a classification head indicates that the learning difficulty of the corresponding classification task is higher and the training speed is slower in the current training stage. Therefore, the weight coefficient of the classification head can be increased to increase the impact of the classification task on model optimization.

[0052] In one embodiment, the classification accuracy of each classifier is determined based on a validation sample set, and the weight coefficient of the corresponding classifier is determined based on the classification accuracy. For example, each sample frame sequence from the validation sample set is input into the multi-task network during the current training phase to obtain the category of the corresponding scene output by each classifier in the multi-task network. It is determined whether the category of the corresponding scene output by each classifier is the same as the label of the corresponding scene in the sample frame sequence. When the output category and label are the same, the classifier is determined to have correctly classified the sample frame sequence. The number of times each classifier correctly classifies is counted, and the percentage of this number to the number of samples in the validation sample set is determined as the classification accuracy of that classifier during the current training phase.

[0053] In this embodiment, the classification accuracy is substituted into a preset weight coefficient calculation formula to obtain the weight coefficients of the classification head. The loss function of the classification head is then weighted and summed based on its weight coefficients to obtain the total loss function. For example, the expression for the total loss function is as follows:

[0054] L=∑ i w i *L i

[0055] The formula for calculating the weighting coefficient is as follows:

[0056] wi (t)=-(1-p i )log(p i )

[0057] Among them, L i Let w be the loss function for the i-th classifier head. i Let p be the weight coefficient of the i-th classifier head, L be the total loss function, and p be the weight coefficient of the i-th classifier head. i Let be the classification accuracy of the i-th classification head.

[0058] S240. Adjust the model parameters of the multi-task network based on the total loss value.

[0059] For example, the model parameters of the backbone network and the classifier are adjusted by the total loss value until the backbone network and the classifier converge or the number of training iterations reaches a threshold.

[0060] As discussed above, during the training phase of a multi-task network, the backbone network and all classification heads learn simultaneously using the training sample set, meaning multiple classification tasks begin learning concurrently. However, for some more challenging classification tasks, such as the basic scene classification task mentioned above which has more than ten categories, and the dark scene classification task which only has two categories, the basic scene classification task has several times more categories than other classification tasks, making its training significantly more difficult. Therefore, even with increased weights for the basic scene classification task, its convergence speed is still much slower than other classification tasks, making it difficult to balance the learning progress of this task with that of other classification tasks.

[0061] To address this, this embodiment employs a training strategy of first selectively training the most complex classification task and then uniformly training other classification tasks to ensure the model's learning effectiveness. For example, Figure 5 This is a flowchart illustrating the targeted-then-unified training process of the multi-task network provided in this application embodiment. For example... Figure 5 As shown, the specific steps of the multi-task network's training process, which involves targeting specific tasks before unifying them, include S310-S320:

[0062] S310. Train the backbone network and the first classification head in the multi-task network based on the training sample set. The first classification head is used to determine the category of the first scene in the sample frame sequence. The first scene is the scene with the most label categories in the training sample set.

[0063] Here, the first classification head refers to the classification head of the most complex classification task in the multi-task network, and the first scene is the scene corresponding to the most complex classification task. If the first classification head is the classification head of the basic scene classification task, then the first scene is the basic scene. For example, the basic scene classification task has the most categories among the five classification tasks mentioned above. Correspondingly, the sample frame sequences in the training sample set correspond to the basic scene with the most label categories, reaching more than ten categories. Therefore, based on the label categories of each sample frame sequence in the training sample set, the scene with the most label categories can be determined as the basic scene, and the basic scene classification task can be determined as the classification task to be used for training.

[0064] In the targeted training phase of the multi-task network, one classifier head is randomly selected as the first classifier head for priority training. (Reference) Figure 2 In this embodiment, the third classification head in the multi-task network is used as the first classification head, i.e., it is trained specifically for the basic scene classification task. For example, a sample frame sequence is input into the backbone network to obtain spatial and temporal feature information output by the backbone network. The spatial and temporal feature information is then input into the third classification head to obtain the category of the basic scene output by the third classification head. Based on the category of the basic scene and the label of the basic scene in the sample frame sequence, the loss value of the third classification head is determined. The model parameters of the third classification head and the backbone network are optimized based on this loss value to train the classification ability of the third classification head for basic scenes and the feature extraction ability of the backbone network for frame sequences.

[0065] In one embodiment, the labels of the training sample set for the classification task must cover all categories of the corresponding scene, and the number of samples for each category must be sufficient. Therefore, the training samples required for the basic scene classification task need to be more abundant. To this end, this embodiment obtains open-source datasets and business datasets as training sample sets to meet the sample requirements of the basic scene classification task. For example, Figure 6 This is a flowchart illustrating the training process for a basic scene classification task provided in an embodiment of this application. For example... Figure 6 As shown, the specific steps for training the basic scene classification task include S3101-S3102:

[0066] S3101, train the backbone network and the first classification head based on the open source dataset.

[0067] S3102. Train the backbone network and the first classification head based on the business dataset.

[0068] For example, the open-source dataset contains a large number of sample frame sequences. The backbone network and the first classifier are pre-trained using these sample frame sequences, enabling them to develop good feature extraction and scene classification capabilities on large-scale datasets. The business dataset consists of training samples generated from videos collected in real-world business scenarios. The pre-trained backbone network and the first classifier are trained using this business dataset, promoting their adaptation to real-world business usage scenarios.

[0069] S320. After the backbone network and the first classification head have been trained, the second classification head in the multi-task network is trained based on the training sample set.

[0070] The second classification header refers to the classification headers in a multi-task network that are other than the first classification header, such as... Figure 2 The fourth, fifth, sixth, and seventh classification heads in this embodiment are all second classification heads. For example, the classification accuracy of the basic scene classification task can be determined by validating the sample set. When the classification accuracy of the basic scene classification task reaches a preset accuracy threshold, it is determined that the basic scene classification task has completed targeted training. After the targeted training of the basic scene classification task is completed, other classification tasks are trained with uniformly adaptively adjusted weight parameters.

[0071] During the training phase of the second classification head, sample frame sequences are input into the trained backbone network to obtain spatial and temporal feature information output by the backbone network. This spatial and temporal feature information is then input into each second classification head to obtain the corresponding scene category output by the second classification head. Based on the scene category output by the second classification head and the label of the scene corresponding to the sample frame sequence, the loss value of the corresponding second classification head is determined. The classification accuracy of the second classification head in the current training phase is determined using a validation sample set, and the weight coefficients of the second classification head are determined based on the classification accuracy. The loss values ​​of the second classification heads are then weighted and summed according to their loss coefficients to obtain the total loss value. The model parameters of the second classification head are adjusted based on the total loss value.

[0072] In this embodiment, after the first classification head and backbone network are trained on a specific basis using an open-source dataset and a business dataset, the second classification head is trained on the business dataset, so that the second classification head is adapted to the actual business usage scenario.

[0073] After the multi-task network completes training, a multi-dimensional scene classification model is obtained. When performing multi-dimensional scene classification on the video to be classified based on this model, the frame sequence of the video is used as input. The backbone network of the multi-dimensional scene classification model extracts spatial and temporal feature information from the frame sequence, and uses this information as input to each classification head. Each classification head then outputs the scene category of the corresponding frame sequence based on the spatial and temporal feature information.

[0074] S130. Determine the scene categories of the video to be classified from the multiple scene categories output by the multi-dimensional scene classification model.

[0075] For example, a classification head outputs a scene category in one dimension, while multiple classification heads of a multi-dimensional scene classification model output scene categories in multiple dimensions. These multiple-dimensional scene categories are then determined as the multi-dimensional scene categories of the video to be classified, thus enabling a single model to classify multi-dimensional scenes.

[0076] Figure 7 This is a schematic diagram of any video frame in the frame sequence provided in the embodiments of this application. For example... Figure 7 As shown, this video frame displays a television and a television cabinet; the television is off. Assume that each video frame in the frame sequence is as follows... Figure 7 As shown, the network structure of the multi-dimensional scene classification model is as follows: Figure 2 As shown, when the frame sequence is input into the multi-dimensional scene classification model, the third classification head outputs an indoor scene, the fourth classification head outputs a video that is not a dark scene, the fifth classification head outputs a video that is not a static scene, the sixth classification head outputs a video scene that is not covered by a lot of text, and the seventh classification head outputs a scene without people.

[0077] In summary, the video scene multi-dimensional classification method provided in this application pre-trains a multi-task network using a training sample set to obtain a multi-dimensional scene classification model, and adaptively adjusts the loss weights of each task during the training process of the multi-task network. Multiple video frames are extracted from the video to be classified and assembled into a frame sequence. This frame sequence is input into the multi-dimensional scene classification model. The backbone network of the multi-dimensional scene classification model extracts spatial and temporal feature information from the frame sequence, and multiple classification heads in the multi-dimensional scene classification model predict the scene category of the video to be classified based on the spatial and temporal feature information in the frame sequence. Through the above technical means, the multi-task network combines multiple scene classification tasks together, using the multi-task network as a multi-dimensional scene classification model to perform multi-dimensional classification of video scenes using a single model. This saves the resource cost of running the multi-dimensional scene classification model and promotes its application in practical business. When training the multi-task network, multiple scene classification tasks are trained together, and the feature representations learned by each task are shared. The tasks mutually promote learning, improving the generalization effect and training efficiency of the multi-dimensional scene classification model and ensuring its classification accuracy. During training, the loss weights for each task are adaptively adjusted based on the learning progress of each task, balancing the learning speed of each task and improving the training effect of the multi-task network. Video frame sequences include spatial feature information of video frames and temporal feature information between video frames. Using video frame sequences as input to the multi-dimensional scene classification model enriches the video feature information that the model can extract, thereby improving the classification accuracy of the multi-dimensional scene classification model.

[0078] Based on the above embodiments, Figure 8 This is a schematic diagram of the structure of a video scene multi-dimensional classification device provided in an embodiment of this application. Figure 8 As shown, the video scene multi-dimensional classification device includes a frame sequence acquisition module 41, a classification model prediction module 42, and a scene category determination module 43.

[0079] The frame sequence acquisition module is configured to acquire the video to be classified and extract multiple video frames from the video to be classified to form a frame sequence.

[0080] The classification model prediction module is configured to input the frame sequence into a pre-trained multi-dimensional scene classification model to obtain multiple scene categories output by the multi-dimensional scene classification model. The multi-dimensional scene classification model is trained on a multi-task network based on a training sample set, and the loss weights of each task are adaptively adjusted during the training of the multi-task network.

[0081] The scene category determination module is configured to determine the scene categories of the video to be classified from the multiple scene categories output by the multi-dimensional scene classification model.

[0082] In summary, the video scene multi-dimensional classification device provided in this application pre-trains a multi-task network using a training sample set to obtain a multi-dimensional scene classification model, and adaptively adjusts the loss weights of each task during the training process of the multi-task network. Multiple video frames are extracted from the video to be classified and assembled into a frame sequence. This frame sequence is input into the multi-dimensional scene classification model. The backbone network of the multi-dimensional scene classification model extracts spatial and temporal feature information from the frame sequence, and multiple classification heads in the multi-dimensional scene classification model predict the scene category of the video to be classified based on the spatial and temporal feature information in the frame sequence. Through the above technical means, the multi-task network combines multiple scene classification tasks together, using the multi-task network as a multi-dimensional scene classification model to perform multi-dimensional classification of video scenes using a single model. This saves the resource cost of running the multi-dimensional scene classification model and promotes its application in practical business. When training the multi-task network, multiple scene classification tasks are trained together, and the feature representations learned by each task are shared. The tasks mutually promote learning, improving the generalization effect and training efficiency of the multi-dimensional scene classification model and ensuring its classification accuracy. During training, the loss weights for each task are adaptively adjusted based on the learning progress of each task, balancing the learning speed of each task and improving the training effect of the multi-task network. Video frame sequences include spatial feature information of video frames and temporal feature information between video frames. Using video frame sequences as input to the multi-dimensional scene classification model enriches the video feature information that the model can extract, thereby improving the classification accuracy of the multi-dimensional scene classification model.

[0083] This application also provides a video scene multi-dimensional classification device, which can integrate the video scene multi-dimensional classification apparatus provided in this application. Figure 9 This is a schematic diagram of the structure of a video scene multi-dimensional classification device provided in an embodiment of this application. (Reference) Figure 9 The video scene multi-dimensional classification device includes: an input device 53, an output device 54, a memory 52, and one or more processors 51; the memory 52 is used to store one or more programs; when one or more programs are executed by one or more processors 51, the one or more processors 51 implement the video scene multi-dimensional classification method provided in the above embodiments.

[0084] This application also provides a computer-readable storage medium storing a program that, when executed by a computer processor, performs the video scene multi-dimensional classification method as described in the above embodiments. Of course, the computer-readable storage medium provided in this application may include programs not limited to the video scene multi-dimensional classification method described above, and may also perform related operations within the video scene multi-dimensional classification method provided in any embodiment of this application.

[0085] This application also provides a computer program product. The technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. The computer program product is stored in a storage medium and includes several instructions to cause a computer device, mobile terminal, or processor therein to execute all or part of the steps of the video scene multi-dimensional classification method of various embodiments of this application.

[0086] The video scene multi-dimensional classification apparatus, device, storage medium, and computer program product provided in the above embodiments can execute the video scene multi-dimensional classification method provided in any embodiment of this application, and have corresponding functions and beneficial effects. Technical details not described in detail in the above embodiments can be found in the video scene multi-dimensional classification method provided in any embodiment of this application.

Claims

1. A multi-dimensional classification method for video scenes, characterized in that, include: Obtain the video to be classified, and extract multiple video frames from the video to be classified to form a frame sequence; The frame sequence is input into a pre-trained multi-dimensional scene classification model to obtain multiple scene categories output by the multi-dimensional scene classification model. The multiple scene categories output by the multi-dimensional scene classification model are determined as the scene categories of the video to be classified. The multi-dimensional scene classification model is trained on a multi-task network based on a training sample set. The multi-task network includes a backbone network, a first classification head, and multiple second classification heads. The first classification head corresponds to the scene with the most label categories. The training process of the multi-task network includes: The sample frame sequence from the training sample set is input into the backbone network to obtain spatial and temporal feature information extracted by the backbone network from the sample frame sequence; the spatial and temporal feature information output by the backbone network is input into the first classification head to obtain the category of the first scene output by the first classification head; the loss value of the first classification head is determined according to the category of the first scene and the label of the first scene in the sample frame sequence; the model parameters of the first classification head and the backbone network are optimized according to the loss value. After the first classifier and the backbone network have finished training, the sample frame sequence is input into the trained backbone network to obtain the spatial and temporal feature information extracted by the trained backbone network from the sample frame sequence. The spatial and temporal feature information output by the trained backbone network is then input into each of the second classifiers to obtain the category of the corresponding scene output by each second classifier. The loss value of the second classifier is determined based on the category of the corresponding scene output by the second classifier and the label of the scene corresponding to the sample frame sequence. The weight coefficient of the second classifier is determined based on the learning difficulty, learning progress, and learning effect of the task corresponding to the second classifier. The loss coefficient and loss value of each second classifier are weighted and summed to obtain the total loss value. The model parameters of each second classifier are optimized based on the total loss value.

2. The video scene multi-dimensional classification method according to claim 1, characterized in that, The training sample set includes an open-source dataset and a business dataset. The backbone network and the first classification head are first trained using the open-source dataset, and then trained using the business dataset.

3. The video scene multi-dimensional classification method according to claim 1, characterized in that, The second classification head is trained based on the business dataset.

4. A multi-dimensional video scene classification device, characterized in that, include: The frame sequence acquisition module is configured to acquire the video to be classified and extract multiple video frames from the video to be classified to form a frame sequence. The classification model prediction module is configured to input the frame sequence into a pre-trained multi-dimensional scene classification model to obtain multiple scene categories output by the multi-dimensional scene classification model. The scene category determination module is configured to determine the scene categories of the video to be classified from the multiple scene categories output by the multi-dimensional scene classification model. The multi-dimensional scene classification model is trained on a multi-task network based on a training sample set. The multi-task network includes a backbone network, a first classification head, and multiple second classification heads. The first classification head corresponds to the scene with the most label categories. The video scene multi-dimensional classification device is further configured as follows: The sample frame sequence from the training sample set is input into the backbone network to obtain spatial and temporal feature information extracted by the backbone network from the sample frame sequence; the spatial and temporal feature information output by the backbone network is input into the first classification head to obtain the category of the first scene output by the first classification head; the loss value of the first classification head is determined according to the category of the first scene and the label of the first scene in the sample frame sequence; the model parameters of the first classification head and the backbone network are optimized according to the loss value. After the first classifier and the backbone network have finished training, the sample frame sequence is input into the trained backbone network to obtain the spatial and temporal feature information extracted by the trained backbone network from the sample frame sequence. The spatial and temporal feature information output by the trained backbone network is then input into each of the second classifiers to obtain the category of the corresponding scene output by each second classifier. The loss value of the second classifier is determined based on the category of the corresponding scene output by the second classifier and the label of the scene corresponding to the sample frame sequence. The weight coefficient of the second classifier is determined based on the learning difficulty, learning progress, and learning effect of the task corresponding to the second classifier. The loss coefficient and loss value of each second classifier are weighted and summed to obtain the total loss value. The model parameters of each second classifier are optimized based on the total loss value.

5. A video scene multi-dimensional classification device, characterized in that, include: Memory and one or more processors; The memory is used to store one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the video scene multi-dimensional classification method as described in any one of claims 1-3.

6. A computer-readable storage medium, characterized in that, The storage medium stores a program that, when executed by a processor, implements the video scene multi-dimensional classification method as described in any one of claims 1-3.

7. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the video scene multi-dimensional classification method as described in any one of claims 1-3.

Images