Video classification method and device, electronic equipment and storage medium

Abstract

This application discloses a video classification method, apparatus, electronic device, and storage medium, belonging to the field of artificial intelligence technology. The method includes: acquiring video feature information of a target video; the target video includes image feature information and text feature information; inputting the video feature information into a video classification model to perform a first classification process on the target video, obtaining M first classification categories corresponding to the target video, and performing a second classification process on the image feature information and text feature information to obtain N second classification categories corresponding to the target video, where M and N are positive integers; calibrating the N second classification categories based on the M first classification categories, and determining the target classification category corresponding to the target video using the calibrated N second classification categories.

Description

Technical Field

[0001] This application belongs to the field of artificial intelligence technology, specifically relating to a video classification method, apparatus, electronic device, and storage medium. Background Technology

[0002] Currently, electronic devices can categorize videos by tagging them, allowing users to quickly find the videos they need. In related technologies, electronic devices can use neural networks to output the network feature distribution of videos as tags, enabling the device to determine the video category based on these tags.

[0003] However, when the neural network outputs a large number of labels (e.g., hundreds of thousands), the computational load for the neural network to identify the video category through the labels also increases, leading to a decrease in the convergence speed of the neural network. As a result, the video classification efficiency of electronic devices is poor. Summary of the Invention

[0004] The purpose of this application is to provide a video classification method, apparatus, electronic device, and storage medium that can solve the problem of poor video classification efficiency in electronic devices.

[0005] In a first aspect, embodiments of this application provide a video classification method, which includes: acquiring video feature information of a target video; the target video includes image feature information and text feature information; inputting the video feature information into a video classification model, performing a first classification process on the target video to obtain M first classification categories corresponding to the target video, and performing a second classification process on each of the multiple video features to obtain N second classification categories corresponding to the target video, where M and N are positive integers; calibrating the N second classification categories based on the M first classification categories, and determining the target classification category corresponding to the target video through the calibrated N second classification categories.

[0006] Secondly, embodiments of this application provide a video classification device, which includes: an acquisition module, a processing module, a calibration module, and a determination module. The acquisition module is used to acquire video feature information of a target video; the target video includes image feature information and text feature information. The processing module is used to input the video feature information into a video classification model, perform a first classification process on the target video to obtain M first classification categories corresponding to the target video, and perform a second classification process on each of the multiple video features to obtain N second classification categories corresponding to the target video, where M and N are positive integers. The calibration module is used to calibrate the N second classification categories based on the M first classification categories. The determination module is used to determine the target classification category corresponding to the target video using the calibrated N second classification categories.

[0007] Thirdly, embodiments of this application provide an electronic device including a processor and a memory, wherein the memory stores programs or instructions executable on the processor, and the programs or instructions, when executed by the processor, implement the steps of the method described in the first aspect.

[0008] Fourthly, embodiments of this application provide a readable storage medium on which a program or instructions are stored, which, when executed by a processor, implement the steps of the method described in the first aspect.

[0009] Fifthly, embodiments of this application provide a chip, the chip including a processor and a communication interface, the communication interface being coupled to the processor, the processor being used to run programs or instructions to implement the method as described in the first aspect.

[0010] In a sixth aspect, embodiments of this application provide a computer program product stored in a storage medium, which is executed by at least one processor to implement the method described in the first aspect.

[0011] In this embodiment, the electronic device can perform a first classification process on the target video to obtain M first classification categories corresponding to the target video, and then perform a second classification process on the image feature information and text feature information of the target video to obtain N second classification categories corresponding to the target video. The electronic device can then calibrate the second classification categories based on the M first classification categories and determine the target classification category corresponding to the target video using the calibrated second classification categories. In this solution, the electronic device can process the feature information of the target video through multiple branches, and the method of guiding the second branch through the first branch can improve the accuracy of the classification of the target video. Therefore, when the number of classification categories output by the electronic device reaches tens of millions, since the electronic device can determine the classification of the target video through guidance between different branches, the electronic device can quickly identify the classification category corresponding to the target video using labels of different fine granularities. This improves the efficiency and accuracy of the electronic device in identifying video classifications. Attached Figure Description

[0012] Figure 1 This is a flowchart of a video classification method provided in an embodiment of this application;

[0013] Figure 2 This is a schematic diagram of an example interface of a video classification method provided in an embodiment of this application;

[0014] Figure 3 This is a schematic diagram of the structure of a video classification device provided in an embodiment of this application;

[0015] Figure 4This is one of the hardware structure diagrams of an electronic device provided in the embodiments of this application;

[0016] Figure 5 This is a second schematic diagram of the hardware structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0017] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0018] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such use of data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class and the number of objects is not limited; for example, a first object can be one or more. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0019] The video classification method provided in this application will be described in detail below with reference to the accompanying drawings, through specific embodiments and application scenarios.

[0020] Currently, with the development of communication technology, electronic devices are increasingly equipped with more and more functions. For example, electronic devices can use neural networks to classify videos, allowing users to quickly find videos that meet their needs based on the classification. Typically, electronic devices can use a fully connected layer in the neural network to output the network features of the video as labels, thereby determining the category to which the video belongs. However, when the neural network outputs a large number of labels (e.g., hundreds of thousands), a single fully connected layer cannot completely depict the mapping between features and categories. Furthermore, there are differences between coarse and fine granularity among the labels. Electronic devices need to use the neural network to first perform coarse-grained classification between categories, and then perform fine-grained classification. Because a single fully connected layer needs to process a large amount of data, the accuracy and convergence speed of the neural network are poor, resulting in low efficiency for video classification in electronic devices.

[0021] In this embodiment, the electronic device can perform a first classification process on the target video to obtain M first classification categories corresponding to the target video, and then perform a second classification process on the image feature information and text feature information of the target video to obtain N second classification categories corresponding to the target video. The electronic device can then calibrate the second classification categories based on the M first classification categories and determine the target classification category corresponding to the target video using the calibrated second classification categories. In this scheme, the electronic device can process the feature information of the target video through multiple branches, and the method of guiding the second branch through the first branch can improve the accuracy of the classification of the target video. Therefore, when the number of classification categories output by the electronic device reaches tens of millions, since the electronic device can determine the classification of the target video through guidance between different branches, the electronic device can quickly identify the classification category corresponding to the target video using labels of different fine granularities. This improves the efficiency and accuracy of the electronic device in identifying video classifications.

[0022] The video classification implementation subject provided in this application embodiment can be a video classification device, which can be an electronic device or a functional module within an electronic device. The following description uses an electronic device as an example to illustrate the technical solution provided in this application embodiment.

[0023] This application provides a video classification method. Figure 1 A flowchart of a video classification method provided in an embodiment of this application is shown. Figure 1 As shown, the video classification method provided in this application embodiment may include the following steps 201 to 203.

[0024] Step 201: The electronic device acquires the video feature information of the target video.

[0025] In this embodiment of the application, the target video includes image feature information and text feature information.

[0026] In this embodiment of the application, the electronic device can obtain video feature information of the target video, and then determine at least one classification category corresponding to the target video through the video feature information.

[0027] Optionally, in the embodiments of this application, the aforementioned video frame feature information may be video features in each video frame of the target video, or video features in key frames of the target video.

[0028] Specifically, the aforementioned keyframes are video frames captured by the electronic device over a preset duration.

[0029] Optionally, in this embodiment of the application, the text feature information may be the title feature information corresponding to the target video and the text (e.g., subtitle) feature information in the target video.

[0030] Optionally, in this embodiment of the application, the above-mentioned classification categories may include at least one of the following: game category, chat category, lifestyle category, and food category, etc.

[0031] Optionally, in this embodiment of the application, after obtaining the video feature information of the target video, the electronic device can save the video feature information in vector form or array form, and determine at least one classification category corresponding to the target video through the video feature information in vector form or array form.

[0032] Optionally, in the embodiments of this application, step 201 can be implemented by the following steps 201a and 201b.

[0033] Step 201a: The electronic device extracts the image feature information corresponding to L video frames of the target video and the text feature information corresponding to the target video, where L is an integer greater than 1.

[0034] In this embodiment of the application, the electronic device can uniformly sample L video frames of the target video, then sort the L video frames according to the first order of the target video, and then input the L video frames into the video feature extraction model to extract the image feature information corresponding to the L video frames. Then, the electronic device can input the text corresponding to the target video into the text feature extraction model to extract the text feature information corresponding to the target video.

[0035] Optionally, in the embodiments of this application, the first order can be determined by the electronic device based on the timing of the target video (e.g., the original timing of the target video); or, the first order can be determined by the electronic device based on the number of image objects in the L-frame video frame.

[0036] Optionally, in the embodiments of this application, the above-mentioned image object may include at least one of the following: a person object, an animal object, and a landscape object.

[0037] Optionally, in the embodiments of this application, the video feature extraction model described above can be a vision transformer neural network, a convolutional neural network, a back propagation (BP) neural network, a radial basis function (RBF) neural network, a linear neural network, or a self-organizing neural network.

[0038] Specifically, the text feature extraction model mentioned above can be the BERT model.

[0039] For example, such as Figure 2 As shown, for electronic devices to extract image feature information corresponding to L frames of the target video, the electronic device uniformly samples 8 frames of the target video. Figure 2 The input consists of 8 video frames (represented by 1-8) forming a vector [8, 3, 224, 224] (hereinafter referred to as the first vector). These 8 video frames are then flattened into an image by the Tokenize module 10 and input into the spatial transformer module 11 of the vision transformer neural network to extract the spatial features of each frame. The output is a second vector 12 of [8, 64, 768], which is then processed by the vision transformer neural network. The temporal transformer module 13 in the transformer neural network aggregates the second vector along the temporal dimension, outputting a third vector 14 of [128, 768]. This third vector represents the image feature information corresponding to 8 video frames. Specifically, the 8 in the first vector indicates the 8 video frames, the 3 in the first vector indicates the three dimensions (e.g., R, G, B) of the 8 video frames, and the 224 in the first vector indicates the image size of each video frame in the 8 video frames. Similarly, the 8 in the second vector indicates the 8 video frames, the 64 in the second vector indicates the number of image feature information in the 8 video frames, and the 768 in the second vector indicates the image size of each video frame in the 8 video frames. Finally, the 128 in the third vector indicates the number of image feature information obtained after secondary processing of the 8 video frames, and the 768 in the third vector indicates the image size of each video frame in the 8 video frames.

[0040] For example, to extract the corresponding text feature information of the target video by an electronic device, the electronic device can input the title text information corresponding to the target video into the BERT model, so as to extract the text features corresponding to the title text information through the BERT model and output the fourth vector [64, 768], where 64 in the fourth vector is used to indicate the number of features of the title text information, and 768 in the fourth vector is used to indicate the corresponding text size of the title text information.

[0041] Step 201b: The electronic device fuses image feature information and text feature information to obtain video feature information.

[0042] In this embodiment of the application, after the electronic device obtains the third vector corresponding to the image feature information and the fourth vector corresponding to the text feature information, the electronic device can fuse the third vector and the fourth vector through the target model to obtain video feature information.

[0043] Specifically, the target model mentioned above can be a cross-attention model.

[0044] For example, electronic devices can use the Cross Attention model to fuse image feature information and text feature information, and finally output a feature vector (i.e., the feature vector corresponding to the video feature information).

[0045] In this embodiment, the electronic device can integrate the image feature information and text feature information of the target video through the Cross Attention model, thereby identifying the correlation between the image feature information and the text feature information, and thus improving the accuracy of the electronic device in determining the classification of the target video.

[0046] Step 202: The electronic device inputs the video feature information into the video classification model, performs the first classification processing on the target video to obtain M first classification categories corresponding to the target video, and performs the second classification processing on the image feature information and text feature information to obtain N second classification categories corresponding to the target video, where M and N are positive integers.

[0047] In this embodiment of the application, the video classification model may include two processing branches. The first processing branch may include a first classification processing module, and the second processing branch may include a second classification processing module. Thus, the electronic device can obtain different granularity classification categories corresponding to the target video through the two classification processing modules.

[0048] In this embodiment of the application, the electronic device can perform a first classification process on the video feature information through the fully connected layer in the first classification processing module of the video classification model to obtain M first classification categories.

[0049] In this embodiment of the application, the electronic device performs second classification processing on the video feature information through the fully connected layer in the second classification processing module of the video classification model to obtain N second classification categories.

[0050] In this embodiment, the recognition accuracy of the second classification processing module is greater than that of the first classification processing module.

[0051] It should be noted that the parameters in the fully connected layer of the second classification processing module are different from those in the fully connected layer of the first classification processing module. Therefore, the categories corresponding to the first classification category and the second classification category are not the same.

[0052] For example, after obtaining the vector corresponding to the video feature information (hereinafter referred to as the fifth vector), the electronic device can output the fifth vector to the first classification processing module and the second classification processing module respectively to obtain the sixth vector and the seventh vector. The sixth vector is used to indicate the probability value corresponding to M first classification categories, and the seventh vector is used to indicate the probability value corresponding to N second classification categories. Thus, the electronic device can determine the M first classification categories and N second classification categories corresponding to the target video based on the sixth vector and the seventh vector.

[0053] Optionally, in the embodiments of this application, when the video classification model outputs a large number of classification categories (e.g., tens of millions), the electronic device can add at least one processing branch to improve the efficiency of the electronic device in determining the video category corresponding to the target video.

[0054] Step 203: The electronic device calibrates N second-class categories based on M first-class categories, and determines the target category corresponding to the target video through the calibrated N second-class categories.

[0055] In this embodiment of the application, the electronic device can use M first classification categories as a benchmark and calibrate N second classification categories, so that the electronic device can determine the target classification category corresponding to the target video through the calibrated N second classification categories.

[0056] Optionally, in the embodiments of this application, the "electronic device calibrates N second classification categories based on M first classification categories" in step 203 above can be specifically implemented through the following steps 203a and 203b.

[0057] Step 203a: The electronic device acquires the first category of information.

[0058] In this embodiment of the application, the aforementioned first category information is used to characterize the probability that the target video belongs to any of the M first category classifications.

[0059] In this embodiment of the application, the electronic device can sample M first classification categories to obtain first classification information.

[0060] Optionally, in this embodiment of the application, after obtaining M first classification categories, the electronic device can perform cross-entropy loss processing on the M first classification categories and the manually labeled classifications respectively to obtain the degree of difference between each of the M first classification categories and the manually labeled classifications. Thus, the electronic device can adjust the video classification model according to the cross-entropy loss value, thereby enabling the electronic device to further improve the accuracy of the video classification model in determining the video classification.

[0061] Optionally, in the embodiments of this application, step 203a can be implemented by steps 301 and 302 as described below.

[0062] Step 301: The electronic device performs cross-entropy calculation on the M first classification categories and the preset categories to obtain M first loss values.

[0063] In this embodiment of the application, each of the above M first loss values ​​is used to indicate the degree of difference between each of the M first classification categories and the preset category.

[0064] For example, an electronic device can use the BCELoss algorithm to calculate the cross-entropy between M first classification categories and preset categories to obtain M first loss values.

[0065] Step 302: The electronic device samples the M first loss values ​​to obtain the first category information.

[0066] For example, the electronic device can perform gumbel sigmoid mask processing on the M first classification categories after calculating the first loss value to obtain the first category information.

[0067] For example, Gumbel sigmoid is a discrete, backpropagation-enabled sampling strategy, as shown in Equation 1 below:

[0068]

[0069] Here, Es represents the M first-class categories after calculating the first loss value (hereinafter referred to as L1). This sampling strategy ensures that positions with larger values ​​in L1 are sampled as 1 with a higher probability, but also with a lower probability of being sampled as 0; similarly, positions with smaller values ​​are sampled as 0 with a higher probability, but also with a lower probability of being sampled as 1. Since L1 is trained using manually labeled video tags, the model outputs a probability close to 0 at positions with tags completely unrelated to the video. Therefore, in the first-class information of L1, the values ​​corresponding to positions with tags completely unrelated to the video will be smaller, while the values ​​corresponding to positions with tags related to the video will be larger. Thus, the result of the gumbel-sigmoid mask is more likely to be 0 at positions with tags completely unrelated to the video, and more likely to be 1 at positions with tags possibly related to the video. This allows the model to mask out most categories completely unrelated to the video from the N second-class categories, focusing only on certain categories that may be related to the video.

[0070] τ is a temperature parameter, and the sampling strategy can be adjusted during training by changing its value. When τ is large, the model tends to mask fewer categories; when τ is small, the model tends to mask more categories. Setting a large τ in the initial stage of model training will cause the model to mask fewer categories in branch one, leaving more categories unmasked and allowing them to enter branch two for training. This helps prevent the model from missing some video-related categories due to insufficient training in the initial stage, thus preventing these categories from being trained in branch two. As the model converges better, gradually decreasing τ will cause the model to mask more categories, allowing the most relevant categories to enter branch two, which is beneficial for the model to focus more on difficult-to-classify categories in branch two.

[0071] G' and G” represent two Gumbel noises, which are added to achieve sampling. This is illustrated in Formula 2 below:

[0072] G=-log(-log(Uniform(0,1))) (Formula 2)

[0073] Uniform(0,1) represents the electronic device sampling normally between 0 and 1.

[0074] In this embodiment of the application, when the model is trained, after the gumbel sigmoid mask sampling strategy, a mask vector M of length k (i.e., the first category information) will be obtained, M ~ [0,1].

[0075] It should be noted that the above mask is used by the electronic device to determine the classification value in the first category information that is greater than or equal to a preset threshold as the first value (for example, 1, where 1 represents that the target video belongs to the category 100%), and then determine the classification value in the first category information that is less than the preset threshold as the second value (for example, 0, where 0 represents that the target video does not belong to the category 100%).

[0076] Step 203b: The electronic device calibrates N second classification categories based on the first category information, M first classification categories, and N second classification categories.

[0077] For example, the electronic device can add the vector corresponding to the first category and the vector corresponding to the second category, and then multiply the vector after adding the vectors by the vector corresponding to the first category information to obtain the vector corresponding to the second category information.

[0078] Optionally, in this embodiment of the application, the step 203 above, "the electronic device determines the target classification category corresponding to the target video through the calibrated N second classification categories", can be specifically implemented through the following steps 203c and 203d.

[0079] Step 203c: The electronic device samples the calibrated N second classification categories to obtain N second classification information.

[0080] In this embodiment of the application, each of the above N second classification information is used to characterize the probability that the target video belongs to any of the calibrated N second classification categories.

[0081] For example, an electronic device can perform sigmoid sampling on N calibrated second classification categories to obtain N second classification information.

[0082] Step 203d: The electronic device determines the target category corresponding to the target video based on N second-classification information.

[0083] In this embodiment of the application, after obtaining the vector corresponding to the second category information, the electronic device can perform masking on the vector to determine the target classification category corresponding to the target video.

[0084] In this embodiment, the electronic device uses the output of the first processing branch to guide the output of the second branch. This allows the electronic device to select the most relevant part of the labels for classification and perform a more precise classification. Furthermore, the electronic device can distinguish between coarse and fine granularity. The multi-branch structure can also accelerate the convergence speed of the electronic device and improve its efficiency in classifying target videos.

[0085] This application provides a video classification method. An electronic device can perform a first classification process on a target video to obtain M first classification categories, and then perform a second classification process on the image and text feature information of the target video to obtain N second classification categories. The electronic device can then calibrate the second classification categories based on the M first classification categories and determine the target classification category of the target video using the calibrated second classification categories. In this solution, the electronic device can process the feature information of the target video through multiple branches. Furthermore, by using the method of the first branch guiding the second branch, the accuracy of the target video's classification can be improved. Therefore, when the electronic device outputs classification categories at the tens of millions level, since the electronic device can determine the category of the target video through guidance between different branches, it can quickly identify the classification category corresponding to the target video using labels of different fine granularities. This improves the efficiency and accuracy of the electronic device in identifying video classifications.

[0086] Optionally, in this embodiment of the application, after step 203 above, the video classification method provided in this embodiment of the application further includes the following steps 401 and 402.

[0087] Step 401: The electronic device performs cross-entropy calculation on the calibrated N second classification categories and the preset category to obtain N second loss values.

[0088] In this embodiment of the application, each of the above N second loss values ​​is used to characterize the degree of difference between each of the N second classification categories after calibration and the preset category.

[0089] For example, an electronic device can use the BCELoss algorithm to calculate the cross-entropy between the calibrated N second classification categories and the preset category to obtain N second loss values.

[0090] Step 402: The electronic device updates the video classification model based on M first loss values ​​and N second loss values.

[0091] For example, the electronic device can add M first loss values ​​and N second loss values ​​to obtain a third loss value. Then, the electronic device can adjust the parameters of the video classification model according to the third loss value to update the video classification model, thereby making the convergence effect of the video classification model better.

[0092] In this embodiment, the electronic device can adjust the video classification model by using the loss values ​​between M first classification categories and preset categories and the loss values ​​between N second classification categories and preset categories, thereby improving the convergence effect of the video classification model and thus improving the accuracy of the electronic device in determining the video classification.

[0093] It should be noted that the video classification method provided in this application can be executed by a video classification device, an electronic device, or a functional module or entity within an electronic device. This application uses a video classification device executing the video classification method as an example to illustrate the video classification device provided in this application.

[0094] Figure 3 A schematic diagram of a possible structure of the video classification device involved in an embodiment of this application is shown. For example... Figure 3 As shown, the video classification device 70 may include: an acquisition module 71, a processing module 72, and a determination module 73.

[0095] The acquisition module 71 is used to acquire video feature information of the target video, which includes image feature information and text feature information. The processing module 72 is used to input the video feature information into the video classification model, perform a first classification process on the target video to obtain M first classification categories corresponding to the target video, and perform a second classification process on each of the multiple video features to obtain N second classification categories corresponding to the target video, where M and N are positive integers. The determination module 73 is used to calibrate the N second classification categories based on the M first classification categories, and determine the target classification category corresponding to the target video through the calibrated N second classification categories.

[0096] In one possible implementation, the aforementioned determining module 73 is specifically used to obtain first category information, which is used to characterize the probability that the target video belongs to any of the M first classification categories; and to calibrate the N second classification categories based on the first category information, the M first classification categories and the N second classification categories.

[0097] In one possible implementation, the acquisition module 71 is specifically used to perform cross-entropy calculation on M first classification categories and preset categories to obtain M first loss values, each first loss value being used to indicate the degree of difference between each of the M first classification categories and the preset category; and to sample the M first loss values ​​to obtain first category information.

[0098] In one possible implementation, the video classification apparatus provided in this application embodiment further includes an update module. The aforementioned processing module 72 is further configured to perform cross-entropy calculation on the calibrated N second classification categories and a preset category to obtain N second loss values. Each second loss value is used to characterize the degree of difference between each of the calibrated N second classification categories and the preset category. The update module is configured to update the video classification model based on the M first loss values ​​and the N second loss values.

[0099] In one possible implementation, the aforementioned determining module 73 is specifically used to sample the calibrated N second classification categories to obtain N second classification information, each second classification information being used to characterize the probability that the target video belongs to any of the calibrated N second classification categories; and to determine the target classification category corresponding to the target video based on the N second classification information.

[0100] In one possible implementation, the acquisition module 71 is specifically used to extract image feature information corresponding to L video frames of the target video, and text feature information corresponding to the target video, where L is an integer greater than 1; and to fuse the image feature information and text feature information to obtain video feature information.

[0101] This application provides a video classification device. Since the electronic device can process the feature information of the target video through multiple branches, and the accuracy of the target video's classification can be improved by using the first branch to guide the second branch, when the electronic device outputs classification categories at the tens of millions level, the electronic device can determine the category of the target video through guidance between different branches. Thus, the electronic device can quickly identify the category corresponding to the target video through labels of different fine granularities, thereby improving the efficiency and accuracy of the electronic device in identifying video classification.

[0102] The video classification device in this application embodiment can be a device, or a component, integrated circuit, or chip in an electronic device. The device can be a mobile electronic device or a non-mobile electronic device. For example, a mobile electronic device can be a mobile phone, tablet computer, laptop computer, PDA, in-vehicle electronic device, mobile internet device (MID), augmented reality (AR) / virtual reality (VR) device, robot, wearable device, ultra-mobile personal computer (UMPC), netbook, or personal digital assistant (PDA), etc. It can also be a server, network attached storage (NAS), personal computer (PC), television set (TV), ATM, or self-service machine, etc. This application embodiment does not specifically limit the device.

[0103] The video classification device in this application embodiment can be a device with an operating system. This operating system can be Android, iOS, or other possible operating systems; this application embodiment does not specifically limit the specific operating system used.

[0104] The video classification device provided in this application embodiment can achieve... Figures 1 to 2 The various processes implemented in the method implementation examples will not be described again here to avoid repetition.

[0105] Optionally, such as Figure 4 As shown, this application embodiment also provides an electronic device 90, including a processor 91 and a memory 92. The memory 92 stores a program or instructions that can run on the processor 91. When the program or instructions are executed by the processor 91, they implement the various steps of the above-described video classification method embodiment and can achieve the same technical effect. To avoid repetition, they will not be described again here.

[0106] It should be noted that the electronic devices in the embodiments of this application include the mobile electronic devices and non-mobile electronic devices described above.

[0107] Figure 5 A schematic diagram of the hardware structure of an electronic device to implement an embodiment of this application.

[0108] The electronic device 100 includes, but is not limited to, components such as: radio frequency unit 101, network module 102, audio output unit 103, input unit 104, sensor 105, display unit 106, user input unit 107, interface unit 108, memory 109, and processor 110.

[0109] Those skilled in the art will understand that the electronic device 100 may also include a power supply (such as a battery) for supplying power to various components. The power supply may be logically connected to the processor 110 through a power management system, thereby enabling functions such as managing charging, discharging, and power consumption through the power management system. Figure 5 The electronic device structure shown does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown, or combine certain components, or have different component arrangements, which will not be elaborated here.

[0110] The processor 110 is configured to acquire video feature information of a target video, which includes image feature information and text feature information; input the video feature information into a video classification model to perform a first classification process on the target video to obtain M first classification categories corresponding to the target video; perform a second classification process on each of the multiple video features to obtain N second classification categories corresponding to the target video, where M and N are positive integers; and calibrate the N second classification categories based on the M first classification categories, and determine the target classification category corresponding to the target video using the calibrated N second classification categories.

[0111] This application provides an electronic device that can process the feature information of a target video through multiple branches. By using the method of guiding the second branch through the first branch, the accuracy of the target video's classification can be improved. When the electronic device outputs classification categories at the tens of millions level, since the electronic device can determine the category of the target video through guidance between different branches, the electronic device can quickly identify the category corresponding to the target video through labels of different fine granularities. Thus, the efficiency and accuracy of the electronic device in identifying video classification are improved.

[0112] Optionally, in this embodiment of the application, the processor 110 is specifically used to obtain first category information, which is used to characterize the probability that the target video belongs to any of the M first classification categories; and to calibrate the N second classification categories based on the first category information, the M first classification categories and the N second classification categories.

[0113] Optionally, in this embodiment of the application, the processor 110 is specifically used to perform cross-entropy calculation on M first classification categories and preset categories to obtain M first loss values, each first loss value being used to indicate the degree of difference between each of the M first classification categories and the preset category; and to perform sampling processing on the M first loss values ​​to obtain first category information.

[0114] Optionally, in this embodiment of the application, the processor 110 is further configured to, after determining the target classification category corresponding to the target video through the calibrated N second classification categories, perform cross-entropy calculation on the calibrated N second classification categories and the preset category to obtain N second loss values, each second loss value being used to characterize the degree of difference between each of the calibrated N second classification categories and the preset category; and update the video classification model according to the M first loss values ​​and the N second loss values.

[0115] Optionally, in this embodiment of the application, the processor 110 is specifically used to sample the calibrated N second classification categories to obtain N second classification information, each second classification information being used to characterize the probability that the target video belongs to any of the calibrated N second classification categories; and to determine the target classification category corresponding to the target video based on the N second classification information.

[0116] Optionally, in this embodiment of the application, the processor 110 is specifically used to extract image feature information corresponding to L video frames of the target video and text feature information corresponding to the target video, where L is an integer greater than 1; and to fuse the image feature information and text feature information to obtain video feature information.

[0117] The electronic device provided in this application embodiment can implement the various processes implemented in the above method embodiments and achieve the same technical effect. To avoid repetition, it will not be described again here.

[0118] For details on the beneficial effects of the various implementation methods in this embodiment, please refer to the beneficial effects of the corresponding implementation methods in the above method embodiments. To avoid repetition, these will not be repeated here.

[0119] It should be understood that, in this embodiment, the input unit 104 may include a graphics processing unit (GPU) 1041 and a microphone 1042. The GPU 1041 processes image data of still images or videos obtained by an image capture device (such as a camera) in video capture mode or image capture mode. The display unit 106 may include a display panel 1061, which may be configured in the form of a liquid crystal display, an organic light-emitting diode, or the like. The user input unit 107 includes at least one of a touch panel 1071 and other input devices 1072. The touch panel 1071 is also called a touch screen. The touch panel 1071 may include a touch detection device and a touch controller. Other input devices 1072 may include, but are not limited to, a physical keyboard, function keys (such as volume control buttons, power buttons, etc.), a trackball, a mouse, and a joystick, which will not be described in detail here.

[0120] The memory 109 can be used to store software programs and various data. The memory 109 may primarily include a first storage area for storing programs or instructions and a second storage area for storing data. The first storage area may store the operating system, application programs or instructions required for at least one function (such as sound playback, image playback, etc.). Furthermore, the memory 109 may include volatile memory or non-volatile memory, or both. The non-volatile memory may be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory can be random access memory (RAM), static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct memory bus RAM (DRRAM). The memory 109 in the embodiments of this application includes, but is not limited to, these and any other suitable types of memory.

[0121] Processor 110 may include one or more processing units; optionally, processor 110 integrates an application processor and a modem processor, wherein the application processor mainly handles operations involving the operating system, user interface, and applications, and the modem processor mainly handles wireless communication signals, such as a baseband processor. It is understood that the aforementioned modem processor may also not be integrated into processor 110.

[0122] This application also provides a readable storage medium storing a program or instructions. When the program or instructions are executed by a processor, they implement the various processes of the above method embodiments and achieve the same technical effect. To avoid repetition, they will not be described again here.

[0123] The processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes computer-readable storage media, such as computer read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0124] This application embodiment also provides a chip, which includes a processor and a communication interface. The communication interface is coupled to the processor. The processor is used to run programs or instructions to implement the various processes of the above method embodiments and achieve the same technical effect. To avoid repetition, it will not be described again here.

[0125] It should be understood that the chip mentioned in the embodiments of this application may also be referred to as a system-on-a-chip, system chip, chip system, or system-on-a-chip, etc.

[0126] This application provides a computer program product, which is stored in a storage medium and executed by at least one processor to implement the various processes of the video classification method embodiments described above, and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0127] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, it should be noted that the scope of the methods and apparatuses in the embodiments of this application is not limited to performing functions in the order shown or discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

[0128] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a computer software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0129] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

Claims

1. A video classification method, characterized in that, The method includes: Obtain video feature information of the target video; the target video includes image feature information and text feature information; The video feature information is input into a video classification model to perform a first classification process on the target video, resulting in M ​​first classification categories corresponding to the target video. The image feature information and the text feature information are then subjected to a second classification process to obtain N second classification categories corresponding to the target video, where M and N are positive integers. Based on the M first classification categories, the N second classification categories are calibrated, and the target classification category corresponding to the target video is determined through the calibrated N second classification categories; The calibration of the N second classification categories based on the M first classification categories includes: Obtain first category information, which is used to characterize the probability that the target video belongs to any of the M first category classifications; Based on the first category information, the M first classification categories, and the N second classification categories, calibrate the N second classification categories; The video classification model includes at least two processing branches, which are used to obtain different granularity classification categories corresponding to the target video, and each processing branch has a different recognition accuracy.

2. The method according to claim 1, characterized in that, The acquisition of the first category of information includes: Cross-entropy is calculated between the M first categories and the preset category to obtain M first loss values. Each first loss value is used to indicate the degree of difference between each of the M first categories and the preset category. The M first loss values ​​are sampled to obtain the first category information.

3. The method according to claim 1 or 2, characterized in that, After determining the target classification category corresponding to the target video through the calibrated N second classification categories, the method further includes: Cross-entropy is calculated between the calibrated N second classification categories and the preset category to obtain N second loss values. Each second loss value is used to characterize the degree of difference between each of the calibrated N second classification categories and the preset category. The video classification model is updated based on the M first loss values ​​and the N second loss values.

4. The method according to claim 1, characterized in that, The step of determining the target classification category corresponding to the target video through the calibrated N second classification categories includes: The N calibrated second classification categories are sampled to obtain N second classification information. Each second classification information is used to characterize the probability that the target video belongs to any of the N calibrated second classification categories. Based on the N second classification information, the target classification category corresponding to the target video is determined.

5. The method according to claim 1, characterized in that, Obtain video feature information of the target video, including: Extract the image feature information corresponding to L video frames of the target video, and the text feature information corresponding to the target video, where L is an integer greater than 1; The image feature information and the text feature information are fused to obtain the video feature information.

6. A video classification device, characterized in that, The video classification device includes: an acquisition module, a processing module, a calibration module, and a determination module; The acquisition module is used to acquire video feature information of the target video; the target video includes image feature information and text feature information; The processing module is used to input the video feature information into the video classification model, perform a first classification process on the target video to obtain M first classification categories corresponding to the target video, and perform a second classification process on the image feature information and the text feature information to obtain N second classification categories corresponding to the target video, where M and N are positive integers; The calibration module is used to calibrate the N second classification categories based on the M first classification categories; The determining module is used to determine the target classification category corresponding to the target video through the calibrated N second classification categories; The calibration module is specifically used to acquire first category information, which characterizes the probability that the target video belongs to any of the M first category classifications; and to calibrate the N second category classifications based on the first category information, the M first category classifications, and the N second category classifications. The video classification model includes at least two processing branches, which are used to obtain different granularity classification categories corresponding to the target video, and each processing branch has a different recognition accuracy.

7. The apparatus according to claim 6, characterized in that, The acquisition module specifically performs cross-entropy calculation on the M first classification categories and the preset category to obtain M first loss values. Each first loss value is used to indicate the degree of difference between each of the M first classification categories and the preset category. The module then samples the M first loss values ​​to obtain the first category information.

8. The apparatus according to claim 6 or 7, characterized in that, The processing module is further configured to, after determining the target classification category corresponding to the target video through the calibrated N second classification categories, perform cross-entropy calculation on the calibrated N second classification categories and the preset category to obtain N second loss values, each second loss value being used to characterize the degree of difference between each of the calibrated N second classification categories and the preset category; The video classification model is then updated based on the M first loss values ​​and the N second loss values.

9. The apparatus according to claim 6, characterized in that, The determining module is specifically used to sample the calibrated N second classification categories to obtain N second classification information, each second classification information being used to characterize the probability that the target video belongs to any of the calibrated N second classification categories; and to determine the target classification category corresponding to the target video based on the N second classification information.

10. The apparatus according to claim 6, characterized in that, The acquisition module is specifically used to extract image feature information corresponding to L video frames of the target video, and text feature information corresponding to the target video, where L is an integer greater than 1; The image feature information and the text feature information are fused to obtain the video feature information.

11. An electronic device, characterized in that, It includes a processor, a memory, and a program or instructions stored in the memory and executable on the processor, wherein the program or instructions, when executed by the processor, implement the steps of the video classification method as described in any one of claims 1 to 5.

12. A readable storage medium, characterized in that, The readable storage medium stores a program or instructions that, when executed by a processor, implement the steps of the video classification method as described in any one of claims 1 to 5.

Images