Video classification method and device, electronic equipment and storage medium

Abstract

The application provides a video classification method and device, electronic equipment and storage medium, and belongs to the multimedia technical field. The method comprises the following steps: inputting at least one image frame acquired from a target video into an operation model based on a convolutional neural network to obtain at least one image frame vector; determining target clusters corresponding to the at least one image frame vector from at least two clusters, wherein one cluster is used to represent a type of image frame vector; acquiring to-be-recognized text corresponding to the target video based on the target clusters corresponding to the at least one image frame vector; inputting the to-be-recognized text into a natural language processing model, decoding the to-be-recognized text through the natural language processing model, and obtaining a video type to which the target video belongs. In the above scheme, when classifying the target video, the image frames in the target video are converted into text content in the to-be-recognized text, the calculation complexity is reduced, the processing time is shortened, and the requirement for the processing capacity of the equipment is reduced.

Description

Technical Field

[0001] This application relates to the field of multimedia technology, and in particular to a video classification method, apparatus, electronic device, and storage medium. Background Technology

[0002] With the development of multimedia technology, various video websites have emerged, and users increasingly enjoy watching videos of interest on these websites during their free time. For video websites, given the vast number of videos and the different interests of various users, they typically categorize or tag videos to help users quickly find what they're looking for. This categorization indicates the main content of each video, making it easier for users to select.

[0003] In related technologies, deep learning methods are typically used. Models such as Convolutional Neural Networks (CNN) or Long Short-Term Memory (LSTM) are used to determine the type of each image frame based on the image vectors corresponding to each image frame in the video, thereby achieving the purpose of classifying the video or adding labels.

[0004] Because image vectors have a large dimension, the above-mentioned technical solutions have high computational complexity when processing image frames, which requires a long time and places high demands on the processing power of the equipment. Summary of the Invention

[0005] This application provides a video classification method, apparatus, electronic device, and storage medium. By using one-dimensional cluster center identifiers instead of high-dimensional image vectors, computational complexity is reduced, processing time is shortened, and the requirements for device processing capabilities are lowered. The technical solution is as follows:

[0006] On the one hand, a video classification method is provided, the method comprising:

[0007] At least one image frame obtained from the target video is input into a convolutional neural network-based computational model to obtain at least one image frame vector.

[0008] Determine the target cluster corresponding to the at least one image frame vector from at least two clusters, where a cluster is used to represent a class of image frame vectors;

[0009] Based on the target clusters corresponding to the at least one image frame vector, obtain the text to be recognized corresponding to the target video;

[0010] The text to be identified is input into a natural language processing model, and the natural language processing model decodes the text to obtain the video type to which the target video belongs.

[0011] On the other hand, a video classification device is provided, the device comprising:

[0012] The vector acquisition module is used to input at least one image frame obtained from the target video into a convolutional neural network-based computational model to obtain at least one image frame vector.

[0013] A determination module is used to determine the target clusters corresponding to the at least one image frame vector from at least two clusters, wherein a cluster is used to represent a class of image frame vectors;

[0014] The text acquisition module is used to acquire the text to be recognized corresponding to the target video based on the target clusters corresponding to the at least one image frame vector;

[0015] The model processing module is used to input the text to be identified into a natural language processing model, and decode the text to be identified through the natural language processing model to obtain the video type to which the target video belongs.

[0016] In an optional implementation, the determining module is further configured to, for any image frame vector among the at least one image frame vector, determine the cluster with the highest similarity between the cluster center vector and the image frame vector from the at least two clusters as the target cluster of the image frame vector.

[0017] In an optional implementation, the determining module is further configured to determine the Euclidean distance between the image frame vector and the cluster center vectors of the at least two clusters respectively; in response to the Euclidean distance between the cluster center vector of any cluster and the image frame vector being the smallest, the cluster is designated as the target cluster of the image frame vector.

[0018] In an optional implementation, the text acquisition module is further configured to use the cluster center identifiers of the target clusters corresponding to the at least one image frame vector as words in the text to be identified corresponding to the target video.

[0019] In one alternative implementation, the apparatus further includes:

[0020] The vector acquisition module is also used to input multiple sample image frames into the convolutional neural network-based computation model to obtain multiple sample image frame vectors.

[0021] The clustering module is used to cluster the multiple sample image frame vectors to obtain at least two clusters;

[0022] The allocation module is used to assign a unique cluster core identifier to each of the at least two clusters, wherein the cluster core identifier is in the form of a digital code.

[0023] In one optional implementation, the model processing module is used to decode the text to be identified using the natural language processing model to obtain at least one classification probability, the classification probability being used to represent the probability that the text to be identified belongs to different classification types; and based on the at least one classification probability, to determine the classification type that satisfies the first target condition as the video type to which the target video belongs.

[0024] In an optional implementation, the model processing module is further configured to input the text to be identified into a label model, and decode the text to be identified through the label model to obtain at least one label probability, wherein the label probability is used to represent the probability that the text to be identified belongs to different video labels;

[0025] The determining module is further configured to determine, based on the at least one tag probability, a video tag that satisfies the second target condition as the video tag of the target video.

[0026] On the other hand, an electronic device is provided, comprising a processor and a memory, the memory being used to store at least one piece of program code, which is loaded and executed by the processor to implement the operations performed in the video classification method of the embodiments of this application.

[0027] On the other hand, a storage medium is provided, wherein at least one piece of program code is stored in the storage medium, the at least one piece of program code being used to execute the video classification method in the embodiments of this application.

[0028] The beneficial effects of the technical solutions provided in this application are:

[0029] In this embodiment of the application, by determining the target cluster corresponding to the image frame vector of the image frame in the target video, the text to be identified corresponding to the target video can be obtained. This allows the image frames in the target video to be converted into the text content in the text to be identified when classifying the target video, thereby reducing computational complexity, shortening processing time, and reducing the requirements on device processing capabilities. Attached Figure Description

[0030] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0031] Figure 1 This is a structural block diagram of a video classification system provided according to an embodiment of this application;

[0032] Figure 2 This is a flowchart of a video classification method provided according to an embodiment of this application;

[0033] Figure 3 This is a schematic diagram of a cluster center vector provided according to an embodiment of this application;

[0034] Figure 4 This is a schematic diagram illustrating the acquisition of a target cluster according to an embodiment of this application;

[0035] Figure 5 This is a flowchart of another video classification method provided according to an embodiment of the present disclosure;

[0036] Figure 6 This is a block diagram of a video classification device according to an embodiment of this application;

[0037] Figure 7 This is a schematic diagram of the structure of a terminal according to an embodiment of this application;

[0038] Figure 8 This is a schematic diagram of the structure of a server according to an embodiment of this application. Detailed Implementation

[0039] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0040] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0041] The following is a description of the technologies that may be used in this application:

[0042] Clustering: Clustering is the process of classifying and organizing data members in a dataset that are similar in some way. A cluster is a collection of data objects that are similar to objects within the same cluster and dissimilar to objects in other clusters. Clustering techniques are often referred to as unsupervised learning.

[0043] K-means clustering is an iterative clustering algorithm. Its steps are: randomly select K objects as initial cluster centers; calculate the distance between each object and each cluster center; and assign each object to the nearest cluster center. Each cluster center and the objects assigned to it represent a cluster. Each time a sample is assigned, the cluster centers are recalculated based on the existing objects in the cluster. This process is repeated until a termination condition is met. The termination condition may be that no (or a minimum number) objects are reassigned to different clusters, no (or a minimum number) cluster centers change, or the sum of squared errors reaches a local minimum.

[0044] Faiss: Faiss is an open-source search library for clustering and similarity from Facebook AI. It provides efficient similarity search and clustering for dense vectors, supports searches on the order of billions of vectors, and is currently the most mature approximate nearest neighbor search library. It includes various algorithms for searching vector sets of arbitrary size (note: the size of the vector set is determined by RAM memory), as well as support code for algorithm evaluation and parameter tuning.

[0045] InceptionV3: The Inception network is a significant milestone in the development of CNN (Convolutional Neural Networks) classifiers. Proposed by Google, it has evolved from version V1 to V3 and is currently the preferred open-source model for pre-training image vectors in many image processing domains.

[0046] The video classification method provided in this application can be applied to scenarios involving the classification of videos on websites. These videos can be uploaded by website operators or users, and include genres such as movies, TV series, micro-films, and short videos. According to the video classification method provided in this application, website operators can classify the videos on their websites and display them based on the classification results. Users browsing the website can view videos by category and filter them accordingly.

[0047] The following is a brief overview of the main steps of the video classification method provided in this application: First, at least one image frame obtained from the target video is input into a convolutional neural network-based computational model to obtain at least one image frame vector. The target clusters corresponding to each of the at least one image frame vectors are determined from at least two clusters, where one cluster represents a class of image frame vectors. Then, based on the target clusters corresponding to the at least one image frame vector, the text to be recognized corresponding to the target video is obtained. Finally, the text to be recognized is input into a natural language processing model, which decodes the text to obtain the video type to which the target video belongs. This method obtains the text to be recognized corresponding to the target video by using the target clusters corresponding to the image frame vectors of the target video. This allows the processing of image frames in the target video to be converted into processing text content in the text to be recognized when classifying the target video, thereby reducing computational complexity, shortening processing time, and lowering the requirements for device processing power. For example, matrix calculations on image frames require a GPU (Graphics Processing Unit), while calculations on text can be performed using a CPU (Central Processing Unit), reducing the requirements for device processing power and saving costs.

[0048] Figure 1 This is a structural block diagram of a video classification system 100 provided according to an embodiment of this application. The video classification system 100 includes a terminal 110 and a video classification platform 120.

[0049] Terminal 110 is connected to video classification platform 120 via a wireless or wired network. Terminal 110 can be at least one of a smartphone, game console, desktop computer, tablet computer, e-book reader, MP3 player, MP4 player, and laptop computer. Terminal 110 has an application that supports video classification installed and running. This application can be a social networking application, a multimedia playback application, etc. Illustratively, terminal 110 is a user-operated terminal, and the application running on terminal 110 contains a user account logged in.

[0050] The video classification platform 120 includes at least one of a server, multiple servers, a cloud computing platform, and a virtualization center. The video classification platform 120 provides backend services for applications that support video classification. Optionally, the video classification platform 120 performs the primary classification task, and the terminal 110 performs the secondary classification task; or, the video classification platform 120 performs the secondary classification task, and the terminal 110 performs the primary classification task; or, the video classification platform 120 or the terminal 110 can each perform the classification task independently.

[0051] Optionally, the video classification platform 120 includes: an access server, a video classification server, and a database. The access server provides access services to the terminal 110. The video classification server provides backend services related to video classification. There may be one or more video classification servers. When there are multiple video classification servers, at least two video classification servers may provide different services, and / or at least two video classification servers may provide the same service, such as providing the same service in a load-balanced manner. This embodiment of the application does not limit this.

[0052] Terminal 110 can refer to one of multiple terminals. This embodiment only uses terminal 110 as an example.

[0053] Those skilled in the art will understand that the number of terminals described above can be more or less. For example, there may be only one terminal, or there may be dozens or hundreds of terminals, or even more. In this case, the video classification system may also include other terminals. This application does not limit the number or type of terminals in its embodiments.

[0054] Figure 2 This is a flowchart of a video classification method provided according to an embodiment of this application, such as... Figure 2 As shown. The aforementioned electronic device can be configured as a terminal or as a server. In this embodiment, the electronic device is configured as a server as an example. The video classification method includes the following steps:

[0055] 201. The server will input at least one image frame obtained from the target video into a convolutional neural network-based computational model to obtain at least one image frame vector.

[0056] In this embodiment, the server can be a server for video classification. The target video can be a video uploaded by a user through a client, a video uploaded by staff of a video website through a backend server, or a video from another video website linked to the aforementioned video website. This embodiment does not impose any limitations on this. After obtaining the target video, the server can obtain at least one image frame from the target video. An image frame is actually an image matrix, the elements of which are the pixel values ​​of each pixel in the image frame. The server can convert the image matrix into a vector, thereby obtaining at least one image frame vector. Correspondingly, for any image frame in the at least one image frame, the server can input the image frame into a convolutional neural network-based computational model to obtain the corresponding image frame vector. The convolutional neural network-based computational model can be an InceptionV2 model, an InceptionV3 model, or a VGG16 (Visual Geometry Group 16) model, etc.

[0057] For example, an image frame is a 32×32 image matrix. The server can calculate a 1024-dimensional vector from this image matrix using a convolutional neural network-based computational model. Alternatively, the server can use the NumPy (Numeric Python) library in Python to calculate the same 1024-dimensional vector. During the conversion, the server creates a 1×1024 array using the NumPy library, then iteratively reads the 32 rows of elements from the image matrix and stores the 32 elements of each row in the array. This application does not limit the method used to determine the image frame vector.

[0058] It should be noted that after acquiring the target video, the server can obtain at least one image frame in several ways: the server can obtain the image frames included in the target video frame by frame, obtaining multiple image frames; the server can also obtain one image frame from the target video at fixed frame intervals, obtaining multiple image frames, such as obtaining one image frame every 10 frames; the server can also obtain one image frame from the target video at fixed time intervals, obtaining multiple image frames, such as obtaining one image frame every 2 seconds; the server can also randomly obtain one image frame from the target video. The server can determine the method and number of image frames to be obtained based on the total duration of the target video, and this application embodiment does not impose any restrictions on this.

[0059] For example, if the target video is 2 hours long and contains 30 frames per second, the server would obtain 216,000 frames if it acquires the frames one by one; 21,600 frames if it acquires one frame every 10 frames; 2,160 frames if it acquires one frame every 100 frames; 7,200 frames if it acquires one frame every 1 second; 720 frames if it acquires one frame every 10 seconds; and 1 frame if it acquires a frame randomly. The more frames the server acquires, the more accurate the classification of the target video, but the greater the computational burden on the server. Conversely, fewer frames result in less accurate classification, but faster computation. For a 2-hour target video, the server could acquire either 2,160 or 720 frames to classify it. Correspondingly, when the total duration of the target video is relatively short, such as a short video with a total duration of no more than 120 seconds, or even a short video with a total duration of no less than 30 seconds, the server can choose to acquire image frames frame by frame, or acquire an image frame every 3, 5, 10 frames, 1 second, or 2 seconds, etc., ensuring both computational speed and the accuracy of the classification results. In addition, the server can also randomly acquire an image frame multiple times from the target video, giving the acquired image frames a certain degree of randomness, thereby preventing video uploaders from exploiting regular acquisition methods and causing the video to be misclassified.

[0060] 202. The server determines the target cluster corresponding to the at least one image frame vector from at least two clusters, where a cluster is used to represent a class of image frame vectors.

[0061] In this embodiment, the at least two clusters can be obtained by the server based on the clustering of sample image frame vectors, or they can be obtained directly by the server. After obtaining the at least one image frame vector, the server can determine the target cluster corresponding to each image frame vector from the at least two clusters based on the similarity between each image frame vector and the cluster center vectors of the at least two clusters.

[0062] In one optional implementation, for any image frame vector among the at least one image frame vectors, the server determines the cluster with the highest similarity between its cluster center vector and the image frame vector from at least two clusters as the target cluster of the image frame vector. The similarity between the cluster center vector and the image frame vector can be represented by Euclidean distance. Accordingly, the step of the server determining the target clusters corresponding to the at least one image frame vector from at least two clusters can be as follows: for any image frame vector among the at least one image frame vectors, the server can determine the Euclidean distance between the image frame vector and the cluster center vectors of the at least two clusters respectively. The server can select the cluster with the smallest Euclidean distance between its cluster center vector and the image frame vector as the target cluster of the image frame vector. The server can obtain the target clusters corresponding to the at least one image frame vector in the same way, thereby obtaining at least one target cluster. Of course, the server can also determine the similarity between the cluster center vector and the image frame vector in other ways, such as through cosine similarity, Pearson correlation coefficient, and Manhattan distance. Since the cluster corresponding to the cluster center vector with the highest similarity to the image frame vector is selected as the target cluster, the vectors in the target cluster have a high similarity to the image frame vector, so the image frame vector can be represented by the cluster center vector of the target cluster.

[0063] For example, the server can use the search function provided by the Faiss search library to retrieve the cluster whose cluster center vector has the highest similarity to the image frame vector, i.e., the smallest Euclidean distance, from at least two clusters. The server can also use the k-Nearest Neighbor (kNN) algorithm to determine the cluster with the highest similarity to the image frame vector, i.e., the smallest Euclidean distance, from the cluster center vectors of k clusters. Here, k is a positive integer. Of course, the server can also choose other algorithms to determine the cluster whose cluster center vector is most similar to the image frame vector; this application embodiment does not limit this choice.

[0064] It should be noted that when the above-mentioned at least two clusters are obtained by the server based on multiple sample image frame vectors, the steps for obtaining the at least two clusters can be as follows: the server can acquire multiple sample image frames, input the acquired sample image frames into a convolutional neural network-based computational model to obtain multiple sample image frame vectors, and cluster these multiple sample image frame vectors to obtain at least two clusters. The server can also assign a unique cluster core identifier to each of the at least two clusters, and the cluster core identifier can be in the form of a numerical code. The server can cluster the sample image frame vectors using algorithms such as k-means, k-medoids, or CLARA (Clustering Large Applications), and this application embodiment does not impose any limitations on this.

[0065] For example, see Figure 3 As shown, Figure 3 This is a schematic diagram of a cluster center vector according to an embodiment of this application. The server calculates a large number of sample image frame vectors using the InceptionV3 model, generating a 1024-dimensional vector for each sample image frame. The number of sample image frames can be 100,000, 1 million, or 10 million, etc. Then, the server performs clustering based on the sample image frame vectors using the k-means algorithm, obtaining multiple clusters. The number of clusters can be freely set before clustering, such as 1000, 10000, or 100000, etc. Figure 3 The diagram illustrates three clusters, C1, C2, and C3, with cluster center vectors c1, c2, and c3 for C1 and c2 respectively. The other points represent sample image frame vectors. Correspondingly, for any given image frame, when obtaining the target cluster corresponding to that image frame, one can refer to... Figure 4 As shown, Figure 4 This is a schematic diagram of obtaining a target cluster according to an embodiment of this application. Figure 4 exist Figure 3 Based on this, an image frame vector t1 corresponding to an image frame T is also shown. The server calculates the Euclidean distance between t1 and c1, the Euclidean distance between t1 and c2, and the Euclidean distance between t1 and c3 respectively. Among them, the Euclidean distance between c2 and t1 is the smallest, so C2 is determined to be the target cluster corresponding to the image frame vector t1.

[0066] 203. The server obtains the text to be recognized corresponding to the target video based on the target clusters corresponding to at least one of the above image frame vectors.

[0067] In this embodiment, after obtaining the target clusters corresponding to at least one image frame vector, the server can obtain the cluster center identifier of each target cluster. Each target cluster corresponds to a unique cluster center identifier, which can be in character or number form, etc. The server can use the cluster center identifiers of the target clusters corresponding to the at least one image frame as words in the text to be recognized corresponding to the target video. In fact, the target video is treated as text, and at least one image frame vector in the target video is treated as at least one word in the text. By mapping the image frame vectors to cluster center identifiers, the mapping process from high-dimensional vectors to low-dimensional identifiers is realized, achieving data dimensionality reduction.

[0068] For example, if the server obtains 2160 image frames from a target video, resulting in 2160 image frame vectors, it then determines 2160 target clusters based on these vectors. Further, the server obtains 2160 cluster center identifiers, which may contain duplicates. These 2160 cluster center identifiers are then stored as a list of 2160 words. These 2160 words are equivalent to the words obtained from word segmentation of the text to be recognized corresponding to the target video.

[0069] In one optional implementation, the server can store a correspondence between cluster center identifiers and video types. For any image frame vector, after determining the target cluster corresponding to the image frame vector, the server can determine at least one classification type corresponding to the cluster center identifier of the target cluster based on the aforementioned correspondence, thereby determining the at least one classification type to which the image frame belongs. After determining the at least one classification type to which the at least one image frame belongs, the server can determine the video type to which the target video belongs based on the frequency of occurrence of each classification type. The server can select the classification type with the highest occurrence frequency as the video type of the target video, or it can select multiple classification types with high occurrence frequencies as the video type of the target video.

[0070] For example, if the server determines that at least one image frame in the target video belongs to the category of comedy, game, animation, movie, or educational, and comedy appears most frequently, then the server will classify the target video as comedy. Alternatively, if comedy, game, and educational appear more frequently, while animation and movie appear less frequently, the server can classify the target video as comedy, game, or educational.

[0071] 204. The server inputs the text to be recognized into the natural language processing model, and the natural language processing model decodes the text to be recognized to obtain the video type to which the target video belongs.

[0072] In this embodiment, the server can directly input the text to be recognized into a natural language processing (NLP) model. During training, this NLP model uses a large number of sample texts and their corresponding classification types. The sample text is obtained by acquiring a large number of sample videos, each containing at least one sample image frame. For any given sample image frame, the frame is input into a convolutional neural network-based computational model to obtain a sample image frame vector. Then, the target cluster corresponding to this vector is determined, and the cluster center identifier represents the corresponding sample image frame, thus obtaining the sample text for the sample video. The server can decode the input text based on the NLP model to obtain at least one classification probability, representing the probability that the text belongs to different classification types. Based on this at least one classification probability, the server determines the classification type that satisfies a first objective condition as the video type to which the target video belongs. This first objective condition can be the highest probability, a probability greater than a classification probability threshold, etc. The classification probability threshold can be 90%, 80%, 75%, etc., and this application does not impose any limitations on this.

[0073] For example, the server determines that the target video has a 90% probability of belonging to the comedy category, an 80% probability of belonging to the game category, a 30% probability of belonging to the anime category, a 20% probability of belonging to the movie category, and a 73% probability of belonging to the educational category. If only one category needs to be assigned to the target video, the server can choose comedy as the video category, given the highest probability. If multiple categories need to be assigned to the target video, the server can choose comedy, game, and educational categories, all with a probability greater than 60%, as the video categories for the target video.

[0074] It should be noted that the video classification method provided in this application can also be used to identify video quality, that is, to perform binary classification on the target video, with only two classification types: high quality and low quality. The server can obtain high-quality clusters by clustering the vectors of high-quality sample image frames included in high-quality videos, and low-quality clusters by clustering the vectors of low-quality sample image frames included in low-quality videos. Then, a natural language processing model is trained using the high-quality text corresponding to the high-quality video and the low-quality text corresponding to the low-quality video, so that the natural language processing model outputs the probability that the target video is high quality. When classifying the target video, the server can determine the high-quality cluster and low-quality cluster corresponding to at least one image frame of the target video, and based on these high-quality and low-quality clusters, obtain the text to be identified corresponding to the target video, thereby outputting the probability that the target video is high quality based on the natural language processing model, and determining whether the target video is high-quality or low-quality. Of course, the server can also choose not to use a natural language processing model, but instead statistically analyze the frequency of high-quality and low-quality clusters within the acquired target clusters. A higher frequency of high-quality clusters indicates a high-quality video, and a higher frequency of low-quality clusters indicates a low-quality video. By employing different training methods, the video classification method provided in this application can be used to achieve various classification tasks, thus expanding the applicability of the embodiments of this application.

[0075] In an optional implementation, the server can also add video tags to the target video using a tag model. Accordingly, the steps for the server to add video tags to the target video can be as follows: the server can input the text to be recognized into the tag model, decode the text to be recognized through the tag model, and obtain at least one tag probability. This tag probability represents the probability that the text to be recognized belongs to different video tags. The server can determine the video tags that satisfy a second objective condition as the video tags of the target video based on the at least one tag probability. The second objective condition can be the highest probability, a probability greater than a tag probability threshold, etc. The tag probability threshold can be 90%, 80%, 75%, etc., and this application does not limit this. Of course, the tag model can be the same model as the natural language processing model described above; that is, when the natural language model outputs at least one classification probability, it can also output at least one tag probability. This application embodiment does not limit this.

[0076] For example, the server identifies video categories such as comedy, games, anime, movies, and educational videos. The comedy category includes tags like "dumb jokes," "pranks," and "meme videos," while the game category includes tags like "Game A," "Game B," and "Game C." The anime category includes tags like "Anime D," "Anime E," and "Anime F." The server can further filter suitable video tags from the video tags based on the target video's category; this embodiment does not impose any limitations on this.

[0077] It should be noted that steps 201 to 204 above are optional implementations of the video classification method provided in this application. Correspondingly, this video classification method can also be implemented in other ways. For example, see... Figure 5 As shown, Figure 5 This is a flowchart of another video classification method provided according to embodiments of this disclosure. Figure 5 The video classification method can be divided into three steps. Step one: Obtain a large number of sample image frames from multiple sample videos, calculate the sample image frame vector for each sample image frame, and cluster the sample image frame vectors using the K-means algorithm to obtain the cluster center vectors of at least two clusters. Step two: Obtain at least one image frame from the target video, calculate the image frame vector for each image frame, and use the KNN algorithm to determine the cluster center vector most similar to each image frame vector from the cluster center vectors of the at least two clusters, obtaining at least one target cluster, and obtaining the cluster center identifier of the at least one target cluster. Step three: Determine the text to be identified corresponding to the target video based on the cluster center identifier of the at least one target cluster, and based on the text to be identified, perform at least one of the following actions using natural language processing: classify the target video, add video tags, or label video quality. This video classification method, by using the correspondence between the cluster center vectors of the target clusters and the image frame vectors, uses the cluster center identifier to replace the image frames as words in the text to be identified, transforming the processing of video image frames into the processing of words in the text, which can effectively reduce the computational load. For example, it can be applied to a Linux system with a 16-core CPU and 32GB of RAM, without requiring a GPU, to complete video classification.

[0078] In this embodiment of the application, by determining the target cluster corresponding to the image frame vector of the image frame in the target video, the text to be identified corresponding to the target video can be obtained. This allows the image frames in the target video to be converted into the text content in the text to be identified when classifying the target video, thereby reducing computational complexity, shortening processing time, and reducing the requirements on device processing capabilities.

[0079] Figure 6 This is a block diagram of a video classification apparatus according to an embodiment of this application. The apparatus is used to perform the steps of the video classification method described above, see below. Figure 6 The device includes: a vector acquisition module 601, a determination module 602, a text acquisition module 603, and a model processing module 604.

[0080] The vector acquisition module 601 is used to input at least one image frame obtained from the target video into a convolutional neural network-based computational model to obtain at least one image frame vector.

[0081] The determining module 602 is used to determine the target clusters corresponding to the at least one image frame vector from at least two clusters, wherein a cluster is used to represent a class of image frame vectors;

[0082] The text acquisition module 603 is used to acquire the text to be recognized corresponding to the target video based on the target clusters corresponding to the at least one image frame vector;

[0083] The model processing module 604 is used to input the text to be identified into the natural language processing model, and decode the text to be identified through the natural language processing model to obtain the video type to which the target video belongs.

[0084] In an alternative implementation, the determining module 602 is further configured to, for any image frame vector in the at least one image frame vector, determine the cluster with the highest similarity between the cluster center vector and the image frame vector from the at least two clusters as the target cluster of the image frame vector.

[0085] In one alternative implementation, the determining module 602 is further configured to determine the Euclidean distance between the image frame vector and the cluster center vectors of the at least two clusters respectively; in response to the Euclidean distance between the cluster center vector of any cluster and the image frame vector being the smallest, the cluster is taken as the target cluster of the image frame vector.

[0086] In one optional implementation, the text acquisition module 603 is further configured to use the cluster center identifiers of the target clusters corresponding to the at least one image frame vector as words in the text to be identified corresponding to the target video.

[0087] In one alternative implementation, the device further includes:

[0088] The vector acquisition module 601 is also used to input multiple sample image frames into the convolutional neural network-based operation model to obtain multiple sample image frame vectors;

[0089] The clustering module is used to cluster the multiple sample image frame vectors to obtain at least two clusters;

[0090] The allocation module is used to assign a unique cluster core identifier to each of the at least two clusters. The cluster core identifier is in the form of a numerical code.

[0091] In one alternative implementation, the model processing module 604 is used to decode the text to be identified through the natural language processing model to obtain at least one classification probability, which represents the probability that the text to be identified belongs to different classification types; and based on the at least one classification probability, to determine the classification type that satisfies the first target condition as the video type to which the target video belongs.

[0092] In an optional implementation, the model processing module 604 is further configured to input the text to be identified into a label model, and decode the text to be identified through the label model to obtain at least one label probability, which is used to represent the probability that the text to be identified belongs to different video labels.

[0093] The determining module 602 is further configured to determine, based on the at least one label probability, a video label that satisfies the second target condition as the video label of the target video.

[0094] In this embodiment of the application, by determining the target cluster corresponding to the image frame vector of the image frame in the target video, the text to be identified corresponding to the target video can be obtained. This allows the image frames in the target video to be converted into the text content in the text to be identified when classifying the target video, thereby reducing computational complexity, shortening processing time, and reducing the requirements on device processing capabilities.

[0095] It should be noted that the video classification device provided in the above embodiments is only illustrated by the division of the above functional modules when running the application. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the video classification device and the video classification method embodiments provided in the above embodiments belong to the same concept, and the specific implementation process can be found in the method embodiments, which will not be repeated here.

[0096] In the embodiments of this application, the electronic device can be implemented as a terminal or a server. When implemented as a terminal, the operation performed by the video classification method described above can be implemented by the terminal. When implemented as a server, the operation performed by the video classification method described above can be implemented by the server. Alternatively, the operation performed by the video classification method can be implemented through the interaction between the server and the terminal.

[0097] Electronic devices can be implemented as terminals. Figure 7 This is a structural block diagram of a terminal 700 provided according to an embodiment of this application. The terminal... Figure 7The diagram illustrates a structural block diagram of a terminal 700 provided in an exemplary embodiment of the present invention. The terminal 700 may be a smartphone, tablet computer, MP3 player (Moving Picture Experts Group Audio Layer III), MP4 player (Moving Picture Experts Group Audio Layer IV), laptop computer, or desktop computer. The terminal 700 may also be referred to as a user device, portable terminal, laptop terminal, desktop terminal, or other names.

[0098] Typically, terminal 700 includes a processor 701 and a memory 702.

[0099] Processor 701 may include one or more processing cores, such as a quad-core processor, an octa-core processor, etc. Processor 701 may be implemented using at least one hardware form selected from DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). Processor 701 may also include a main processor and a coprocessor. The main processor, also known as a CPU (Central Processing Unit), is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, processor 701 may integrate a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the screen. In some embodiments, processor 701 may also include an AI (Artificial Intelligence) processor, which is used to handle computational operations related to machine learning.

[0100] The memory 702 may include one or more computer-readable storage media, which may be non-transitory. The memory 702 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices or flash memory devices. In some embodiments, the non-transitory computer-readable storage media in the memory 702 are used to store at least one instruction, which is executed by the processor 701 to implement the XXXX method provided in the method embodiments of this application.

[0101] In some embodiments, the terminal 700 may also optionally include a peripheral device interface 703 and at least one peripheral device. The processor 701, memory 702, and peripheral device interface 703 can be connected via a bus or signal line. Each peripheral device can be connected to the peripheral device interface 703 via a bus, signal line, or circuit board. Specifically, the peripheral device includes at least one of the following: a radio frequency circuit 704, a display screen 705, a camera assembly 706, an audio circuit 707, a positioning assembly 708, and a power supply 709.

[0102] Peripheral device interface 703 can be used to connect at least one I / O (Input / Output) related peripheral device to processor 701 and memory 702. In some embodiments, processor 701, memory 702 and peripheral device interface 703 are integrated on the same chip or circuit board; in some other embodiments, any one or two of processor 701, memory 702 and peripheral device interface 703 can be implemented on separate chips or circuit boards, which is not limited in this embodiment.

[0103] The radio frequency (RF) circuit 704 is used to receive and transmit RF (Radio Frequency) signals, also known as electromagnetic signals. The RF circuit 704 communicates with communication networks and other communication devices via electromagnetic signals. The RF circuit 704 converts electrical signals into electromagnetic signals for transmission, or converts received electromagnetic signals back into electrical signals. Optionally, the RF circuit 704 includes: an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, a user identity module card, etc. The RF circuit 704 can communicate with other terminals through at least one wireless communication protocol. This wireless communication protocol includes, but is not limited to: metropolitan area networks (MANs), various generations of mobile communication networks (2G, 3G, 4G, and 5G), wireless local area networks (WLANs), and / or WiFi (Wireless Fidelity) networks. In some embodiments, the RF circuit 704 may also include circuitry related to NFC (Near Field Communication), which is not limited in this application.

[0104] Display screen 705 is used to display a UI (User Interface). This UI may include graphics, text, icons, videos, and any combination thereof. When display screen 705 is a touch display screen, it also has the ability to collect touch signals on or above its surface. These touch signals can be input as control signals to processor 701 for processing. In this case, display screen 705 can also be used to provide virtual buttons and / or a virtual keyboard, also known as soft buttons and / or a soft keyboard. In some embodiments, there may be one display screen 705, which serves as the front panel of terminal 700; in other embodiments, there may be at least two display screens 705, respectively disposed on different surfaces of terminal 700 or in a folded design; in still other embodiments, display screen 705 may be a flexible display screen, disposed on a curved or folded surface of terminal 700. Furthermore, display screen 705 may be configured as a non-rectangular, irregular shape, i.e., a non-rectangular screen. Display screen 705 may be made of materials such as LCD (Liquid Crystal Display) or OLED (Organic Light-Emitting Diode).

[0105] The camera assembly 706 is used to acquire images or videos. Optionally, the camera assembly 706 includes a front-facing camera and a rear-facing camera. Typically, the front-facing camera is located on the front panel of the terminal, and the rear-facing camera is located on the back of the terminal. In some embodiments, there are at least two rear-facing cameras, which are any one of a main camera, a depth-sensing camera, a wide-angle camera, and a telephoto camera, to achieve background blurring by fusion of the main camera and the depth-sensing camera, panoramic shooting by fusion of the main camera and the wide-angle camera, VR (Virtual Reality) shooting, or other fusion shooting functions. In some embodiments, the camera assembly 706 may also include a flash. The flash can be a single-color temperature flash or a dual-color temperature flash. A dual-color temperature flash refers to a combination of a warm-light flash and a cool-light flash, which can be used for light compensation at different color temperatures.

[0106] The audio circuit 707 may include a microphone and a speaker. The microphone is used to collect sound waves from the user and the environment, converting the sound waves into electrical signals that are input to the processor 701 for processing, or input to the radio frequency circuit 704 for voice communication. For stereo sound acquisition or noise reduction purposes, multiple microphones may be used, each located at a different part of the terminal 700. The microphone may also be an array microphone or an omnidirectional microphone. The speaker is used to convert the electrical signals from the processor 701 or the radio frequency circuit 704 into sound waves. The speaker may be a conventional diaphragm speaker or a piezoelectric ceramic speaker. When the speaker is a piezoelectric ceramic speaker, it can convert electrical signals not only into audible sound waves but also into inaudible sound waves for purposes such as distance measurement. In some embodiments, the audio circuit 707 may also include a headphone jack.

[0107] The positioning component 708 is used to determine the current geographic location of the terminal 700 in order to enable navigation or LBS (Location Based Service). The positioning component 708 can be a positioning component based on the US GPS (Global Positioning System), China's BeiDou system, Russia's Granas system, or the EU's Galileo system.

[0108] The power supply 709 is used to power the various components in the terminal 700. The power supply 709 can be AC ​​power, DC power, a disposable battery, or a rechargeable battery. When the power supply 709 includes a rechargeable battery, the rechargeable battery can support wired or wireless charging. The rechargeable battery can also be used to support fast charging technology.

[0109] In some embodiments, the terminal 700 further includes one or more sensors 710. The one or more sensors 710 include, but are not limited to: an accelerometer 711, a gyroscope 712, a pressure sensor 713, a fingerprint sensor 714, an optical sensor 715, and a proximity sensor 716.

[0110] Accelerometer 711 can detect the magnitude of acceleration along the three coordinate axes of a coordinate system established by terminal 700. For example, accelerometer 711 can be used to detect the components of gravitational acceleration along the three coordinate axes. Processor 701 can control display screen 705 to display the user interface in either a landscape or portrait view based on the gravitational acceleration signal acquired by accelerometer 711. Accelerometer 711 can also be used for games or for acquiring user motion data.

[0111] The gyroscope sensor 712 can detect the orientation and rotation angle of the terminal 700. The gyroscope sensor 712, in conjunction with the accelerometer sensor 711, can collect 3D motion data from the user on the terminal 700. Based on the data collected by the gyroscope sensor 712, the processor 701 can perform the following functions: motion sensing (e.g., changing the UI based on the user's tilt), image stabilization during shooting, game control, and inertial navigation.

[0112] The pressure sensor 713 can be disposed on the side bezel of the terminal 700 and / or the lower layer of the display screen 705. When the pressure sensor 713 is disposed on the side bezel of the terminal 700, it can detect the user's grip signal on the terminal 700, and the processor 701 can perform left / right hand recognition or quick operation based on the grip signal collected by the pressure sensor 713. When the pressure sensor 713 is disposed on the lower layer of the display screen 705, the processor 701 can control the operable controls on the UI interface based on the user's pressure operation on the display screen 705. The operable controls include at least one of button controls, scroll bar controls, icon controls, and menu controls.

[0113] The fingerprint sensor 714 is used to collect a user's fingerprint. The processor 701 identifies the user based on the fingerprint collected by the fingerprint sensor 714, or vice versa. When the user's identity is identified as trusted, the processor 701 authorizes the user to perform relevant sensitive operations, including unlocking the screen, viewing encrypted information, downloading software, making payments, and changing settings. The fingerprint sensor 714 can be located on the front, back, or side of the terminal 700. When the terminal 700 has a physical button or manufacturer logo, the fingerprint sensor 714 can be integrated with the physical button or manufacturer logo.

[0114] An optical sensor 715 is used to collect ambient light intensity. In one embodiment, the processor 701 can control the display brightness of the display screen 705 based on the ambient light intensity collected by the optical sensor 715. Specifically, when the ambient light intensity is high, the display brightness of the display screen 705 is increased; when the ambient light intensity is low, the display brightness of the display screen 705 is decreased. In another embodiment, the processor 701 can also dynamically adjust the shooting parameters of the camera assembly 706 based on the ambient light intensity collected by the optical sensor 715.

[0115] The proximity sensor 716, also known as a distance sensor, is typically located on the front panel of the terminal 700. The proximity sensor 716 is used to detect the distance between the user and the front of the terminal 700. In one embodiment, when the proximity sensor 716 detects that the distance between the user and the front of the terminal 700 is gradually decreasing, the processor 701 controls the display screen 705 to switch from a screen-on state to a screen-off state; when the proximity sensor 716 detects that the distance between the user and the front of the terminal 700 is gradually increasing, the processor 701 controls the display screen 705 to switch from a screen-off state to a screen-on state.

[0116] Those skilled in the art will understand that Figure 7 The structure shown does not constitute a limitation on terminal 700, and may include more or fewer components than shown, or combine certain components, or use different component arrangements.

[0117] Electronic devices can be implemented as servers. Figure 8 This is a schematic diagram of a server structure according to an embodiment of this application. The server 800 can vary significantly due to different configurations or performance. It may include one or more Central Processing Units (CPUs) 801 and one or more memories 802. The memory 802 stores at least one instruction, which is loaded and executed by the processor 801 to implement the methods provided in the various method embodiments described above. Of course, the server may also have wired or wireless network interfaces, a keyboard, and input / output interfaces for input and output. The server may also include other components for implementing device functions, which will not be elaborated upon here.

[0118] This application also provides a computer-readable storage medium applied to an electronic device. The computer-readable storage medium stores at least one piece of program code, which is executed by a processor to implement the operations performed by the electronic device in the video classification method of this application.

[0119] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0120] The above description is merely an optional embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A method of video classification, characterized by, The method comprises: inputting a plurality of image frames obtained from a target video into a convolutional neural network-based operation model to obtain a plurality of image frame vectors; determining a series of target clusters corresponding to the plurality of image frame vectors from at least two clusters, wherein one cluster is used to represent a type of image frame vector; based on the series of target clusters, for any image frame vector, taking a cluster center identifier of a target cluster corresponding to the image frame vector as a word in a to-be-recognized text corresponding to the target video, wherein the to-be-recognized text comprises a plurality of words, and each word corresponds to an image frame vector of an image frame in the target video; inputting the to-be-recognized text into a natural language processing model, decoding the to-be-recognized text through the natural language processing model, and obtaining a video type to which the target video belongs.

2. The method of claim 1, wherein, The method comprises: for any image frame vector in the plurality of image frame vectors, determining a cluster with the highest similarity between a cluster center vector and the image frame vector from the at least two clusters as a target cluster of the image frame.

3. The method of claim 2, wherein, The method comprises: respectively determining the Euclidean distance between the image frame vector and the cluster center vectors of the at least two clusters; in response to the Euclidean distance between the cluster center vector of any cluster and the image frame vector being the smallest, taking the cluster as the target cluster of the image frame vector.

4. The method of claim 1, wherein, Before the method comprises determining a series of target clusters corresponding to the plurality of image frame vectors from at least two clusters, the method further comprises: inputting a plurality of sample image frames obtained into the convolutional neural network-based operation model to obtain a plurality of sample image frame vectors; clustering the plurality of sample image frame vectors to obtain at least two clusters; assigning unique cluster center identifiers to the at least two clusters, wherein the cluster center identifiers are in the form of numerical codes.

5. The method of claim 1, wherein, The method comprises: decoding the to-be-recognized text through the natural language processing model to obtain at least one classification probability, wherein the classification probability is used to represent the probability that the to-be-recognized text belongs to different classification types; determining a classification type that meets a first target condition as the video type to which the target video belongs according to the at least one classification probability.

6. The method of claim 1, wherein, After the method comprises decoding the to-be-recognized text through the natural language processing model to obtain the video type to which the target video belongs, the method further comprises: inputting the to-be-recognized text into a label model, decoding the to-be-recognized text through the label model to obtain at least one label probability, wherein the label probability is used to represent the probability that the to-be-recognized text belongs to different video labels; determining a video label that meets a second target condition as the video label of the target video according to the at least one label probability.

7. A video classification apparatus characterized by comprising: The device comprises: The vector obtaining module is configured to input a plurality of image frames obtained from a target video into a convolutional neural network-based operation model to obtain a plurality of image frame vectors. The determining module is configured to determine a series of target clusters corresponding to the plurality of image frame vectors respectively from at least two clusters, and one cluster is used to represent one type of image frame vector. The text obtaining module is configured to, based on the series of target clusters, for any image frame vector, take a cluster center identifier of a target cluster corresponding to the image frame vector as one vocabulary in a to-be-identified text corresponding to the target video, and the to-be-identified text includes a plurality of vocabularies, and each vocabulary corresponds to an image frame vector of an image frame in the target video. The model processing module is configured to input the to-be-identified text into a natural language processing model, decode the to-be-identified text through the natural language processing model, and obtain a video type to which the target video belongs.

8. An electronic device, comprising: The electronic device includes a processor and a memory, the memory is used to store at least one segment of program code, the at least one segment of program code is loaded and executed by the processor, and the video classification method in any one of claims 1 to 6 is executed.

9. A storage medium, characterized by The storage medium is used to store at least one segment of program code, and the at least one segment of program code is used to execute the video classification method in any one of claims 1 to 6.

Images