Video classification method and apparatus, electronic device, and storage medium

Abstract

This invention provides a video classification method, apparatus, electronic device, and storage medium. The method includes: acquiring a target video to be processed; identifying target audio features and video frame features from the target video using an artificial intelligence model; using the target audio features as input to a pre-trained feature classification model to obtain multiple audio features; wherein the feature classification model includes multiple parallel branch models, each branch model being obtained by training a convolutional neural network with audio feature samples carrying a label type; and determining the video category of the target video based on the multiple audio features and the video frame features. This invention improves the accuracy of video classification by performing fine-grained segmentation of the audio features in the target video and combining the obtained classification results with the video frame features of the target video to determine the video category.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence, and more specifically, to a video classification method, apparatus, electronic device, and storage medium. Background Technology

[0002] In recent years, with the rapid development of internet technology, internet social platforms have also developed rapidly. Various live streaming platforms generate tens of millions of short videos every minute, encompassing thousands of categories. Accurately classifying these short videos is a crucial issue. With the emergence of convolutional neural networks and the continuous development of artificial intelligence technology, research at the video level is increasing, and multimodal technologies such as video, audio, and text are constantly evolving. Only by correctly classifying short videos can we provide different experiences for different users, thus better serving them.

[0003] Existing video classification methods involve averaging extracted video features to obtain a single feature, which is then used for classification. However, this approach has the drawback of failing to learn the relationships between features. The averaging of all features is too brute-force, resulting in a lack of emphasis among the features and leading to inaccurate video classification results. Summary of the Invention

[0004] To solve the above-mentioned technical problems, or at least partially solve them, the present invention provides a video classification method, apparatus, electronic device, and storage medium.

[0005] In a first aspect, the present invention provides a video classification method, the method comprising: acquiring a target video to be processed; identifying target audio features and video frame features from the target video using an artificial intelligence model; using the target audio features as input to a pre-trained feature classification model to obtain multiple audio features; wherein the feature classification model includes multiple parallel branch models, each branch model being obtained by training a convolutional neural network with audio feature samples carrying a label type; and determining the video category of the target video based on the multiple audio features and the video frame features.

[0006] Optionally, identifying target audio features from the target video using an artificial intelligence model includes: inputting the target video into a speech recognition-based artificial intelligence model to identify audio frames contained in the target video to obtain an audio frame sequence; drawing a spectrogram corresponding to the audio frame sequence; and extracting the target audio features by inputting the spectrogram into a preset convolutional neural network.

[0007] Optionally, identifying video frame features from the target video using an artificial intelligence model includes: extracting video frame images from the target video based on a preset frame extraction strategy; and using the video frame images as input to an image classification-based artificial intelligence model to obtain the video frame features.

[0008] Optionally, the target audio features are used as input to a pre-trained feature classification model to obtain multiple audio features. Each audio feature is obtained in the following way: a first branch model that has not been selected among multiple branch models is selected. The first branch model is obtained by training a convolutional neural network with audio feature samples carrying a label type of the first category. The target audio features are used as input to the first branch model and feature extraction is performed through a fully connected layer in the convolutional neural network to separate audio features with the label type of the first category from the target audio features.

[0009] Optionally, determining the video category of the target video based on the multiple audio features and the video frame features includes: inputting a first audio feature that has not been selected from the multiple audio features and the video frame features into a convolutional neural network to perform feature fusion, obtaining multiple first features corresponding to the multiple audio features; inputting the multiple first features into a logistic regression model to obtain multiple first weights; wherein the first weights are used to characterize the degree of influence of each audio feature on the video category of the target video; performing a multiplication operation on each first weight and its corresponding audio feature to obtain multiple second features; performing a summation operation on the multiple second features to obtain a third feature; and identifying the third feature as the video category of the target video.

[0010] Optionally, after extracting video frame images from the target video based on a preset frame extraction strategy, the method further includes: recognizing the text in the video frame images using a character recognition-based artificial intelligence model to obtain target text; and using a BERT model to extract text features from the target text to characterize the label type of the target text.

[0011] Optionally, determining the video category of the target video based on the multiple audio features and the video frame features includes: inputting a second audio feature that has not been selected from the multiple audio features and the video frame features into a convolutional neural network for feature fusion to obtain a fourth feature corresponding to the multiple audio features; inputting the multiple fourth features into a logistic regression model to obtain multiple second weights; wherein the second weights are used to characterize the degree of influence of each audio feature on the video category of the target video; performing a multiplication operation on each second weight and its corresponding audio feature to obtain multiple fifth features; performing a summation operation on the multiple fifth features to obtain a sixth feature; generating a seventh feature by concatenating the sixth feature, the text feature, and the video frame feature; and identifying the seventh feature as the video category of the target video.

[0012] Secondly, the present invention provides a video classification apparatus, the apparatus comprising: an acquisition module for acquiring a target video to be processed; an identification module for identifying target audio features and video frame features from the target video using an artificial intelligence model; a classification module for using the target audio features as input to a pre-trained feature classification model to obtain multiple audio features; wherein the feature classification model includes multiple parallel branch models, each branch model being obtained by training a convolutional neural network with audio feature samples carrying a label type; and a determination module for determining the video category of the target video based on the multiple audio features and the video frame features.

[0013] Optionally, the recognition module includes: a first recognition unit, configured to input the target video into an artificial intelligence model based on speech recognition to recognize the audio frames contained in the target video and obtain an audio frame sequence; a drawing unit, configured to draw a spectrogram corresponding to the audio frame sequence; and a first extraction unit, configured to extract the target audio features by inputting the spectrogram into a preset convolutional neural network.

[0014] Optionally, the recognition module includes: an extraction unit, used to extract video frame images from the target video based on a preset frame extraction strategy; and a second extraction unit, used to use the video frame images as input to an image classification-based artificial intelligence model to obtain the video frame features.

[0015] Optionally, the classification module obtains each audio feature in the following way: a classification unit is used to select a first branch model that has not been selected among multiple branch models. The first branch model is obtained by training a convolutional neural network with audio feature samples carrying a label type of the first type of label. The target audio feature is used as the input of the first branch model and the feature is extracted through a fully connected layer in the convolutional neural network to separate the audio feature with the label type of the first type of label from the target audio feature.

[0016] Optionally, the determining module includes: a first calculation unit, configured to input the first audio features that have not been selected from the multiple audio features and the video frame features into a convolutional neural network for feature fusion to obtain multiple first features corresponding to the multiple audio features; a second calculation unit, configured to input the multiple first features into a logistic regression model to obtain multiple first weights; wherein the first weights are used to characterize the degree of influence of each audio feature on the video category of the target video; a third calculation unit, configured to perform multiplication operations on each first weight and the corresponding audio feature to obtain multiple second features; a fourth calculation unit, configured to perform summation operations on the multiple second features to obtain a third feature; and a first identification unit, configured to identify the third feature as the video category of the target video.

[0017] Optionally, the recognition module further includes: a second recognition unit, configured to, after the extraction unit extracts video frame images from the target video based on a preset frame extraction strategy, recognize the text in the video frame images using a character recognition-based artificial intelligence model to obtain the target text; and a third extraction unit, configured to, using a BERT model to extract text features from the target text that characterize the label type of the target text.

[0018] Optionally, the determining module includes: a fifth calculation unit, used to input the second audio features that have not been selected from the multiple audio features and the video frame features into a convolutional neural network for feature fusion to obtain multiple fourth features corresponding to the multiple audio features; a sixth calculation unit, used to input the multiple fourth features into a logistic regression model to obtain multiple second weights; wherein the second weights are used to characterize the degree of influence of each audio feature on the video category of the target video; a seventh calculation unit, used to perform multiplication operations on each second weight and the corresponding audio feature to obtain multiple fifth features; an eighth calculation unit, used to perform summation operations on the multiple fifth features to obtain a sixth feature; a feature concatenation unit, used to generate a seventh feature by concatenating the sixth feature, the text feature, and the video frame feature; and a second identification unit, used to identify the seventh feature as the video category of the target video.

[0019] Thirdly, the present invention provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the above-described method.

[0020] Fourthly, the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the above-described method.

[0021] The technical solution of this invention can be applied to the field of deep learning technology based on computer vision. Compared with related technologies, the technical solution provided in this invention has the following advantages:

[0022] The video classification method provided in this invention identifies target audio features and video frame features from the target video using an artificial intelligence model. Then, it uses a pre-trained feature classification model to perform fine-grained segmentation of the audio features to obtain audio features with multiple label types. The video category of the target video is determined using the classified audio features and video frame features, which improves the accuracy of video classification and solves the technical problem of inaccurate video classification in related technologies. Attached Figure Description

[0023] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0024] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0025] Figure 1 A flowchart illustrating a video classification method provided in an embodiment of the present invention;

[0026] Figure 2 This is a schematic diagram of short video classification provided in a specific embodiment of the present invention;

[0027] Figure 3 This is a schematic diagram of the structure of a video classification device provided in an embodiment of the present invention;

[0028] Figure 4 This is a schematic diagram of the structure of a computer device provided in an embodiment of the present invention. Detailed Implementation

[0029] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0030] Firstly, embodiments of the present invention provide a video classification method. Figure 1 This is a flowchart illustrating a video classification method provided in an embodiment of the present invention, as shown below. Figure 1 As shown, the method includes the following steps:

[0031] Step S102: Obtain the target video to be processed;

[0032] It is understood that the executing entity of this invention can be a video classification device or a server, and the specific implementation is not limited here. This embodiment of the invention uses a server as the executing entity for illustration. The server can be an independent server or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms. This embodiment of the invention can acquire and process relevant data based on artificial intelligence technology. Artificial intelligence (AI) is the theory, method, technology, and application system that uses digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use knowledge to obtain optimal results.

[0033] Specifically, the server searches for the target video to be processed from a pre-built video database.

[0034] Step S104: Identify target audio features and video frame features from the target video using an artificial intelligence model;

[0035] In specific implementation, step S104 includes: inputting the target video into an artificial intelligence model based on speech recognition to identify the audio frames contained in the target video and obtain an audio frame sequence; drawing a spectrogram corresponding to the audio frame sequence; and extracting target audio features by inputting the spectrogram into a preset convolutional neural network.

[0036] In this embodiment, an artificial intelligence model based on speech recognition, such as an automated speech recognition model, is used to extract audio files (i.e., the aforementioned audio frame sequence) from the target video. Then, the corresponding spectrogram can be extracted from the audio file. A convolutional neural network (e.g., ResNet18) can be used to extract the audio features corresponding to the spectrogram. The spectrogram is a display image of time-related Fourier analysis, reflecting the changes in the audio signal spectrum over time. The horizontal axis of the spectrogram represents time, the vertical axis represents frequency, and the coordinate point value is the speech data energy. Since the spectrogram uses a two-dimensional plane to represent three-dimensional information, the energy value is represented by color; a darker color indicates stronger speech energy at that point.

[0037] In another embodiment, step S104 further includes: extracting video frame images from the target video based on a preset frame extraction strategy; and using the video frame images as input to an image classification-based artificial intelligence model to obtain video frame features.

[0038] In an optional example of this embodiment, after the short video is extracted into frames according to a certain frame extraction strategy, features are extracted from the video frames using ResNet50. After extracting features frame by frame, the frame features are then spliced ​​together according to the temporal order of the frames to form a video frame feature.

[0039] Step S106: Use the target audio features as input to the pre-trained feature classification model to obtain multiple audio features;

[0040] The feature classification model includes multiple parallel branch models, each of which is obtained by training a convolutional neural network with audio feature samples carrying a label type.

[0041] In this embodiment, target audio features in the spectrogram can be extracted using a convolutional neural network (e.g., ResNet18). However, audio in short videos may contain multiple sounds, which can be categorized into at least three types: no sound, valuable sounds (e.g., main sounds), and background sounds. When extracting the spectrogram from the audio, all sounds are mixed together. In reality, only the valuable parts of the sound are useful to us, such as the main sounds of a human voice. Therefore, it is necessary to classify the extracted audio features to better utilize the audio signal and perform fine-grained segmentation of the audio signal, thereby extracting the sounds that are valuable to the user.

[0042] In specific implementation, step S106 above obtains each audio feature in the following way: selecting the first branch model that has not been selected among multiple branch models. The first branch model is obtained by training a convolutional neural network with audio feature samples carrying the label type of the first type of label. The target audio feature is used as the input of the first branch model and the feature is extracted through the fully connected layer in the convolutional neural network to separate the audio feature with the label type of the first type of label from the target audio feature.

[0043] In one example of this embodiment, the above-mentioned feature classification model can be constructed by selecting audio feature samples carrying the label "no sound", audio feature samples carrying the label "main sound", and audio feature samples carrying the label "background sound" from a preset video database, inputting the three types of training feature samples into three convolutional neural networks for training, and obtaining three parallel branch models.

[0044] Furthermore, the target audio features are copied into three copies and input into the three branch models constructed above. Feature learning is then performed using fully connected layers of a convolutional neural network to classify the target audio features into three categories: audio features labeled "no sound," audio features labeled "main sound," and audio features labeled "background sound." This embodiment, by performing fine-grained segmentation of the audio signal, reduces noise and makes short video classification more accurate.

[0045] Step S108: Determine the video category of the target video based on various audio features and video frame features.

[0046] The video classification method provided in this invention identifies target audio features and video frame features from the target video using an artificial intelligence model. Then, it uses a pre-trained feature classification model to perform fine-grained segmentation of the audio features to obtain audio features with multiple label types. The video category of the target video is determined using the classified audio features and video frame features, which improves the accuracy of video classification and solves the technical problem of inaccurate video classification in related technologies.

[0047] In an optional embodiment of this case, step S108 includes: inputting a first audio feature that has not been selected from multiple audio features and video frame features into a convolutional neural network to perform feature fusion and obtain multiple first features corresponding to multiple audio features; inputting the multiple first features into a logistic regression model to obtain multiple first weights; wherein, the first weights are used to characterize the degree of influence of each audio feature on the video category of the target video; performing a multiplication operation on each first weight and the corresponding audio feature to obtain multiple second features; performing a summation operation on the multiple second features to obtain a third feature; and identifying the third feature as the video category of the target video.

[0048] For example, Figure 2 This is a schematic diagram of short video classification provided in a specific embodiment of the present invention, such as... Figure 2 As shown, an audio frame sequence is extracted from a short video using a convolutional neural network (ResNet18) and the corresponding audio spectrogram is generated. Then, the label feature vector (i.e. the target audio feature mentioned above) is extracted from the audio spectrogram through a convolutional layer. At the same time, after the short video is extracted into frames according to a certain frame extraction strategy, the features of the video frames are extracted using ResNet50. After extracting features frame by frame, the frame features are concatenated together according to the temporal order of the frames to form a video frame feature.

[0049] Furthermore, the label feature vectors are input into a pre-trained feature classification model, which includes three parallel branch models. These models are trained using audio feature samples carrying the label "no sound", audio feature samples carrying the label "main sound", and audio feature samples carrying the label "background sound". After passing through fully connected layers (convolutional layer-convolutional layer), the audio features of the three label types are output.

[0050] Then, the audio features of each label type are added to the video frame features. Convolutional neural networks are used to extract features from the audio features and video frame features separately, so that the audio features and video frame features have the same vector dimension. Then, the softmax function (i.e., the logistic regression model mentioned above) is used to pass the three values ​​obtained by the softmax function (i.e., the first weight mentioned above). These three values ​​are multiplied by the original three audio features (to obtain the second feature mentioned above) and then summed (to obtain the third feature mentioned above). The third feature is then used to identify the video category of the short video. The final third feature is used for classification. Thus, based on the relative weights of the three audio features in the short video, the final video category of the video is calculated, resulting in a more accurate classification of the short video.

[0051] In another embodiment of this case, after extracting video frame images from the target video based on a preset frame extraction strategy, the method further includes: recognizing the text in the video frame images using a character recognition-based artificial intelligence model to obtain the target text; and using a BERT model to extract text features from the target text to represent the tag type of the target text.

[0052] Further, step S108 includes: inputting the second audio features that have not been selected from the multiple audio features and the video frame features into a convolutional neural network to perform feature fusion, thereby obtaining multiple fourth features corresponding to the multiple audio features; inputting the multiple fourth features into a logistic regression model to obtain multiple second weights; wherein, the second weights are used to characterize the degree of influence of each audio feature on the video category of the target video; performing multiplication operations on each second weight and the corresponding audio feature respectively to obtain multiple fifth features; performing summation operations on the multiple fifth features to obtain a sixth feature; generating a seventh feature by concatenating the sixth feature, the text features, and the video frame features; and identifying the seventh feature as the video category of the target video.

[0053] In this embodiment, based on video frames, the OCR text (i.e., the target text mentioned above) of the video can be extracted using a character recognition model, and the corresponding text features can be extracted using a BERT model; then, it participates in... Figure 2 As shown, the audio features of each label type are added to the video frame features. The audio features and video frame features are extracted separately by a convolutional neural network so that the audio features and video frame features have the same vector dimension. Then, the softmax function (i.e., the logistic regression model mentioned above) is used. The three values ​​obtained by the softmax function (i.e., the second weight mentioned above) are multiplied with the original three audio features (to obtain the fifth feature mentioned above) and then summed (to obtain the sixth feature mentioned above).

[0054] Furthermore, the sixth feature, the aforementioned text features, and the video frame features are concatenated, and the concatenated features are used to identify the short video before classification. Alternatively, a convolutional neural network can be used to extract features from the sixth feature, the aforementioned text features, and the video frame features, ensuring that the vector dimensions of the three features are the same. These features are then added together, and the resulting feature is used to identify the short video.

[0055] By implementing the above steps, audio signals can be divided into fine-grained segments, reducing noise in the audio signals. The implementation method is simple, which not only reduces training costs but also enables accurate classification of short videos, allowing for better recommendation of favorite short videos to users on internet platforms.

[0056] Secondly, based on the video classification methods provided in the above embodiments, and based on the same inventive concept, this embodiment also provides a video classification device for implementing the above embodiments and preferred embodiments, which will not be repeated hereafter. As used below, the term "module" can refer to a combination of software and / or hardware that performs a predetermined function. Although the device described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0057] Figure 3 This is a schematic diagram of the structure of a video classification device provided in an embodiment of the present invention, as shown below. Figure 3 As shown, the device includes: an acquisition module 30 for acquiring a target video to be processed; an identification module 32 connected to the acquisition module 30 for identifying target audio features and video frame features from the target video using an artificial intelligence model; a classification module 34 connected to the identification module 32 for using the target audio features as input to a pre-trained feature classification model to obtain multiple audio features; wherein the feature classification model includes multiple parallel branch models, each branch model being obtained by training a convolutional neural network with audio feature samples carrying a label type; and a determination module 36 connected to the classification module 34 for determining the video category of the target video based on the multiple audio features and video frame features.

[0058] Optionally, the recognition module 32 includes: a first recognition unit, used to input the target video into an artificial intelligence model based on speech recognition to recognize the audio frames contained in the target video and obtain an audio frame sequence; a drawing unit, used to draw the spectrogram corresponding to the audio frame sequence; and a first extraction unit, used to extract the target audio features by inputting the spectrogram into a preset convolutional neural network.

[0059] Optionally, the recognition module 32 includes: an extraction unit for extracting video frame images from the target video based on a preset frame extraction strategy; and a second extraction unit for using the video frame images as input to an image classification-based artificial intelligence model to obtain video frame features.

[0060] Optionally, the classification module 34 obtains each audio feature in the following way: a classification unit is used to select the first branch model that has not been selected among multiple branch models. The first branch model is obtained by training a convolutional neural network with audio feature samples carrying the label type of the first type of label. The target audio feature is used as the input of the first branch model and the feature is extracted through the fully connected layer in the convolutional neural network to separate the audio feature with the label type of the first type of label from the target audio feature.

[0061] Optionally, the determining module 36 includes: a first calculation unit, used to input the first audio features that have not been selected from multiple audio features and video frame features into a convolutional neural network to perform feature fusion and obtain multiple first features corresponding to the multiple audio features; a second calculation unit, used to input the multiple first features into a logistic regression model to obtain multiple first weights; wherein, the first weights are used to characterize the degree of influence of each audio feature on the video category of the target video; a third calculation unit, used to perform multiplication operations on each first weight and the corresponding audio feature respectively to obtain multiple second features; a fourth calculation unit, used to perform summation operations on the multiple second features to obtain a third feature; and a first identification unit, used to identify the third feature as the video category of the target video.

[0062] Optionally, the recognition module 32 further includes: a second recognition unit, used to recognize the text in the video frame images by a character recognition-based artificial intelligence model after the extraction unit extracts video frame images from the target video based on a preset frame extraction strategy, thereby obtaining the target text; and a third extraction unit, used to extract text features that represent the label type of the target text from the target text using a BERT model.

[0063] Optionally, the determining module 36 includes: a fifth calculation unit, used to input the second audio features that have not been selected from multiple audio features and video frame features into a convolutional neural network for feature fusion to obtain multiple fourth features corresponding to multiple audio features; a sixth calculation unit, used to input the multiple fourth features into a logistic regression model to obtain multiple second weights; wherein the second weights are used to characterize the degree of influence of each audio feature on the video category of the target video; a seventh calculation unit, used to perform multiplication operations on each second weight and the corresponding audio feature respectively to obtain multiple fifth features; an eighth calculation unit, used to perform summation operations on the multiple fifth features to obtain a sixth feature; a feature concatenation unit, used to generate a seventh feature by concatenating the sixth feature, text features and video frame features; and a second identification unit, used to identify the seventh feature as the video category of the target video.

[0064] It should be noted that the above modules can be implemented by software or hardware. For the latter, they can be implemented in the following ways, but are not limited to: all the above modules are located in the same processor; or, the above modules are located in different processors in any combination.

[0065] Thirdly, embodiments of the present invention provide a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the statement translation method provided in the first aspect.

[0066] Figure 4This is a schematic diagram of the structure of a computer device provided in an embodiment of the present invention. Figure 4 As shown, the computer device includes a processor, memory, network interface, input device, and display screen connected via a system bus. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores an operating system and may also store computer programs. When executed by the processor, these programs enable the processor to implement a video classification method. The internal memory may also store computer programs, which, when executed by the processor, enable the processor to implement the video classification method. The display screen can be an LCD screen or an e-ink screen. The input device can be a touch layer covering the display screen, buttons, a trackball, or a touchpad mounted on the computer device's casing, or an external keyboard, touchpad, or mouse.

[0067] Those skilled in the art will understand that Figure 4 The structure shown is merely a block diagram of a portion of the structure related to the present invention and does not constitute a limitation on the computer device to which the present invention is applied. A specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0068] In one embodiment, the video classification device provided by the present invention can be implemented as a computer program, and the computer program can be implemented as follows: Figure 4 It runs on the computer device shown. The computer device's memory can store the various program modules that make up the translation device, for example, Figure 3 The acquisition module 30, identification module 32, classification module 34, and determination module 36 are shown. The computer program comprised of these modules causes the processor to execute the steps in the video classification methods of the various embodiments of the present invention described in this specification.

[0069] For example, Figure 4 The computer equipment shown can be used as follows Figure 3 The acquisition module 30 in the video classification device shown acquires the target video to be processed; the computer device can use the recognition module 32 to identify target audio features and video frame features from the target video using an artificial intelligence model; the computer device can use the classification module 34 to use the target audio features as input to a pre-trained feature classification model to obtain multiple audio features; wherein, the feature classification model includes multiple parallel branch models, each branch model is obtained by training a convolutional neural network with audio feature samples carrying a label type; the computer device can use the determination module 36 to determine the video category of the target video based on the multiple audio features and the video frame features.

[0070] It is understood that the explanations, examples, and beneficial effects of the computer device provided in the embodiments of the present invention can be referred to the corresponding parts of the first aspect, and will not be repeated here.

[0071] Fourthly, embodiments of the present invention provide a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the video classification method provided in the first aspect.

[0072] It is understood that the explanations, examples, and beneficial effects of the computer-readable storage medium provided in the embodiments of the present invention can be referred to the corresponding parts of the first aspect, and will not be repeated here.

[0073] It is understood that any references to memory, storage, database, or other media used in the embodiments provided by this invention may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and RAMbus dynamic RAM (RDRAM), etc.

[0074] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0075] The above description is merely a specific embodiment of the present invention, enabling those skilled in the art to understand or implement the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the present invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features claimed herein.

Claims

1. A video classification method, characterized in that, The method includes: Obtain the target video to be processed; The target audio features and video frame features are identified from the target video using an artificial intelligence model; The target audio features are used as input to a pre-trained feature classification model to obtain multiple audio features; wherein, the feature classification model includes multiple parallel branch models, each branch model is obtained by training a convolutional neural network with audio feature samples carrying a label; the multiple audio features are audio features of various label types separated from the target audio features; The video category of the target video is determined based on the various audio features and the video frame features; The step of determining the video category of the target video based on the multiple audio features and the video frame features includes: The second audio feature that was not selected from the multiple audio features and the video frame features are input into a convolutional neural network to perform feature fusion, thereby obtaining multiple fourth features corresponding to the multiple audio features; the multiple fourth features are input into a logistic regression model to obtain multiple second weights; wherein, the second weights are used to characterize the degree of influence of each audio feature on the video category of the target video; Each second weight is multiplied by its corresponding audio feature to obtain multiple fifth features; the multiple fifth features are summed to obtain a sixth feature; the sixth feature, text features, and video frame features are concatenated to generate a seventh feature; the seventh feature is used to identify the video category of the target video; wherein, the OCR text of the video is extracted using a character recognition model, and the corresponding text features are extracted using a BERT model.

2. The method according to claim 1, characterized in that, The process of identifying target audio features from the target video using an artificial intelligence model includes: The target video is input into an artificial intelligence model based on speech recognition to identify the audio frames contained in the target video and obtain an audio frame sequence. Draw the spectrogram corresponding to the audio frame sequence; The target audio features are extracted by inputting the spectrogram into a preset convolutional neural network.

3. The method according to claim 1, characterized in that, The process of identifying video frame features from the target video using an artificial intelligence model includes: Video frame images are extracted from the target video based on a preset frame extraction strategy; The video frame image is used as input to an image classification-based artificial intelligence model to obtain the video frame features.

4. The method according to claim 1, characterized in that, The target audio features are used as input to a pre-trained feature classification model to obtain multiple audio features. Each audio feature is obtained in the following way: A first branch model that has not been selected from multiple branch models is selected. The first branch model is obtained by training a convolutional neural network with audio feature samples carrying the first type of label. The target audio features are used as the input of the first branch model and feature extraction is performed through the fully connected layer in the convolutional neural network to separate the audio features with the first type of label from the target audio features.

5. The method according to any one of claims 1-4, characterized in that, Determining the video category of the target video based on the multiple audio features and the video frame features includes: The first audio feature that was not selected from the multiple audio features and the video frame features are input into a convolutional neural network to perform feature fusion and obtain multiple first features corresponding to the multiple audio features. The multiple first features are input into a logistic regression model to obtain multiple first weights; wherein, the first weights are used to characterize the degree of influence of each audio feature on the video category of the target video; Each first weight is multiplied by its corresponding audio feature to obtain multiple second features; The third feature is obtained by summing the plurality of second features; The third feature identifies the video category of the target video.

6. The method according to claim 3, characterized in that, After extracting video frame images from the target video based on a preset frame extraction strategy, the method further includes: The target text is obtained by recognizing the text in the video frame image using an artificial intelligence model based on character recognition. The BERT model is used to extract text features from the target text to characterize the label type of the target text.

7. A video classification device, characterized in that, The device includes: The acquisition module is used to acquire the target video to be processed; The recognition module is used to identify target audio features and video frame features from the target video using an artificial intelligence model; The classification module is used to take the target audio features as input to a pre-trained feature classification model to obtain multiple audio features; wherein, the feature classification model includes multiple parallel branch models, each branch model is obtained by training a convolutional neural network with audio feature samples carrying a label type; the multiple audio features are audio features of various label types separated from the target audio features; The determination module is used to determine the video category of the target video based on the multiple audio features and the video frame features; The determination module includes: a fifth calculation unit, used to input unselected second audio features and video frame features from multiple audio features into a convolutional neural network for feature fusion to obtain multiple fourth features corresponding to the various audio features; a sixth calculation unit, used to input multiple fourth features into a logistic regression model to obtain multiple second weights; wherein the second weights are used to characterize the degree of influence of each audio feature on the video category of the target video; a seventh calculation unit, used to perform multiplication operations on each second weight and its corresponding audio feature to obtain multiple fifth features; an eighth calculation unit, used to sum the multiple fifth features to obtain a sixth feature; a feature concatenation unit, used to generate a seventh feature by concatenating the sixth feature, text features, and video frame features; and a second identification unit, used to identify the seventh feature as the video category of the target video; wherein the OCR text of the video is extracted using a character recognition model, and the corresponding text features are extracted using a BERT model.

8. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

Images