Video tag intelligent classification method and device, electronic equipment and storage medium

Abstract

The invention relates to the technical field of artificial intelligence, and discloses a video tag intelligent classification method and device, electronic equipment and a storage medium. The method comprises the steps of obtaining a to-be-classified target video, and extracting video feature data of the target video through a preset convolutional neural network; obtaining target video subtitle data, and processing the subtitle data through a preset recurrent neural network to obtain corresponding subtitle feature data; obtaining audio data corresponding to the target video, and processing the audio data through a preset convolutional neural network to obtain audio feature data; performing feature fusion processing on the video feature data, and / or the subtitle feature data, and / or the audio feature data to obtain video fusion feature data; and inputting the video fusion feature data to the trained video tag classifier to obtain a tag set corresponding to the target video. The method improves the accuracy of video tag classification, reduces the labor cost, and can be applied to an outbound service robot scene in the field of financial science and technology.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a method, device, electronic device and storage medium for intelligent classification of video tags. Background Art

[0002] Video label classification refers to identifying the category of a short video composed of several consecutive key frames in the video. Its input is a short video composed of consecutive key frames, and the output is the label category corresponding to the video clip. The context information of the video refers to the use of different types of data contained in the video metadata to complete the deep learning task, such as the video image information, letters, audio, etc. contained in the video clip. The rapid development of information technology has created a huge movie market and short video platforms. Videos are usually divided into different categories according to their labels in order to achieve accurate push. The current idea of ​​solving video label classification in the existing technology is to use convolutional neural networks to gradually extract features from video clips, and then design a classifier based on these features to complete the classification of video labels. However, traditional video label classification technology solutions often only use the image information of the video, ignoring the information contained in the text and sound. Summary of the invention

[0003] The main technical problem solved by the implementation methods of the present application is that the accuracy of video tag classification using traditional technologies needs to be improved.

[0004] To solve the above technical problems, the first technical solution adopted in the implementation mode of the present application is: to provide a video tag intelligent classification method, including: obtaining a target video to be classified, extracting first video feature data of the target video through a preset first convolutional neural network; obtaining subtitle data corresponding to the target video, processing the subtitle data through a preset first recurrent neural network, and obtaining corresponding first subtitle feature data; obtaining audio data corresponding to the target video, processing the audio data through a preset second convolutional neural network, and obtaining corresponding first audio feature data; performing feature fusion processing on the first video feature data, and / or the first subtitle feature data, and / or the first audio feature data, to obtain video fusion feature data; inputting the video fusion feature data into a trained video tag classifier to obtain a tag set corresponding to the target video.

[0005] Optionally, the preset first convolutional neural network is a 3D convolutional neural network, and the step of extracting the first video feature data of the target video through the preset first convolutional neural network includes: converting the target video into video sequence data in a preset data format; sliding the convolution kernel in three dimensions of height, width and time through the convolution layer of the 3D convolutional neural network to extract spatial features and temporal features; reducing the spatial dimension of the feature map corresponding to the video sequence data through the pooling layer of the 3D convolutional neural network; and mapping the extracted spatial features and temporal features to the first video feature data through the fully connected layer of the 3D convolutional neural network.

[0006] Optionally, the preset first recurrent neural network is a long short-term memory network, and the step of processing the subtitle data through the preset first recurrent neural network to obtain corresponding first subtitle feature data includes: converting the subtitle data into a subtitle text sequence in a preset data format; performing a word embedding operation on each word in the subtitle text sequence, converting the subtitle text sequence into a corresponding subtitle word vector sequence, and if the current video segment is a segment without subtitles, using a preset default value as a word embedding vector; sending the subtitle word vector sequence to the preset long short-term memory network; extracting hidden state data in the subtitle word vector sequence through the long short-term memory network, and setting the hidden state data as the first subtitle feature data.

[0007] Optionally, the step of processing the audio data through a preset second convolutional neural network to obtain corresponding first audio feature data includes: performing audio preprocessing operations on the audio data through a preset audio tool library to obtain preprocessed audio data; using a tool function in the preset audio tool library to extract short-time Fourier transform spectrum data of the preprocessed audio data; inputting the short-time Fourier transform spectrum data into the second convolutional neural network; performing a preset number of convolution operations and downsampling operations on the short-time Fourier transform spectrum data through the second convolutional neural network to obtain the first audio feature data.

[0008] Optionally, the step of performing feature fusion processing on the first video feature data, and / or the first subtitle feature data, and / or the first audio feature data to obtain video fusion feature data includes: obtaining a preset feature dimension, adjusting the first video feature data, and / or the first subtitle feature data, and / or the first audio feature data to the preset feature dimension, to obtain second video feature data, and / or second subtitle feature data, and / or second audio feature data; and splicing the second video feature data, and / or the second subtitle feature data, and / or the second audio feature data according to a preset splicing order to obtain the video fusion feature data.

[0009] Optionally, the steps of constructing the video label classifier include: creating an initial video label classifier model using a preset neural network; obtaining video training data with video labels, and video label data corresponding to the video training data; processing the video training data using a pre-trained video feature extraction model to obtain third video feature data; processing the video label data using single-hot encoding to obtain video label feature data; and training the initial video label classifier model using the third video feature data and the video label feature data to obtain the trained video label classifier.

[0010] Optionally, after the step of inputting the video fusion feature data into the trained video label classifier to obtain the label set corresponding to the target video, it also includes: determining whether the number of label elements in the label set is within a preset label number threshold range; if it is not within the preset label number threshold range, re-executing the steps from extracting the first video feature data of the target video through a preset first convolutional neural network to obtaining the label set corresponding to the target video, until the number of label elements in the label set is within the preset label number threshold range, or the number of re-executions is greater than a preset repeated execution number threshold.

[0011] To solve the above technical problems, the second technical solution adopted by the implementation mode of the present application is: to provide a video label intelligent classification device, including: a video feature data module, used to obtain a target video to be classified, and extract the first video feature data of the target video through a preset first convolutional neural network; a subtitle feature data module, used to obtain the subtitle data corresponding to the target video, and process the subtitle data through a preset first recurrent neural network to obtain the corresponding first subtitle feature data; an audio feature data module, used to obtain the audio data corresponding to the target video, and process the audio data through a preset second convolutional neural network to obtain the corresponding first audio feature data; a fusion feature data module, used to perform feature fusion processing on the first video feature data, and / or the first subtitle feature data, and / or the first audio feature data to obtain video fusion feature data; a classification label set module, used to input the video fusion feature data into a trained video label classifier to obtain a label set corresponding to the target video.

[0012] To solve the above technical problems, the third technical solution adopted in the implementation mode of the present application is: to provide an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the video tag intelligent classification method as described above.

[0013] In order to solve the above technical problems, the fourth technical solution adopted in the implementation mode of the present application is: providing a non-volatile computer-readable storage medium, wherein the non-volatile computer-readable storage medium stores computer-executable instructions. When the computer-executable instructions are executed by an electronic device, the electronic device executes the video tag intelligent classification method as described above.

[0014] Different from the related art, the present application extracts the first video feature data of the target video by presetting a first convolutional neural network; obtains the subtitle data corresponding to the target video, processes the subtitle data by presetting a first recurrent neural network, and obtains the corresponding first subtitle feature data; obtains the audio data corresponding to the target video, processes the audio data by presetting a second convolutional neural network, and obtains the corresponding first audio feature data; performs feature fusion processing on the first video feature data, and / or the first subtitle feature data, and / or the first audio feature data, and obtains video fusion feature data; inputs the video fusion feature data into a trained video label classifier, and obtains a label set corresponding to the target video. This method improves the accuracy of video label classification, reduces labor costs, and can be applied to outbound call sales robot scenarios in the financial technology field. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] One or more embodiments are exemplarily described by corresponding drawings, which do not constitute limitations on the embodiments. Elements with the same reference numerals in the drawings represent similar elements, and the figures in the drawings do not constitute proportional limitations unless otherwise stated.

[0016] Figure 1 It is a schematic diagram of the operating environment of the video tag intelligent classification method provided in the embodiment of the present application.

[0017] Figure 2 It is a schematic diagram of the execution flow of the video tag intelligent classification method provided in an embodiment of the present application.

[0018] Figure 3 It is a schematic diagram of the execution flow of obtaining the first video feature data provided in an embodiment of the present application.

[0019] Figure 4 It is a schematic diagram of the execution flow of obtaining first subtitle feature data provided by an embodiment of the present application.

[0020] Figure 5 It is a schematic diagram of the system structure of the video tag intelligent classification device provided in an embodiment of the present application.

[0021] Figure 6 It is a schematic diagram of the hardware structure of an electronic device for executing the video tag intelligent classification method provided in an embodiment of the present application. DETAILED DESCRIPTION

[0022] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0023] It should be noted that, if there is no conflict, the various features in the embodiments of the present application can be combined with each other, all within the scope of protection of the present application. In addition, although the functional modules are divided in the device schematic diagram and the logical order is shown in the flow chart, in some cases, the steps shown or described can be performed in a different order than the module division in the device schematic diagram or the order in the flow chart.

[0024] Unless otherwise defined, all technical and scientific terms used in this specification have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used in the specification of this application are only for the purpose of describing specific embodiments and are not intended to limit this application. The term "and / or" used in this specification includes any and all combinations of one or more of the related listed items.

[0025] To facilitate understanding of this embodiment, firstly, a video tag intelligent classification method disclosed in the embodiment of the present application is described in detail. Figure 1 , Figure 1 Schematic diagram of the operating environment of the video tag intelligent classification method provided in the embodiment of the present application. Figure 1 As shown, the execution subject of the video tag intelligent classification method provided in the embodiment of the present application is generally an electronic device with certain computing capabilities, such as a computer device. In some possible implementations, the video tag intelligent classification method can be implemented by a processor calling a computer-readable instruction stored in a memory. Figure 1 The computer device in the term can be a server. The server can be an independent server or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms. It can be understood that Figure 1 The number of computer devices in the figure is only for reference and can be expanded to any number according to actual needs.

[0026] Please continue reading Figure 2 , Figure 2 FIG. 1 is a schematic diagram of the execution flow of the video tag intelligent classification method provided in the embodiment of the present application. Figure 2 As shown, the process includes the following steps S1 to S5.

[0027] S1. Obtain a target video to be classified, and extract first video feature data of the target video by using a preset first convolutional neural network.

[0028] Among them, the target video can be obtained according to the type of video to be classified. For example, the target video can be a short video with a short video length, or a long video with a long video length. For the comprehensive video classification needs of both long videos and short videos, two methods can be adopted. One is to use short video training data and long video training data to train different video classification models for label classification, and the other is to use short video training data and long video training data to train a robust comprehensive video label classification model for video label classification.

[0029] As an optional implementation, please continue to refer to Figure 3 , Figure 3 is a schematic diagram of an execution flow of obtaining first video feature data provided by an embodiment of the present application, such as Figure 3 As shown, the following steps S11 to S14 may be included.

[0030] S11, converting the target video into video sequence data in a preset data format.

[0031] S12. The convolution kernel is slid in the three dimensions of height, width and time through the convolution layer of the 3D convolutional neural network to extract spatial and temporal features.

[0032] Among them, 3D convolutional neural network (3D CNN) is a deep learning model suitable for video data with spatiotemporal characteristics. Unlike traditional 2D convolutional neural networks that focus on processing image data, 3D CNN can effectively capture the temporal and spatial information in video data. The network architecture of 3DCNN is usually similar to that of 2D CNN, including convolutional layers, pooling layers, normalization layers, and activation functions. Through the stacking and alternation of multiple layers, the network can gradually learn the spatiotemporal characteristics in video data and capture the time information in video data, so that the network takes into account the temporal relationship between frames when processing video data, which helps to improve the understanding of video content and classification accuracy.

[0033] S13. Reduce the spatial dimension of the feature map corresponding to the video sequence data through the pooling layer of the 3D convolutional neural network.

[0034] S14. Mapping the extracted spatial features and temporal features into first video feature data through a fully connected layer of a 3D convolutional neural network.

[0035] Among them, by extracting video features through 3D convolutional neural networks, it is possible to extract the feature changes of people or things in video images within a certain period of time, and the feature change information is highly correlated with the label information of the video, so the accuracy of target video label classification can be further improved.

[0036] S2. Acquire subtitle data corresponding to the target video, and process the subtitle data through a preset first recurrent neural network to obtain corresponding first subtitle feature data.

[0037] As another optional implementation, please continue to refer to Figure 4 , Figure 4 is a schematic diagram of an execution flow of obtaining first subtitle feature data provided by an embodiment of the present application, such as Figure 4 As shown, the following steps S21 to S24 may be included.

[0038] S21. Convert subtitle data into a subtitle text sequence in a preset data format.

[0039] S22. Perform a word embedding operation on each word in the subtitle text sequence, convert the subtitle text sequence into a corresponding subtitle word vector sequence. If the current video segment is a segment without subtitles, use a preset default value as the word embedding vector.

[0040] Among them, word embedding is used to map words in the text to real vectors in a high-dimensional space. By considering the occurrence of words in the context to learn word vectors, the semantic relationship between words can be captured. Word embedding models are usually trained using large-scale text corpora, and common methods include Word2Vec, GloVe, and FastText. Word embedding takes into account the semantics of words in context and can better express the semantic information of words, thereby further improving the accuracy of target video label classification.

[0041] S23, sending the subtitle word vector sequence to the preset long short-term memory network.

[0042] Among them, the Long Short-Term Memory (LSTM) network is a commonly used recurrent neural network (RNN) variant that is used to solve the gradient vanishing and gradient exploding problems that occur in traditional RNNs when processing long sequence data. LSTM contains multiple repeated "memory units", each of which consists of three gating structures: Forget Gate, Input Gate, and Output Gate. Through the gating structure, LSTM can effectively avoid the gradient vanishing problem in traditional RNNs, allowing the network to better train and learn long sequence data, such as long subtitle sequences in long videos.

[0043] S24 extracts hidden state data in the subtitle word vector sequence through a long short-term memory network, and sets the hidden state data as the first subtitle feature data.

[0044] S3. Obtain audio data corresponding to the target video, process the audio data through a preset second convolutional neural network, and obtain corresponding first audio feature data.

[0045] As another optional implementation, the above step S3 may specifically include the following steps S31 to S34.

[0046] S31. Perform an audio preprocessing operation on the audio data through a preset audio tool library to obtain preprocessed audio data.

[0047] S32. Use a tool function in a preset audio tool library to extract short-time Fourier transform spectrum data of the preprocessed audio data.

[0048] Among them, Short-Time Fourier Transform (STFT) is a signal processing technology used to analyze the spectral information of a signal that changes over time. STFT divides the signal into multiple short-time windows and applies Fourier transform to each window to obtain the spectral characteristics of the signal in different time periods. The signal is divided into multiple overlapping windows, and the signal in each window is considered to be steady-state. Commonly used window functions include Hanning window, Hamming window, etc. The window size determines the spectral resolution. The narrower the window, the higher the spectral resolution, and the more detailed the spectral characteristics of the signal can be analyzed.

[0049] S33, inputting short-time Fourier transform spectrum data into the second convolutional neural network.

[0050] S34. Perform a preset number of convolution operations and downsampling operations on the short-time Fourier transform spectrum data through a second convolutional neural network to obtain first audio feature data.

[0051] S4. Perform feature fusion processing on the first video feature data, and / or the first subtitle feature data, and / or the first audio feature data to obtain video fusion feature data.

[0052] Specifically, a preset feature dimension is first obtained, and the first video feature data, and / or the first subtitle feature data, and / or the first audio feature data are adjusted to the preset feature dimension to obtain the second video feature data, and / or the second subtitle feature data, and / or the second audio feature data. Then, the second video feature data, and / or the second subtitle feature data, and / or the second audio feature data are spliced ​​in a preset splicing order to obtain video fusion feature data.

[0053] S5. Input the video fusion feature data into the trained video label classifier to obtain the label set corresponding to the target video.

[0054] As an optional implementation, the step of constructing the video classifier may include the following steps S51 to S55.

[0055] S51. Create an initial video label classifier model using a preset neural network.

[0056] S52: Obtain video training data with video tags, and video tag data corresponding to the video training data.

[0057] S53: Use the pre-trained video feature extraction model to process the video training data to obtain third video feature data.

[0058] S54. Use one-hot encoding to process the video tag data to obtain video tag feature data.

[0059] Among them, One-Hot Encoding is a commonly used categorical data encoding technique, which is usually used to convert categorical variables into digital forms that can be processed by machine learning algorithms. One-hot encoding avoids the misunderstanding of size relationships that may be caused by numericalizing categorical variables. Each category is independent and has no size or order relationship.

[0060] S55: Use the third video feature data and the video label feature data to train an initial video label classifier model to obtain a trained video label classifier.

[0061] As another optional implementation, after the above step S5, it is also possible to determine whether the number of tag elements in the tag set is within a preset tag number threshold range. Then, if it is not within the preset tag number threshold range, the steps from extracting the first video feature data of the target video through the preset first convolutional neural network to obtaining the tag set corresponding to the target video are re-executed until the number of tag elements in the tag set is within the preset tag number threshold range, or the number of re-executions is greater than the preset repeated execution number threshold.

[0062] The video tag intelligent classification method provided in the embodiment of the present application extracts the first video feature data of the target video by presetting the first convolutional neural network; obtains the subtitle data corresponding to the target video, processes the subtitle data by presetting the first recurrent neural network, and obtains the corresponding first subtitle feature data; obtains the audio data corresponding to the target video, processes the audio data by presetting the second convolutional neural network, and obtains the corresponding first audio feature data; performs feature fusion processing on the first video feature data, and / or the first subtitle feature data, and / or the first audio feature data, and obtains video fusion feature data; inputs the video fusion feature data into the trained video tag classifier, and obtains the tag set corresponding to the target video. This method improves the accuracy of video tag classification, reduces labor costs, and can be applied to the outbound call service robot scenario in the financial technology field.

[0063] Please continue reading Figure 5 , Figure 5 is a schematic diagram of the system structure of the video tag intelligent classification device provided in the embodiment of the present application, such as Figure 5 As shown, the device includes: a video feature data module 51, a subtitle feature data module 52, an audio feature data module 53, a fusion feature data module 54 and a classification label collection module 55.

[0064] The video feature data module 51 is used to obtain a target video to be classified, and extract first video feature data of the target video through a preset first convolutional neural network.

[0065] The subtitle feature data module 52 is used to obtain subtitle data corresponding to the target video, and process the subtitle data through a preset first recurrent neural network to obtain corresponding first subtitle feature data.

[0066] The audio feature data module 53 is used to obtain the audio data corresponding to the target video, and process the audio data through a preset second convolutional neural network to obtain corresponding first audio feature data.

[0067] The fusion feature data module 54 is used to perform feature fusion processing on the first video feature data, and / or the first subtitle feature data, and / or the first audio feature data to obtain video fusion feature data.

[0068] The classification label set module 55 is used to input the video fusion feature data into a trained video label classifier to obtain a label set corresponding to the target video.

[0069] As an optional implementation, the video feature data module 51 is specifically used to convert the target video into video sequence data in a preset data format; slide the convolution kernel in the three dimensions of height, width, and time through the convolution layer of the 3D convolutional neural network to extract spatial features and temporal features; reduce the spatial dimension of the feature map corresponding to the video sequence data through the pooling layer of the 3D convolutional neural network; and map the extracted spatial features and temporal features to the first video feature data through the fully connected layer of the 3D convolutional neural network.

[0070] As an optional implementation, the subtitle feature data module 52 is specifically used to convert the subtitle data into a subtitle text sequence in a preset data format; perform a word embedding operation on each word in the subtitle text sequence, and convert the subtitle text sequence into a corresponding subtitle word vector sequence. If the current video segment is a segment without subtitles, a preset default value is used as the word embedding vector; send the subtitle word vector sequence to the preset long short-term memory network; extract the hidden state data in the subtitle word vector sequence through the long short-term memory network, and set the hidden state data as the first subtitle feature data.

[0071] As an optional implementation, the audio feature data module 53 is specifically used to perform audio preprocessing operations on the audio data through a preset audio tool library to obtain preprocessed audio data; use the tool function in the preset audio tool library to extract short-time Fourier transform spectrum data of the preprocessed audio data; input the short-time Fourier transform spectrum data into the second convolutional neural network; perform a preset number of convolution operations and downsampling operations on the short-time Fourier transform spectrum data through the second convolutional neural network to obtain the first audio feature data.

[0072] As an optional implementation, the fusion feature data module 54 is specifically used to obtain a preset feature dimension, adjust the first video feature data, and / or the first subtitle feature data, and / or the first audio feature data to the preset feature dimension, and obtain the second video feature data, and / or the second subtitle feature data, and / or the second audio feature data; splice the second video feature data, and / or the second subtitle feature data, and / or the second audio feature data in a preset splicing order to obtain the video fusion feature data.

[0073] As an optional implementation, the classification label set module 55 is also specifically used to create an initial video label classifier model using a preset neural network; obtain video training data with video labels, and video label data corresponding to the video training data; use a pre-trained video feature extraction model to process the video training data to obtain third video feature data; use single-hot encoding to process the video label data to obtain video label feature data; use the third video feature data and the video label feature data to train the initial video label classifier model to obtain the trained video label classifier.

[0074] As an optional implementation, the classification label set module 55 is also specifically used to determine whether the number of label elements in the label set is within a preset label number threshold range; if it is not within the preset label number threshold range, then re-execute the steps from extracting the first video feature data of the target video through the preset first convolutional neural network to obtaining the label set corresponding to the target video, until the number of label elements in the label set is within the preset label number threshold range, or the number of re-executions is greater than the preset repeated execution number threshold.

[0075] It should be noted that the above-mentioned video tag intelligent classification device can execute the video tag intelligent classification method provided in the embodiment of the present application, and has the corresponding functional modules and beneficial effects of the execution method. For technical details not fully described in the embodiment of the video tag intelligent classification device, please refer to the video tag intelligent classification method provided in the embodiment of the present application.

[0076] Please continue reading Figure 6 , Figure 6 is a schematic diagram of the hardware structure of an electronic device 600 for executing a video tag intelligent classification method provided in an embodiment of the present application, such as Figure 6 As shown, the electronic device 600 includes:

[0077] One or more processors 610 and memory 620, Figure 6 A processor 610 is taken as an example.

[0078] The processor 610 and the memory 620 may be connected via a bus or other means. Figure 6 The example of connecting through bus is taken in the following.

[0079] The memory 620 is a non-volatile computer-readable storage medium that can be used to store non-volatile software programs, non-volatile computer executable programs and modules, such as program instructions / modules corresponding to the video tag intelligent classification method in the embodiment of the present application. The processor 610 executes various functional applications and data processing of the server by running the non-volatile software programs, instructions and modules stored in the memory 620, that is, the video tag intelligent classification method in the above method embodiment is implemented.

[0080] The memory 620 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the video tag intelligent classification device, etc. In addition, the memory 620 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory 620 may optionally include a memory remotely arranged relative to the processor 610, and these remote memories may be connected to the video tag intelligent classification device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0081] The one or more modules are stored in the memory 620, and when executed by the one or more processors 610, the video tag intelligent classification method in any of the above method embodiments is executed, for example, the video tag intelligent classification method described above is executed. Figure 2 Steps S1 to S4 of the method, Figure 3 Steps S11 to S14 of the method, Figure 4 Steps S21 to S24 of the method are implemented Figure 5 Functions of modules 21-25.

[0082] The above-mentioned product can execute the method provided in the embodiment of the present application, and has the functional modules and beneficial effects corresponding to the execution method. For technical details not fully described in this embodiment, please refer to the method provided in the embodiment of the present application.

[0083] The embodiment of the present application provides a non-volatile computer-readable storage medium, wherein the non-volatile computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are executed by one or more processors, such as Figure 6 A processor 610 in the embodiment may enable the one or more processors to execute the video tag intelligent classification method in any of the above method embodiments, for example, to execute the above described Figure 2 Steps S1 to S4 of the method, Figure 3 Steps S11 to S14 of the method, Figure 4Steps S21 to S24 of the method are implemented Figure 5 Functions of modules 21-25.

[0084] The present application embodiment provides a computer program product, the computer program product includes a computer program stored on a non-volatile computer-readable storage medium, the computer program includes program instructions, when the program instructions are executed by the electronic device, the electronic device is enabled to execute the video tag intelligent classification method in any of the above method embodiments, for example, execute the above described Figure 2 Steps S1 to S4 of the method, Figure 3 Steps S11 to S14 of the method, Figure 4 Steps S21 to S24 of the method are implemented Figure 5 Functions of modules 21-25.

[0085] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place, or may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0086] Through the description of the above implementation methods, ordinary technicians in this field can clearly understand that each implementation method can be implemented by means of software plus a general hardware platform, and of course, it can also be implemented by hardware. Ordinary technicians in this field can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing related hardware through a computer program, and the program can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the storage medium can be a disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM), etc.

[0087] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Under the concept of the present application, the technical features in the above embodiments or different embodiments can also be combined, the steps can be implemented in any order, and there are many other changes in different aspects of the present application as described above, which are not provided in detail for the sake of simplicity. Although the present application has been described in detail with reference to the aforementioned embodiments, a person of ordinary skill in the art should understand that the technical solutions described in the aforementioned embodiments can still be modified, or some of the technical features can be replaced by equivalents. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A video tag intelligent classification method, characterized in that: include: Obtain a target video to be classified, and extract first video feature data of the target video by using a preset first convolutional neural network; Acquire subtitle data corresponding to the target video, and process the subtitle data through a preset first recurrent neural network to obtain corresponding first subtitle feature data; Acquire audio data corresponding to the target video, and process the audio data by using a preset second convolutional neural network to obtain corresponding first audio feature data; Performing feature fusion processing on the first video feature data, and / or the first subtitle feature data, and / or the first audio feature data to obtain video fusion feature data; The video fusion feature data is input into a trained video label classifier to obtain a label set corresponding to the target video.

2. The video tag intelligent classification method according to claim 1 is characterized in that: The preset first convolutional neural network is a 3D convolutional neural network, and the step of extracting the first video feature data of the target video by using the preset first convolutional neural network includes: Converting the target video into video sequence data in a preset data format; The convolution layer of the 3D convolutional neural network slides the convolution kernel in three dimensions: height, width, and time to extract spatial features and temporal features; Reducing the spatial dimension of the feature map corresponding to the video sequence data through a pooling layer of the 3D convolutional neural network; The extracted spatial features and the temporal features are mapped into the first video feature data through the fully connected layer of the 3D convolutional neural network.

3. The video tag intelligent classification method according to claim 1 is characterized in that: The preset first recurrent neural network is a long short-term memory network, and the step of processing the subtitle data by the preset first recurrent neural network to obtain corresponding first subtitle feature data includes: Converting the subtitle data into a subtitle text sequence in a preset data format; Performing a word embedding operation on each word in the subtitle text sequence, converting the subtitle text sequence into a corresponding subtitle word vector sequence, and using a preset default value as the word embedding vector if the current video segment is a segment without subtitles; Sending the subtitle word vector sequence to the preset long short-term memory network; The hidden state data in the subtitle word vector sequence is extracted through the long short-term memory network, and the hidden state data is set as the first subtitle feature data.

4. The video tag intelligent classification method according to claim 1 is characterized in that: The step of processing the audio data by a preset second convolutional neural network to obtain corresponding first audio feature data includes: Performing an audio preprocessing operation on the audio data by using a preset audio tool library to obtain preprocessed audio data; Extracting short-time Fourier transform spectrum data of the preprocessed audio data using a tool function in the preset audio tool library; Inputting the short-time Fourier transform spectrum data into the second convolutional neural network; The first audio feature data is obtained by performing a preset number of convolution operations and downsampling operations on the short-time Fourier transform spectrum data through the second convolutional neural network.

5. The video tag intelligent classification method according to claim 1 is characterized in that: The step of performing feature fusion processing on the first video feature data, and / or the first subtitle feature data, and / or the first audio feature data to obtain video fusion feature data includes: Acquire a preset feature dimension, adjust the first video feature data, and / or the first subtitle feature data, and / or the first audio feature data to the preset feature dimension, and obtain second video feature data, and / or second subtitle feature data, and / or second audio feature data; The second video feature data, and / or the second subtitle feature data, and / or the second audio feature data are spliced ​​in a preset splicing order to obtain the video fusion feature data.

6. The video tag intelligent classification method according to claim 1, characterized in that: The steps of constructing the video label classifier include: Create an initial video label classifier model using a preset neural network; Obtaining video training data with video tags and video tag data corresponding to the video training data; Process the video training data using a pre-trained video feature extraction model to obtain third video feature data; Process the video tag data using one-hot encoding to obtain video tag feature data; The initial video label classifier model is trained using the third video feature data and the video label feature data to obtain the trained video label classifier.

7. The video tag intelligent classification method according to claim 1 is characterized in that: After the step of inputting the video fusion feature data into the trained video label classifier to obtain the label set corresponding to the target video, the method further includes: Determine whether the number of tag elements in the tag set is within a preset tag number threshold range; If it is not within the preset label quantity threshold range, the steps from extracting the first video feature data of the target video through the preset first convolutional neural network to obtaining the label set corresponding to the target video are re-executed until the number of label elements in the label set is within the preset label quantity threshold range, or the number of re-executions is greater than the preset repeated execution number threshold.

8. A video tag intelligent classification device, characterized in that: include: A video feature data module, used to obtain a target video to be classified, and extract first video feature data of the target video by using a preset first convolutional neural network; A subtitle feature data module, used to obtain subtitle data corresponding to the target video, and process the subtitle data through a preset first recurrent neural network to obtain corresponding first subtitle feature data; An audio feature data module, used to obtain audio data corresponding to the target video, and process the audio data through a preset second convolutional neural network to obtain corresponding first audio feature data; a fusion feature data module, configured to perform feature fusion processing on the first video feature data, and / or the first subtitle feature data, and / or the first audio feature data to obtain video fusion feature data; The classification label set module is used to input the video fusion feature data into the trained video label classifier to obtain the label set corresponding to the target video.

9. An electronic device, characterized in that: include: at least one processor; as well as, a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the video tag intelligent classification method according to any one of claims 1-7.

10. A non-volatile computer-readable storage medium, characterized in that: The non-volatile computer-readable storage medium stores computer-executable instructions. When the computer-executable instructions are executed by an electronic device, the electronic device executes the video tag intelligent classification method according to any one of claims 1 to 7.

Images