Video processing method and apparatus

Abstract

The method comprises: acquiring image features, text features, audio features and script information of a video to be classified; determining script features of the video to be classified according to word vectors corresponding to each word in the script information; performing fusion processing on the image features, the text features and the audio features by using a gated multimodal unit to obtain semantic features of the video to be classified; performing splicing processing on the semantic features and the script features to obtain fusion features of the video to be classified; and determining a classification result of the video to be classified according to the fusion features, so that the classification effectiveness and accuracy are improved.

Description

Technical Field

[0001] This disclosure relates to the field of computer and image processing technology, and more particularly to a video processing method and apparatus. Background Technology

[0002] Videos (including short videos) have category attributes, such as finance, sports, and entertainment. How to classify videos has become an urgent problem to be solved.

[0003] For example, a video classification model can be trained to classify the video input to the model and obtain the type attribute of the input video.

[0004] However, the inventors of this disclosure have discovered that in some embodiments of video-based classification models that classify videos, the features of the video to be classified are considered in a single dimension. For example, the video to be classified is classified based on its image features to obtain the classification result, which results in a low accuracy of classification. Summary of the Invention

[0005] This disclosure provides a video processing method and apparatus to overcome the problem of low accuracy in video classification.

[0006] In a first aspect, embodiments of this disclosure provide a video processing method, including:

[0007] The image features, text features, audio features, and text information of the video to be classified are obtained, and the text features of the video to be classified are determined based on the word vectors corresponding to each word in the text information.

[0008] The image features, text features, and audio features are fused using a gated multi-mode unit to obtain the semantic features of the video to be classified.

[0009] The semantic features and the textual features are concatenated to obtain the fusion features of the video to be classified, and the classification result of the video to be classified is determined based on the fusion features.

[0010] Secondly, embodiments of this disclosure provide a video processing method, including:

[0011] Obtain image features, text features, and audio features from the sample video;

[0012] Extract the sample text information from the sample video, and determine the text features of the sample video based on the word vectors corresponding to each word in the sample text information;

[0013] The image features, text features, and audio features are fused using a gated multi-mode unit to obtain the semantic features of the sample video. The semantic features are then concatenated with the text features to obtain the fused features of the sample video.

[0014] A video classification model is trained based on the image features, text features, audio features, and fusion features, wherein the video classification model is used to classify the video to be classified.

[0015] Thirdly, embodiments of this disclosure provide a video processing apparatus, including:

[0016] The first acquisition unit is used to acquire image features, text features, audio features, and text information of the video to be classified;

[0017] The first determining unit is used to determine the text features of the video to be classified based on the word vectors corresponding to each word in the text information.

[0018] The first fusion unit is used to perform fusion processing on the image features, the text features, and the audio features using a gated multi-mode unit to obtain the semantic features of the video to be classified.

[0019] The first splicing unit is used to splice the semantic features and the text features to obtain the fusion features of the video to be classified.

[0020] The second determining unit is used to determine the classification result of the video to be classified based on the fusion features.

[0021] Fourthly, embodiments of this disclosure provide a video processing apparatus, including:

[0022] The third acquisition unit is used to acquire image features, text features, and audio features of the sample video;

[0023] The second extraction unit is used to extract the sample text information of the sample video;

[0024] The third determining unit is used to determine the text features of the sample video based on the word vectors corresponding to each word in the sample text information.

[0025] The second fusion unit is used to perform fusion processing on the image features, the text features, and the audio features using a gated multi-mode unit to obtain the semantic features of the sample video;

[0026] The second splicing unit is used to splice the semantic features and the text features to obtain the fusion features of the sample video;

[0027] The training unit is used to train a video classification model based on the image features, the text features, the audio features, and the fused features, wherein the video classification model is used to classify the video to be classified.

[0028] Fifthly, embodiments of this disclosure provide an electronic device, including: at least one processor and a memory;

[0029] The memory stores computer-executed instructions;

[0030] The at least one processor executes the computer execution instructions stored in the memory, causing the at least one processor to perform the method described in the first aspect and various possible designs of the first aspect as described above; or, causing the at least one processor to perform the method described in the second aspect and various possible designs of the second aspect as described above.

[0031] In a sixth aspect, embodiments of this disclosure provide a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the method described in the first aspect and various possible designs of the first aspect; or, implement the method described in the second aspect and various possible designs of the second aspect.

[0032] According to a seventh aspect of this disclosure, a computer program product is provided, the program product comprising: a computer program stored in a readable storage medium, wherein at least one processor of an electronic device can read the computer program from the readable storage medium, and the at least one processor executes the computer program to cause the electronic device to perform the method described in the first aspect or the second aspect.

[0033] The video processing method and apparatus provided in this embodiment acquire textual information of the video to be classified, determine textual features based on the textual information, and combine image features, text features, audio features, and textual features to determine the classification result of the video to be classified. This method can obtain classification results through more modal features of the video to be classified, thereby improving the accuracy and reliability of classification. Attached Figure Description

[0034] To more clearly illustrate the technical solutions in the embodiments of this disclosure or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0035] Figure 1This is an example diagram of a video processing method according to an embodiment of the present disclosure;

[0036] Figure 2 This is a schematic diagram illustrating the principle of generating semantic features according to an embodiment of this disclosure;

[0037] Figure 3 This is an example diagram of a video processing method according to another embodiment of this disclosure;

[0038] Figure 4 This is an example diagram of a video processing method according to another embodiment of this disclosure;

[0039] Figure 5 This is an example diagram of a video processing method according to another embodiment of this disclosure;

[0040] Figure 6 This is a schematic diagram illustrating the principle of the video processing method according to an embodiment of the present disclosure;

[0041] Figure 7 This is an example diagram of a video processing apparatus according to an embodiment of the present disclosure;

[0042] Figure 8 This is an example diagram of a video processing apparatus according to another embodiment of the present disclosure;

[0043] Figure 9 This is an example diagram of a video processing apparatus according to another embodiment of the present disclosure;

[0044] Figure 10 This is an example diagram of a video processing apparatus according to another embodiment of the present disclosure;

[0045] Figure 11 This is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of this disclosure. Detailed Implementation

[0046] To make the objectives, technical solutions, and advantages of the embodiments of this disclosure clearer, the technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this disclosure, and not all embodiments. Based on the embodiments of this disclosure, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.

[0047] Based on the content of the video, a type attribute can be labeled for the video. For example, a video about a basketball game can be labeled as a sports video, a video about how to invest can be labeled as a finance video, a video about entertainment news can be labeled as an entertainment video, and so on. These will not be listed here.

[0048] By combining video type attributes, the system can recommend videos relevant to users' preferences. For example, some users prefer financial videos, so the video recommendation system can recommend videos with the financial type attribute. Therefore, how to classify videos to determine their type attributes has become an urgent problem to be solved.

[0049] In some embodiments, sample videos can be collected to train a video classification model. When the video to be classified is input into the video classification model, the video classification model can output the type attribute corresponding to the video to be classified.

[0050] In scenarios where users upload videos, such as when a user transmits a short video to a video platform, the video platform can classify the short video based on the video's classification model to obtain the video's type attribute.

[0051] For video recommendation applications, the system can retrieve a user's historical video viewing history, classify the historically viewed videos based on a video classification model to obtain the type attributes of the historically viewed videos, and then retrieve videos with the same type attributes from the video platform to push the retrieved videos to the user.

[0052] During training, images of each frame in the sample video can be acquired, and features can be extracted from each frame to obtain the corresponding image features of each frame. Based on these image features, a classification model for the video can be trained.

[0053] In the application process, the acquired video to be classified can be classified based on the trained video classification model to obtain the classification result of the video to be classified.

[0054] However, video classification models trained from image features have relatively limited dimensions and weak generalization ability, resulting in low accuracy in video classification and consequently, low accuracy in classifying the videos to be classified.

[0055] In other embodiments, based on the above embodiments, a classification model for the video can be obtained by training the video title of the sample video. The acquired video to be classified can then be classified based on this classification model to obtain the classification result of the video to be classified.

[0056] For example, on the one hand, image features of the sample video can be obtained as described in the above embodiments. On the other hand, the video title of the sample video can also be obtained to extract the text features corresponding to the video title. A classification model for the video can be trained based on multiple modalities of image features and text features to improve the accuracy of classification processing of the video to be classified.

[0057] In some other embodiments, a video classification model can be trained by combining the audio signal of the sample video with the above embodiments. The acquired video to be classified can then be classified based on the video classification model to obtain the classification result of the video to be classified.

[0058] For example, on the one hand, image features of the sample video can be obtained as described in the above embodiments. On the other hand, audio signals of the sample video can also be obtained to extract audio features corresponding to the audio signals. A classification model for the video can be trained based on multiple modalities of image features and audio features. The obtained video to be classified can be classified based on the classification model of the video to obtain the classification result of the video to be classified.

[0059] In some other embodiments, a video classification model can be trained based on multiple modalities, including image features, text features, and audio features. The acquired video to be classified can then be classified based on this video classification model to obtain the classification result of the video to be classified.

[0060] To further improve the accuracy of video classification models, the inventors of this disclosure, through creative work, arrived at the inventive concept of this disclosure: combining the image features, text features, audio features, and textual information of the image to be classified, to classify the video to be classified, and obtain the classification result of the video to be classified.

[0061] The technical solutions of this disclosure and how they solve the aforementioned technical problems will be described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of this disclosure will now be described with reference to the accompanying drawings.

[0062] Please see Figure 1 , Figure 1 This is an example diagram of a video processing method according to an embodiment of the present disclosure.

[0063] like Figure 1 As shown, the method includes:

[0064] S101: Obtain the image features, text features, audio features, and text information of the video to be classified, and determine the text features of the video to be classified based on the word vectors corresponding to each word in the text information.

[0065] For example, the execution subject of this embodiment can be a video processing device, which can be a computer, a server (such as a local server, a cloud server, or a server cluster), a terminal device, a processor, a chip, etc., and will not be listed here.

[0066] Image features are used to characterize the video to be classified in the image dimension, such as color and texture. Text features are used to characterize the video to be classified in the character, word, and sentence dimensions. Audio features are used to characterize the video to be classified in the audio signal dimension, such as frequency, amplitude, and tempo. The text information is the content of the hashtags (also known as tweet topics) of the video to be classified.

[0067] Text features are determined based on text information. For example, the text information includes at least one word, or multiple words, each word has a corresponding word vector, so that the words are represented in a vector manner.

[0068] S102: The gated multi-mode unit is used to fuse image features, text features, and audio features to obtain the semantic features of the video to be classified.

[0069] For example, the principle of fusion processing of gated multi-mode units is as follows: Figure 2 As shown, x1 represents image features, x2 represents text features, x3 represents audio features, σ represents the activation function of the gated multi-mode unit (Sigmoid function), and tanh represents nonlinear variation.

[0070] like Figure 2 As shown, image feature x1 is activated by function σ, and the output weight z1 is generated; text feature x2 is activated by function σ, and the output weight z2 is generated; audio feature x3 is activated by function σ, and the output weight z3 is generated.

[0071] Apply a nonlinear transformation to image feature x1 and output h1; apply a nonlinear transformation to text feature x2 and output h2; apply a nonlinear transformation to audio feature x3 and output h3.

[0072] Based on h1 and weight z1, determine the image submodal features corresponding to the image features; based on h2 and weight z2, determine the text submodal features corresponding to the text features; based on h1 and weight z3, determine the audio submodal features corresponding to the audio features. Generate semantic features based on the image submodal features, text submodal features, and audio submodal features, such as... Figure 2 The “h” shown.

[0073] In this embodiment, semantic features are determined based on gated multimodal units, which integrates features from image modality, title modality, and audio modality, thereby making the semantic features more accurate and reliable.

[0074] S103: Concatenate the semantic features and textual features to obtain the fusion features of the video to be classified, and determine the classification result of the video to be classified based on the fusion features.

[0075] This embodiment does not limit the splicing method. For example, the textual features can be spliced ​​to the beginning of the semantic features to obtain the fused features; the textual features can be spliced ​​to the end of the semantic features to obtain the fused features; or the textual features can be inserted into the semantic features to obtain the fused features.

[0076] In this embodiment, semantic features are obtained by fusing image features, text features, and audio features through a gated multi-mode unit. These semantic features are then combined with textual features to obtain fused features. These fused features are used to determine the classification result of the video to be classified. This approach analyzes the video from different dimensions to obtain the classification result, thereby improving the accuracy and reliability of the classification result.

[0077] Please see Figure 3 , Figure 3 This is an example diagram of a video processing method according to another embodiment of the present disclosure.

[0078] like Figure 3 As shown, the method includes:

[0079] S301: Obtain the image features of the video to be classified.

[0080] It should be understood that, in order to avoid tedious descriptions, the same technical features as those in the above embodiments will not be repeated in this embodiment.

[0081] In some embodiments, S301 may include the following steps:

[0082] The first step is to perform sparse frame sampling on the video to be classified to obtain multiple video images.

[0083] Among them, any two video frames in the multi-frame video images are non-adjacent frames in the video to be classified.

[0084] For example, sparse frame sampling refers to obtaining multiple non-contiguous frames from the video to be classified as video images, so that any two frames in the multiple video images are non-adjacent frames.

[0085] For example, sparse frame sampling processing can be performed on the video to be classified based on the Temporal Segment Network (TSN). That is, the video to be classified is segmented based on time to obtain segment video, and video images are extracted from each segment video to obtain multi-frame video images.

[0086] The second step is to extract the features corresponding to each frame of the video image, aggregate the features corresponding to each frame of the video image to obtain aggregated features, and enhance the aggregated features to obtain image features.

[0087] In some embodiments, features corresponding to each frame of video image can be extracted by a model, and the extracted features can be aggregated and enhanced.

[0088] In this embodiment, by using sparse frame sampling to determine image features, the drawback of wasting resources due to the small differences between adjacent video images in the video to be classified can be avoided. Furthermore, by combining sparse frame sampling, feature extraction, and aggregation processing to determine image features, the image features can have a strong ability to characterize the characteristics of the video to be classified. Moreover, by combining enhancement processing to determine image features, the ability of image features to characterize the characteristics of the video to be classified can be further improved, thereby making the image features have high accuracy and reliability.

[0089] S302: Obtain the text features of the video to be classified.

[0090] In some embodiments, the video to be classified has a video title, and step S301 may include the following steps:

[0091] The first step is to perform word tokenization on the video title to obtain the word embedding of the video title.

[0092] The second step is to obtain the word features corresponding to each word in the word sequence, and generate text features (title embedding) based on the word features corresponding to each word.

[0093] In this embodiment, by using the word sequence obtained from word segmentation to determine the word features corresponding to each word, and generating text features based on each word feature, the text features can be highly consistent with the video title, thereby improving the accuracy and reliability of the text features.

[0094] S303: Obtain the audio features of the video to be classified.

[0095] For example, if the video to be classified has an audio signal, S303 may include the following steps:

[0096] The first step is to convert the audio signal from the time domain to the frequency domain using Fourier transform.

[0097] The second step is to obtain the Mel spectrogram of the audio signal in the frequency domain and generate audio features based on the Mel spectrogram.

[0098] In this embodiment, by converting the audio signal from the time domain to the frequency domain and obtaining audio features through Mel spectrograms, the audio features can be more accurately characterized in terms of the audio modality of the video to be classified.

[0099] S304: Extract the text information of the video to be categorized.

[0100] Generally, text information has a fixed format, such as "#xxx", so the text information of the video to be categorized can be obtained based on this format.

[0101] S305: Based on the pre-constructed vocabulary matrix, determine the feature vector corresponding to the copy information, and identify the feature vector as the copy features.

[0102] The vocabulary matrix is ​​used to represent the correspondence between historical text information and feature vectors.

[0103] In other words, a correspondence between text information and feature vectors (i.e., a word matrix) can be pre-constructed. When the text information of the video to be classified is obtained, the feature vector corresponding to the text information can be determined from the word matrix. This feature vector is the text feature.

[0104] In this embodiment, the efficiency and reliability of determining copywriting features can be improved by combining a word matrix.

[0105] In some embodiments, a method for constructing a vocabulary matrix may include the following steps:

[0106] First step: Obtain historical copywriting information.

[0107] Among them, historical text information is text information that characterizes the type attributes of historical videos, and historical videos have historical text information.

[0108] The second step is to extract features from the historical text information to obtain feature vectors.

[0109] The third step: Construct a vocabulary matrix to represent the mapping relationship between historical text information and feature vectors.

[0110] For example, historical copywriting information may include copywriting information such as #gesturedance, #heartdance, #magicalcutebabygrowthshow, #justchangename, etc. For each piece of historical copywriting information, the feature vector corresponding to that piece of historical copywriting information can be determined, such as determining the feature vector of the historical copywriting information "#gesturedance", the feature vector of the historical copywriting information "#heartdance", etc., to construct a word matrix.

[0111] In some embodiments, in order to make the vocabulary matrix more comprehensive and to improve the effectiveness of determining the text features of sample text information based on the vocabulary matrix, there are multiple historical text information. Accordingly, the frequency of each historical text information can be obtained, and the top n historical text information with the most frequencies can be extracted from each historical text information, where n is a positive integer greater than 1. Feature extraction processing is then performed on the top n historical text information to obtain the feature vector corresponding to each historical text information in the top n historical text information.

[0112] For example, when acquiring historical copywriting information, the frequency of occurrence of each piece of historical copywriting information can be counted, and the top 500 pieces of historical copywriting information with the most occurrences can be retained to construct a 500*64 word matrix. Each row in the word matrix represents the feature vector of the historical copywriting information.

[0113] S306: A gated multi-mode unit is used to fuse image features, text features, and audio features to obtain the semantic features of the video to be classified.

[0114] S307: Concatenate the semantic features and textual features to obtain the fused features of the video to be classified, and determine the classification result of the video to be classified based on the fused features.

[0115] Please see Figure 4 , Figure 4 This is an example diagram of a video processing method according to another embodiment of the present disclosure.

[0116] like Figure 4 As shown, the method includes:

[0117] S401: Obtain image features, text features, and audio features of the sample video.

[0118] For example, the execution subject in this embodiment can be the same as that in the above embodiments, or it can be a different execution subject. This embodiment does not limit it.

[0119] It should be understood that the number of sample videos is not limited in this embodiment and can be determined based on needs, historical records, and experiments.

[0120] For example, for scenarios requiring relatively high accuracy, the number of sample videos can be relatively large, while for scenarios requiring relatively low accuracy, the number of sample videos can be relatively small.

[0121] Image features are used to characterize the sample video in the image dimension, such as color and texture. Text features are used to characterize the sample video in the character, word, and sentence dimensions. Audio features are used to characterize the sample video in the audio signal dimension, such as frequency, amplitude, and tempo.

[0122] S402: Extract sample text information from the sample video and determine the text features of the sample video based on the sample text information.

[0123] Among them, the sample text information is text information that characterizes the type attributes of the sample video.

[0124] Correspondingly, text security features are features that characterize the type attributes of sample videos.

[0125] S403: The image features, text features, audio features, and copywriting features are fused to obtain the fused features of the sample video, and a classification model for the video is trained based on the image features, text features, audio features, and fused features.

[0126] Among them, the video classification model is used to classify the videos to be classified.

[0127] Based on the above analysis, this disclosure provides a video processing method, including: acquiring image features, text features, and audio features of a sample video; extracting sample text information from the sample video; determining text features of the sample video based on the sample text information, wherein the sample text information is text information characterizing the type attributes of the sample video; fusing the image features, text features, audio features, and text features to obtain fused features of the sample video; and training a video classification model based on the image features, text features, audio features, and fused features. The video classification model is used to classify the video to be classified. In this embodiment, by extracting sample text information to determine text features based on the sample text information, and combining the technical features of image features, text features, audio features, and text features to train the video classification model, the video classification model can be trained using features from more modalities of the sample video, thereby enabling the video classification model to have high generalization and high accuracy.

[0128] Please see Figure 5 , Figure 5 This is an example diagram of a video processing method according to another embodiment of the present disclosure.

[0129] like Figure 5 As shown, the method includes:

[0130] S501: Obtain image features, text features, and audio features of the sample video.

[0131] It should be understood that, in order to avoid tedious descriptions, the same technical features as those in the above embodiments will not be repeated in this embodiment.

[0132] In some embodiments, a method for obtaining image features of a sample video may include the following steps:

[0133] The first step is to perform sparse frame sampling on the sample video to obtain multiple sample images.

[0134] Among them, any two sample images in the multi-frame sample images are non-adjacent frames in the sample video.

[0135] For example, sparse frame sampling refers to obtaining multiple discontinuous frames from a sample video as sample images, so that any two sample images in the multiple sample images are non-adjacent frames.

[0136] For example, sparse frame sampling processing of sample videos can be performed based on Temporal Segment Network (TSN), that is, the sample video is segmented based on time to obtain segment sample video, and sample images are extracted from each segment sample video to obtain multiple frame sample images.

[0137] like Figure 6 As shown, the sample video is sampled (specifically, the sparse frame sampling processing based on TSN as shown above) to obtain multiple sample images. Figure 6 (Three sample images are used as an example to demonstrate this).

[0138] The second step is to extract the features corresponding to each frame of the sample image, aggregate the features corresponding to each frame of the sample image to obtain aggregated features, and enhance the aggregated features to obtain image features (videoembedding).

[0139] For example, features corresponding to each frame of sample images can be extracted based on an image classification model (EfficientNet), such as... Figure 6 The "feature extraction" shown is illustrated. The features output from EfficientNet are input into the multimodal video classification model (NeXt VLAD), so that NeXt VLAD can process the extracted features as follows: Figure 6 The "aggregation processing" shown yields aggregated features. These aggregated features, output from NeXt VLAD, are then input into... Figure 6 The "Feature Enhancement Model" (SENet) shown here enhances aggregated features using SENet, resulting in... Figure 6 The "image features" shown.

[0140] In this embodiment, by using sparse frame sampling to determine image features, the drawback of resource waste caused by the small differences between adjacent frame sample images in the sample video can be avoided. Furthermore, by combining sparse frame sampling, feature extraction, and aggregation processing to determine image features, the image features can have a strong ability to characterize the characteristics of the sample video. Moreover, by combining enhancement processing to determine image features, the ability of image features to characterize the characteristics of the sample video can be further improved, thereby making the image features have high accuracy and reliability.

[0141] In some embodiments, the sample video has a video title; the method for obtaining the text features of the sample video may include the following steps:

[0142] The first step is to perform word tokenization on the video title to obtain the word embedding of the video title.

[0143] The second step is to obtain the word features corresponding to each word in the word sequence, and generate text features (title embedding) based on the word features corresponding to each word.

[0144] For example, for such Figure 6 The "video title" shown is as follows: Figure 6 The "word segmentation" process shown above produces a word sequence. Input the word sequence into a program like this: Figure 6 The "word vector model" shown outputs word features. The word features are then input into a system like... Figure 6 The "Text Classification Model" (TextCNN) shown outputs as follows: Figure 6 The "text features" shown.

[0145] In this embodiment, by using the word sequence obtained from word segmentation to determine the word features corresponding to each word, and generating text features based on each word feature, the text features can be highly consistent with the video title, thereby improving the accuracy and reliability of the text features.

[0146] In some embodiments, the sample video has an audio signal; the method for obtaining the audio features of the sample video includes the following steps:

[0147] The first step is to convert the audio signal from the time domain to the frequency domain using Fourier transform.

[0148] The second step is to obtain the Mel spectrogram of the audio signal in the frequency domain and generate audio features based on the Mel spectrogram.

[0149] For example, such as Figure 6 The "audio signal" shown consists of multiple single-frequency sound waves, and the audio signal is transmitted through, as shown in the figure below, to form a sound wave. Figure 6The "Fast Fourier Transform" (FFT) shown transforms the audio signal from the time domain to the frequency domain, resulting in the following: Figure 6 The Mel spectrogram is shown below. Input the Mel spectrogram into... Figure 6 The "Network Symptom Quantification Models" (PANNs) shown output as follows: Figure 6 The “audio embedding” shown.

[0150] In this embodiment, by converting the audio signal from the time domain to the frequency domain and obtaining audio features through Mel spectrograms, the audio features can accurately characterize the features of the sample video in the audio mode.

[0151] S502: Extract sample text information from sample videos.

[0152] Generally, text information has a fixed format, such as "#xxx", so sample text information of sample videos can be obtained based on this format.

[0153] For example, such as Figure 6 As shown, the "sample title" includes "Show some clues, let me know you're thinking of me too #emotionalcopywriting #gesturedance" and the "sample copywriting information" includes "#emotionalcopywriting #gesturedance".

[0154] S503: Based on the pre-constructed vocabulary matrix, determine the feature vector corresponding to the sample copy information, and define the feature vector as the copy feature.

[0155] The vocabulary matrix is ​​used to represent the correspondence between historical copywriting information and feature vectors. The historical copywriting information includes sample copywriting information.

[0156] In other words, a correspondence between text information and feature vectors (i.e., a word matrix) can be pre-constructed. When sample text information of a sample video is obtained, the feature vector corresponding to the sample text information can be determined from the word matrix. This feature vector is the text feature.

[0157] In this embodiment, the efficiency and reliability of determining copywriting features can be improved by combining a word matrix.

[0158] In some embodiments, a method for constructing a vocabulary matrix may include the following steps:

[0159] First step: Obtain historical copywriting information.

[0160] Among them, historical text information is text information that characterizes the type attributes of historical videos, and historical videos have historical text information.

[0161] The second step is to extract features from the historical text information to obtain feature vectors.

[0162] The third step is to construct a vocabulary matrix to represent the mapping relationship between historical text information and feature vectors.

[0163] For example, historical copywriting information may include copywriting information such as #gesturedance, #heartdance, #magicalcutebabygrowthshow, #justchangename, etc. For each piece of historical copywriting information, the feature vector corresponding to that piece of historical copywriting information can be determined, such as determining the feature vector of the historical copywriting information "#gesturedance", the feature vector of the historical copywriting information "#heartdance", etc., to construct a word matrix.

[0164] In some embodiments, in order to make the vocabulary matrix more comprehensive and to improve the effectiveness of determining the text features of sample text information based on the vocabulary matrix, there are multiple historical text information. Accordingly, the frequency of each historical text information can be obtained, and the top n historical text information with the most frequencies can be extracted from each historical text information, where n is a positive integer greater than 1. Feature extraction processing is then performed on the top n historical text information to obtain the feature vector corresponding to each historical text information in the top n historical text information.

[0165] For example, when acquiring historical copywriting information, the frequency of occurrence of each piece of historical copywriting information can be counted, and the top 500 pieces of historical copywriting information with the most occurrences can be retained to construct a 500*64 word matrix. Each row in the word matrix represents the feature vector of the historical copywriting information.

[0166] S504: Based on the gated multi-mode unit, image features, text features, and audio features are fused to obtain the semantic features of the sample video.

[0167] For example, image features, text features, and audio features are input into a system such as... Figure 6 The output of the "Gated Multimodal Unit" (GMU) shown is as follows: Figure 6 The "semantic features" shown.

[0168] The principle of fusion processing of gated multi-mode units is as follows: Figure 2 As shown, it will not be elaborated further here.

[0169] In this embodiment, semantic features are determined based on gated multimodal units, which integrates features from image modality, title modality, and audio modality, thereby making the semantic features more accurate and reliable.

[0170] S505: The semantic features and copywriting features are fused to obtain the fused features.

[0171] Similarly, the fusion process can be the splicing process described in the above embodiments.

[0172] For example, semantic features and copywriting features can be input into such... Figure 6 The "feature fusion model" shown outputs as follows: Figure 6 The "fusion embedding" shown.

[0173] In some embodiments, the number of sample text information may be multiple, such as Figure 6 The "sample text information" shown includes "#MagicCuteBabyGrowthShow, #EmotionalText, #GestureDance". This allows us to determine the text features corresponding to each sample text information and then calculate the average of these features. Figure 6 The "average vector" (AVG) shown in the figure yields the following: Figure 6 The “copywriting embedding” shown.

[0174] S506: A video classification model is trained based on image features, text features, audio features, and fused features.

[0175] Among them, the video classification model is used to classify the videos to be classified.

[0176] In some embodiments, S506 may include the following steps:

[0177] The first step is to classify the sample video based on image features to obtain the first classification loss information, classify the sample video based on text features to obtain the second classification loss information, classify the sample video based on audio features to obtain the third classification loss information, and classify the sample video based on fusion features to obtain the fourth classification loss information.

[0178] For example, image features can be input into a classifier, which outputs a first predicted classification result to characterize the type attribute of the sample video. The difference between the first predicted classification result and the labeled classification result (i.e., the true type attribute of the sample video) is calculated to obtain the first classification loss information. Similarly, other classification loss information (i.e., second classification loss information, third classification loss information, and fourth classification loss information) can be obtained.

[0179] The second step is to train a video classification model based on the first, second, third, and fourth classification loss information.

[0180] In this embodiment, a classification model for the video is selected by combining the classification loss information corresponding to each of the multimodal features, so that the video classification model can meet the classification requirements of the video to be classified from each modality, thereby improving the classification accuracy and reliability of the video classification model.

[0181] In some embodiments, the second step can be implemented using the cross-entropy loss function, wherein the cross-entropy loss function L is formulated as follows:

[0182]

[0183] Where N is the number of sample videos, i is the number of sample videos, c is the type attribute, M is the category attribute M, y is the indicator variable (i.e., the sign function), and p ic Let c be the predicted probability of sample video i with type attribute c.

[0184] In some embodiments, based on such Figure 4 or Figure 5 After training the video classification model as shown in the embodiment, the video to be classified can be classified based on the video classification model to obtain the classification result of the video to be classified.

[0185] For example, a video to be classified can be obtained, and the video to be classified can be input into a video classification model to output the classification result of the video to be classified.

[0186] Please see Figure 7 , Figure 7 This is an example diagram of a video processing apparatus according to an embodiment of the present disclosure.

[0187] like Figure 7 As shown, the device 700 includes:

[0188] The first acquisition unit 701 is used to acquire image features, text features, audio features, and text information of the video to be classified.

[0189] The first determining unit 702 is used to determine the text features of the video to be classified based on the word vectors corresponding to each word in the text information.

[0190] The first fusion unit 703 is used to perform fusion processing on the image features, the text features, and the audio features using a gated multi-mode unit to obtain the semantic features of the video to be classified.

[0191] The first splicing unit 704 is used to splice the semantic features and the text features to obtain the fusion features of the video to be classified.

[0192] The second determining unit 705 is used to determine the classification result of the video to be classified based on the fusion features.

[0193] Please see Figure 8 , Figure 8 This is an example diagram of a video processing apparatus according to another embodiment of the present disclosure.

[0194] like Figure 8 As shown, the device 800 includes:

[0195] The first acquisition unit 801 is used to acquire image features, text features, audio features, and text information of the video to be classified.

[0196] Combination Figure 8 It is understood that, in some embodiments, the first acquisition unit 801 may include:

[0197] The first processing subunit 80111 is used to perform sparse frame sampling processing on the video to be classified to obtain multiple video images, wherein any two video images in the multiple video images are non-adjacent frames in the video to be classified.

[0198] The first extraction subunit 80112 is used to extract the features corresponding to each frame of video image.

[0199] The first aggregation subunit 80113 is used to aggregate the features corresponding to each frame of video image to obtain aggregated features.

[0200] The first enhancement subunit 80114 is used to enhance the aggregated features to obtain the image features.

[0201] Combination Figure 8 It is understood that, in some embodiments, the video to be classified has a video title; the first acquisition unit 801 may include:

[0202] The first word segmentation subunit 80115 is used to perform word segmentation on the video title to obtain the word sequence of the video title.

[0203] The first acquisition subunit 80116 is used to acquire the word features corresponding to each word in the word sequence;

[0204] The first generation subunit 80117 is used to generate the text features based on the word features corresponding to each word.

[0205] Combination Figure 8 It is understood that in some embodiments, the video to be classified has an audio signal; the first acquisition unit 801 may include:

[0206] The first conversion subunit 80118 is used to convert the audio signal from the time domain to the frequency domain through Fourier transform.

[0207] The second acquisition subunit 80119 is used to acquire the Mel spectrum of the audio signal in the frequency domain.

[0208] The second generation subunit 80120 is used to generate audio features based on the Mel spectrogram.

[0209] The second acquisition unit 802 is used to acquire historical text information, wherein the historical text information is text information that characterizes the type attributes of historical videos, and the historical videos have the historical text information.

[0210] The first extraction unit 803 is used to perform feature extraction processing on the historical text information to obtain a feature vector.

[0211] The first construction unit 804 is used to construct the vocabulary matrix that represents the mapping relationship between historical text information and feature vectors.

[0212] In some embodiments, if there are multiple historical text information items, the second acquisition unit 802 is used to acquire the number of times each historical text information item corresponds to, and extract the top n historical text information items with the most times from each historical text information item, where n is a positive integer greater than 1.

[0213] Furthermore, the first extraction unit 803 is used to perform feature extraction processing on the first n historical copy information to obtain the feature vector corresponding to each historical copy information in the first n historical copy information.

[0214] The first determining unit 805 is used to determine the feature vector corresponding to the copy information based on the pre-constructed word matrix, and to determine the feature vector as the copy feature.

[0215] The vocabulary matrix is ​​used to represent the correspondence between historical text information and feature vectors.

[0216] The first fusion unit 806 is used to perform fusion processing on the image features, the text features, and the audio features using a gated multi-mode unit to obtain the semantic features of the video to be classified.

[0217] The first splicing unit 807 is used to splice the semantic features and the text features to obtain the fusion features of the video to be classified.

[0218] The second determining unit 808 is used to determine the classification result of the video to be classified based on the fusion features.

[0219] Please see Figure 9, Figure 9 This is an example diagram of a video processing apparatus according to another embodiment of the present disclosure.

[0220] like Figure 9 As shown, the device 900 includes:

[0221] The third acquisition unit 901 is used to acquire image features, text features, and audio features of the sample video.

[0222] The second extraction unit 902 is used to extract the sample text information of the sample video.

[0223] The third determining unit 903 is used to determine the text features of the sample video based on the word vectors corresponding to each word in the sample text information.

[0224] The second fusion unit 904 is used to perform fusion processing on the image features, the text features, and the audio features using a gated multi-mode unit to obtain the semantic features of the sample video.

[0225] The second splicing unit 905 is used to splice the semantic features and the text features to obtain the fusion features of the sample video.

[0226] Training unit 906 is used to train a video classification model based on the image features, the text features, the audio features, and the fused features, wherein the video classification model is used to classify the video to be classified.

[0227] Please see Figure 10 , Figure 10 This is an example diagram of a video processing apparatus according to another embodiment of the present disclosure.

[0228] like Figure 10 As shown, the device 1000 includes:

[0229] The third acquisition unit 1001 is used to acquire image features, text features, and audio features of the sample video.

[0230] Combination Figure 10 It is understood that, in some embodiments, the third acquisition unit 1001 may include:

[0231] The second processing subunit 100111 is used to perform sparse frame sampling processing on the sample video to obtain multiple sample images, wherein any two sample images in the multiple sample images are non-adjacent frames in the sample video.

[0232] The second extraction subunit 100112 is used to extract the features corresponding to each frame of sample images.

[0233] The second aggregation subunit 100113 is used to aggregate the features corresponding to each frame of sample images to obtain aggregated features.

[0234] The second enhancement subunit 100114 is used to enhance the aggregated features to obtain the image features.

[0235] Combination Figure 10 It is understood that, in some embodiments, the sample video has a video title; the third acquisition unit 1001 may include:

[0236] The second word segmentation subunit 100115 is used to perform word segmentation on the video title to obtain the word sequence of the video title.

[0237] The third acquisition subunit 100116 is used to acquire the word features corresponding to each word in the word sequence.

[0238] The third generation subunit 100117 is used to generate the text features based on the word features corresponding to each word.

[0239] Combination Figure 10 It is understood that in some embodiments, the sample video has an audio signal; the first acquisition unit 1001 may include:

[0240] The second conversion subunit 100118 is used to convert the audio signal from the time domain to the frequency domain through Fourier transform.

[0241] The third acquisition subunit 100119 is used to acquire the Mel spectrum of the audio signal in the frequency domain.

[0242] The third generation subunit 100120 is used to generate audio features based on the Mel spectrogram.

[0243] The second extraction unit 1002 is used to extract the sample text information of the sample video.

[0244] The fourth acquisition unit 1003 is used to acquire historical text information, wherein the historical text information is text information that characterizes the type attributes of historical videos, and the historical videos have the historical text information.

[0245] The third extraction unit 1004 is used to perform feature extraction processing on the historical text information to obtain a feature vector.

[0246] The second construction unit 1005 is used to construct the vocabulary matrix that represents the mapping relationship between historical text information and feature vectors.

[0247] In some embodiments, if there are multiple historical copywriting information, the fourth acquisition unit 1003 is used to acquire the number of times each historical copywriting information corresponds to, and extract the top n historical copywriting information with the most times from each historical copywriting information, where n is a positive integer greater than 1.

[0248] Furthermore, the third extraction unit 1004 is used to perform feature extraction processing on the first n historical copy information to obtain the feature vector corresponding to each historical copy information in the first n historical copy information.

[0249] The third determining unit 1006 is used to determine the feature vector corresponding to the sample text information based on the pre-constructed word matrix, and to determine the feature vector as the text feature.

[0250] The vocabulary matrix is ​​used to represent the correspondence between historical text information and feature vectors.

[0251] The second fusion unit 1007 is used to perform fusion processing on the image features, the text features, and the audio features using a gated multi-mode unit to obtain the semantic features of the sample video.

[0252] The second splicing unit 1008 is used to splice the semantic features and the text features to obtain the fusion features of the sample video.

[0253] Training unit 1009 is used to train a video classification model based on the image features, the text features, the audio features, and the fused features, wherein the video classification model is used to classify the video to be classified.

[0254] Combination Figure 10 It is understood that, in some embodiments, the training unit 1009 includes:

[0255] The classification subunit 10091 is used to classify the sample video according to the image features to obtain first classification loss information, classify the sample video according to the text features to obtain second classification loss information, classify the sample video according to the audio features to obtain third classification loss information, and classify the sample video according to the fusion features to obtain fourth classification loss information.

[0256] Training subunit 10092 is used to train a classification model for the video based on the first classification loss information, the second classification loss information, the third classification loss information, and the fourth classification loss information.

[0257] According to embodiments of this disclosure, this disclosure also provides an electronic device and a readable storage medium.

[0258] According to embodiments of this disclosure, this disclosure also provides a computer program product, the program product comprising: a computer program stored in a readable storage medium, at least one processor of an electronic device being able to read the computer program from the readable storage medium, and the at least one processor executing the computer program causing the electronic device to perform the scheme provided in any of the above embodiments.

[0259] refer to Figure 11 The diagram illustrates a structural schematic of an electronic device 1100 suitable for implementing embodiments of the present disclosure. The electronic device 1100 can be a terminal device or a server. The terminal device can include, but is not limited to, mobile terminals such as mobile phones, laptops, digital radio receivers, personal digital assistants (PDAs), portable Android devices (PADs), portable media players (PMPs), and in-vehicle terminals (e.g., in-vehicle navigation terminals), as well as fixed terminals such as digital TVs and desktop computers. Figure 11 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein.

[0260] like Figure 11 As shown, electronic device 1100 may include a processing unit (e.g., a central processing unit, a graphics processing unit, etc.) 1101, which can perform various appropriate actions and processes according to a program stored in read-only memory (ROM) 1102 or a program loaded from storage device 1108 into random access memory (RAM) 1103. RAM 1103 also stores various programs and data required for the operation of electronic device 1100. The processing unit 1101, ROM 1102, and RAM 1103 are interconnected via bus 1104. Input / output (I / O) interface 1105 is also connected to bus 1104.

[0261] Typically, the following devices can be connected to I / O interface 1105: input devices 1106 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 1107 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 1108 including, for example, magnetic tapes, hard disks, etc.; and communication devices 1109. Communication device 1109 allows electronic device 1100 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 11 An electronic device 1100 with various devices is shown; however, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively.

[0262] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication device 1109, or installed from storage device 1108, or installed from ROM 1102. When the computer program is executed by processing device 1101, it performs the functions defined in the methods of embodiments of this disclosure.

[0263] It should be noted that the computer-readable medium described in this disclosure can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this disclosure, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in connection with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0264] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device.

[0265] The aforementioned computer-readable medium carries one or more programs, which, when executed by the electronic device, cause the electronic device to perform the methods shown in the above embodiments.

[0266] Computer program code for performing the operations of this disclosure can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a Local Area Network (LAN) or a Wide Area Network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0267] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0268] The units described in the embodiments of this disclosure can be implemented in software or in hardware. The name of a unit does not necessarily limit the unit itself; for example, the first acquisition unit can also be described as "a unit that acquires at least two Internet Protocol addresses".

[0269] The functions described above in this document can be performed, at least in part, by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: Field Programmable Gate Arrays (FPGAs), Application-Specific Integrated Circuits (ASICs), Application Standard Products (ASSPs), System-on-Chip (SoCs), Complex Programmable Logic Devices (CPLDs), and so on.

[0270] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0271] In a first aspect, according to one or more embodiments of this disclosure, a video processing method is provided, comprising:

[0272] The image features, text features, audio features, and text information of the video to be classified are obtained, and the text features of the video to be classified are determined based on the word vectors corresponding to each word in the text information.

[0273] The image features, text features, and audio features are fused using a gated multi-mode unit to obtain the semantic features of the video to be classified.

[0274] The semantic features and the textual features are concatenated to obtain the fusion features of the video to be classified, and the classification result of the video to be classified is determined based on the fusion features.

[0275] According to one or more embodiments of this disclosure, determining the text features of the video to be classified based on the word vectors corresponding to each word in the text information includes:

[0276] Based on a pre-constructed vocabulary matrix, a feature vector corresponding to the text information is determined, and the feature vector is identified as the text feature.

[0277] The vocabulary matrix is ​​used to represent the correspondence between historical text information and feature vectors.

[0278] According to one or more embodiments of this disclosure, before determining the feature vector corresponding to the text information based on a pre-constructed vocabulary matrix, the method further includes:

[0279] Obtain historical text information, wherein the historical text information is text information that characterizes the type attributes of historical videos, and the historical videos have the historical text information;

[0280] The historical text information is subjected to feature extraction processing to obtain a feature vector;

[0281] Construct the vocabulary matrix to represent the mapping relationship between historical text information and feature vectors.

[0282] According to one or more embodiments of this disclosure, if the number of historical text information items is multiple, then after obtaining the historical text information items, the method further includes:

[0283] Obtain the frequency of each historical text message, and extract the n historical text messages with the most frequencies from each historical text message, where n is a positive integer greater than 1;

[0284] Furthermore, feature extraction processing is performed on the historical text information to obtain a feature vector, including:

[0285] Feature extraction processing is performed on the first n historical copy information to obtain the feature vector corresponding to each historical copy information in the first n historical copy information.

[0286] According to one or more embodiments of this disclosure, image features of a video to be classified are obtained, including:

[0287] The video to be classified is subjected to sparse frame sampling processing to obtain multiple video images, wherein any two video images in the multiple video images are non-adjacent frames in the video to be classified.

[0288] Extract the features corresponding to each frame of the video image, aggregate the features corresponding to each frame of the video image to obtain aggregated features, and enhance the aggregated features to obtain the image features.

[0289] According to one or more embodiments of this disclosure, the video to be classified has a video title; obtaining the text features of the video to be classified includes:

[0290] The video title is segmented into words to obtain the word sequence of the video title;

[0291] Obtain the word features corresponding to each word in the word sequence, and generate the text features based on the word features corresponding to each word.

[0292] According to one or more embodiments of this disclosure, the video to be classified has an audio signal; the acquisition of the audio features of the video to be classified includes:

[0293] The audio signal is converted from the time domain to the frequency domain using Fourier transform;

[0294] Obtain the Mel spectrogram of the audio signal in the frequency domain, and generate the audio features based on the Mel spectrogram.

[0295] Secondly, according to one or more embodiments of this disclosure, a video processing method is provided, comprising:

[0296] Obtain image features, text features, and audio features from the sample video;

[0297] Extract the sample text information from the sample video, and determine the text features of the sample video based on the word vectors corresponding to each word in the sample text information;

[0298] The image features, text features, and audio features are fused using a gated multi-mode unit to obtain the semantic features of the sample video. The semantic features are then concatenated with the text features to obtain the fused features of the sample video.

[0299] A video classification model is trained based on the image features, text features, audio features, and fusion features, wherein the video classification model is used to classify the video to be classified.

[0300] According to one or more embodiments of this disclosure, the step of training a video classification model based on the image features, the text features, the audio features, and the fused features includes:

[0301] The sample video is classified according to the image features to obtain first classification loss information; the sample video is classified according to the text features to obtain second classification loss information; the sample video is classified according to the audio features to obtain third classification loss information; and the sample video is classified according to the fusion features to obtain fourth classification loss information.

[0302] A classification model for the video is trained based on the first classification loss information, the second classification loss information, the third classification loss information, and the fourth classification loss information.

[0303] According to one or more embodiments of this disclosure, determining the textual features of the sample video based on the sample textual information includes:

[0304] Based on the pre-constructed vocabulary matrix, the feature vector corresponding to the sample text information is determined, and the feature vector is determined as the text feature;

[0305] The vocabulary matrix is ​​used to represent the correspondence between historical copywriting information and feature vectors, and the historical copywriting information includes the sample copywriting information.

[0306] According to one or more embodiments of this disclosure, before determining the feature vector corresponding to the sample text information based on a pre-constructed vocabulary matrix, the method further includes:

[0307] Obtain historical text information, wherein the historical text information is text information that characterizes the type attributes of historical videos, and the historical videos have the historical text information;

[0308] The historical text information is subjected to feature extraction processing to obtain a feature vector;

[0309] Construct the vocabulary matrix to represent the mapping relationship between historical text information and feature vectors.

[0310] According to one or more embodiments of this disclosure, if the number of historical text information items is multiple, then after obtaining the historical text information items, the method further includes:

[0311] Obtain the frequency of each historical text message, and extract the n historical text messages with the most frequencies from each historical text message, where n is a positive integer greater than 1;

[0312] Furthermore, feature extraction processing is performed on the historical text information to obtain a feature vector, including:

[0313] Feature extraction processing is performed on the first n historical copy information to obtain the feature vector corresponding to each historical copy information in the first n historical copy information.

[0314] According to one or more embodiments of this disclosure, the image features, text features, audio features, and copywriting features are fused to obtain the fused features of the sample video, including:

[0315] The semantic features of the sample video are obtained by fusing the image features, text features, and audio features using a gated multi-mode unit.

[0316] The semantic features and the textual features are fused together to obtain the fused features.

[0317] According to one or more embodiments of this disclosure, the step of training a video classification model based on the image features, the text features, the audio features, and the fused features includes:

[0318] The sample video is classified according to the image features to obtain first classification loss information; the sample video is classified according to the text features to obtain second classification loss information; the sample video is classified according to the audio features to obtain third classification loss information; and the sample video is classified according to the fusion features to obtain fourth classification loss information.

[0319] A classification model for the video is trained based on the first classification loss information, the second classification loss information, the third classification loss information, and the fourth classification loss information.

[0320] According to one or more embodiments of this disclosure, obtaining image features of the sample video includes:

[0321] The sample video is subjected to sparse frame sampling processing to obtain multiple sample images, wherein any two sample images in the multiple sample images are non-adjacent frames in the sample video.

[0322] Extract the features corresponding to each frame of the sample image, aggregate the features corresponding to each frame of the sample image to obtain aggregated features, and enhance the aggregated features to obtain the image features.

[0323] According to one or more embodiments of this disclosure, the sample video has a video title; obtaining the text features of the sample video includes:

[0324] The video title is segmented into words to obtain the word sequence of the video title;

[0325] Obtain the word features corresponding to each word in the word sequence, and generate the text features based on the word features corresponding to each word.

[0326] According to one or more embodiments of this disclosure, the sample video has an audio signal; acquiring the audio features of the sample video includes:

[0327] The audio signal is converted from the time domain to the frequency domain using Fourier transform;

[0328] Obtain the Mel spectrogram of the audio signal in the frequency domain, and generate the audio features based on the Mel spectrogram.

[0329] Thirdly, according to one or more embodiments of this disclosure, a video processing apparatus is provided, comprising:

[0330] The first acquisition unit is used to acquire image features, text features, audio features, and text information of the video to be classified;

[0331] The first determining unit is used to determine the text features of the video to be classified based on the word vectors corresponding to each word in the text information.

[0332] The first fusion unit is used to perform fusion processing on the image features, the text features, and the audio features using a gated multi-mode unit to obtain the semantic features of the video to be classified.

[0333] The first splicing unit is used to splice the semantic features and the text features to obtain the fusion features of the video to be classified.

[0334] The second determining unit is used to determine the classification result of the video to be classified based on the fusion features.

[0335] According to one or more embodiments of this disclosure, the first acquisition unit may include:

[0336] The first processing subunit is used to perform sparse frame sampling processing on the video to be classified to obtain multiple video images, wherein any two video images in the multiple video images are non-adjacent frames in the video to be classified.

[0337] The first extraction subunit is used to extract the features corresponding to each frame of video image;

[0338] The first aggregation subunit is used to aggregate the features corresponding to each frame of video image to obtain aggregated features.

[0339] The first enhancement subunit is used to enhance the aggregated features to obtain the image features.

[0340] According to one or more embodiments of this disclosure, the video to be classified has a video title; the first acquisition unit may include:

[0341] The first word segmentation subunit is used to perform word segmentation on the video title to obtain the word sequence of the video title;

[0342] The first acquisition subunit is used to acquire the word features corresponding to each word in the word sequence;

[0343] The first generation subunit is used to generate the text features based on the word features corresponding to each word.

[0344] According to one or more embodiments of this disclosure, the video to be classified has an audio signal; the first acquisition unit may include:

[0345] The first conversion subunit is used to convert the audio signal from the time domain to the frequency domain through Fourier transform;

[0346] The second acquisition subunit is used to acquire the Mel spectrum of the audio signal in the frequency domain;

[0347] The second generation subunit is used to generate audio features based on the Mel spectrogram.

[0348] According to one or more embodiments of this disclosure, the apparatus further includes:

[0349] The second acquisition unit is used to acquire historical text information, wherein the historical text information is text information that characterizes the type attribute of the historical video, and the historical video has the historical text information;

[0350] The first extraction unit is used to perform feature extraction processing on the historical text information to obtain a feature vector.

[0351] The first construction unit is used to construct the vocabulary matrix, which represents the mapping relationship between historical text information and feature vectors.

[0352] According to one or more embodiments of this disclosure, if the number of historical text information is multiple, the second acquisition unit is used to acquire the number of times corresponding to each historical text information, and extract the top n historical text information with the most times from each historical text information, where n is a positive integer greater than 1.

[0353] Furthermore, the first extraction unit is used to perform feature extraction processing on the first n historical copy information to obtain the feature vector corresponding to each historical copy information in the first n historical copy information.

[0354] Furthermore, the first determining unit is configured to determine a feature vector corresponding to the copywriting information based on a pre-constructed vocabulary matrix, and to determine the feature vector as the copywriting feature.

[0355] The vocabulary matrix is ​​used to represent the correspondence between historical text information and feature vectors.

[0356] Fourthly, according to one or more embodiments of this disclosure, a video processing apparatus is provided, comprising:

[0357] The third acquisition unit is used to acquire image features, text features, and audio features of the sample video;

[0358] The second extraction unit is used to extract the sample text information of the sample video;

[0359] The third determining unit is used to determine the text features of the sample video based on the word vectors corresponding to each word in the sample text information.

[0360] The second fusion unit is used to perform fusion processing on the image features, the text features, and the audio features using a gated multi-mode unit to obtain the semantic features of the sample video;

[0361] The second splicing unit is used to splice the semantic features and the text features to obtain the fusion features of the sample video;

[0362] The training unit is used to train a video classification model based on the image features, the text features, the audio features, and the fused features, wherein the video classification model is used to classify the video to be classified.

[0363] According to one or more embodiments of this disclosure, the third acquisition unit may include:

[0364] The second processing subunit is used to perform sparse frame sampling processing on the sample video to obtain multiple sample images, wherein any two sample images in the multiple sample images are non-adjacent frames in the sample video.

[0365] The second extraction subunit is used to extract the features corresponding to each frame of sample images.

[0366] The second aggregation subunit is used to aggregate the features corresponding to each frame of sample images to obtain aggregated features.

[0367] The second enhancement subunit is used to enhance the aggregated features to obtain the image features.

[0368] According to one or more embodiments of this disclosure, the sample video has a video title; the third acquisition unit may include:

[0369] The second word segmentation subunit is used to segment the video title to obtain the word sequence of the video title;

[0370] The third acquisition subunit is used to acquire the word features corresponding to each word in the word sequence;

[0371] The third generation subunit is used to generate the text features based on the word features corresponding to each word.

[0372] According to one or more embodiments of this disclosure, the sample video has an audio signal; the first acquisition unit may include:

[0373] The second conversion subunit is used to convert the audio signal from the time domain to the frequency domain through Fourier transform;

[0374] The third acquisition subunit is used to acquire the Mel spectrum of the audio signal in the frequency domain;

[0375] The third generation subunit is used to generate audio features based on the Mel spectrogram.

[0376] According to one or more embodiments of this disclosure, the apparatus further includes:

[0377] The fourth acquisition unit is used to acquire historical text information, wherein the historical text information is text information that characterizes the type attribute of the historical video, and the historical video has the historical text information;

[0378] The third extraction unit is used to perform feature extraction processing on the historical text information to obtain a feature vector;

[0379] The second construction unit is used to construct the vocabulary matrix that represents the mapping relationship between historical text information and feature vectors.

[0380] According to one or more embodiments of this disclosure, if the number of historical text information is multiple, the fourth acquisition unit is used to acquire the number of times corresponding to each historical text information, and extract the top n historical text information with the most times from each historical text information, where n is a positive integer greater than 1.

[0381] Furthermore, the third extraction unit is used to perform feature extraction processing on the first n historical copy information to obtain the feature vector corresponding to each historical copy information in the first n historical copy information.

[0382] According to one or more embodiments of this disclosure, a third determining unit is configured to determine a feature vector corresponding to the sample text information based on a pre-constructed vocabulary matrix, and to determine the feature vector as the text feature.

[0383] The vocabulary matrix is ​​used to represent the correspondence between historical text information and feature vectors.

[0384] According to one or more embodiments of this disclosure, the training unit includes:

[0385] The classification subunit is used to classify the sample video according to the image features to obtain first classification loss information, classify the sample video according to the text features to obtain second classification loss information, classify the sample video according to the audio features to obtain third classification loss information, and classify the sample video according to the fusion features to obtain fourth classification loss information.

[0386] The training subunit is used to train a classification model for the video based on the first classification loss information, the second classification loss information, the third classification loss information, and the fourth classification loss information.

[0387] Fifthly, according to one or more embodiments of the present disclosure, an electronic device is provided, comprising: at least one processor and a memory;

[0388] The memory stores computer-executed instructions;

[0389] The at least one processor executes computer execution instructions stored in the memory, causing the at least one processor to perform the information display method as described in the first aspect and various possible designs of the first aspect.

[0390] Fourthly, according to one or more embodiments of the present disclosure, a computer-readable storage medium is provided, wherein computer-executable instructions are stored therein, and when a processor executes the computer-executable instructions, the information display method described in the first aspect and various possible designs of the first aspect is implemented.

[0391] The above description is merely a preferred embodiment of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features disclosed in this disclosure that have similar functions.

[0392] Furthermore, while the operations are described in a specific order, this should not be construed as requiring these operations to be performed in the specific order shown or in a sequential order. In certain environments, multitasking and parallel processing may be advantageous. Similarly, while several specific implementation details are included in the above discussion, these should not be construed as limiting the scope of this disclosure. Certain features described in the context of individual embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments.

[0393] Although the subject matter has been described using language specific to structural features and / or methodological logic, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. Rather, the specific features and actions described above are merely illustrative examples of implementing the claims.

Claims

1. A video processing method, characterized in that, include: The process involves acquiring image features, text features, audio features, and text information of the video to be classified, and determining the feature vector corresponding to the text information based on a pre-constructed vocabulary matrix. This feature vector is then used as the text feature of the video to be classified. The vocabulary matrix represents the correspondence between historical text information and feature vectors. The feature vector is obtained by extracting features from historical text information, which is text information representing the type attributes of historical videos, and these historical videos possess this historical text information. The image features, text features, and audio features are fused using a gated multi-mode unit to obtain the semantic features of the video to be classified. The semantic features and the textual features are concatenated to obtain the fusion features of the video to be classified, and the classification result of the video to be classified is determined based on the fusion features.

2. The method according to claim 1, characterized in that, Before determining the feature vector corresponding to the text information based on the pre-constructed vocabulary matrix, the method further includes: Obtain historical copywriting information; Construct the vocabulary matrix to represent the mapping relationship between historical text information and feature vectors.

3. The method according to claim 2, characterized in that, If there are multiple pieces of historical text information, then after obtaining the historical text information, the method further includes: Obtain the frequency of each historical text message, and extract the n historical text messages with the most frequencies from each historical text message, where n is a positive integer greater than 1; Furthermore, feature extraction processing is performed on the historical text information to obtain a feature vector, including: Feature extraction processing is performed on the first n historical copy information to obtain the feature vector corresponding to each historical copy information in the first n historical copy information.

4. The method according to any one of claims 1-3, characterized in that, Obtain image features of the video to be classified, including: The video to be classified is subjected to sparse frame sampling processing to obtain multiple video images, wherein any two video images in the multiple video images are non-adjacent frames in the video to be classified. Extract the features corresponding to each frame of the video image, aggregate the features corresponding to each frame of the video image to obtain aggregated features, and enhance the aggregated features to obtain the image features.

5. The method according to any one of claims 1-3, characterized in that, The video to be classified has a video title; the text features of the video to be classified are obtained, including: The video title is segmented into words to obtain the word sequence of the video title; Obtain the word features corresponding to each word in the word sequence, and generate the text features based on the word features corresponding to each word.

6. A video processing method, characterized in that, include: Obtain image features, text features, and audio features from the sample video; Extract sample text information from the sample video, and determine the feature vector corresponding to the sample text information based on a pre-constructed vocabulary matrix, and determine the feature vector as the text feature of the sample video; wherein, the vocabulary matrix is ​​used to represent the correspondence between historical text information and feature vector; the feature vector is obtained by feature extraction processing of historical text information, wherein, the historical text information is text information representing the type attribute of historical videos, and the historical videos have the historical text information; The image features, text features, and audio features are fused using a gated multi-mode unit to obtain the semantic features of the sample video. The semantic features are then concatenated with the text features to obtain the fused features of the sample video. A video classification model is trained based on the image features, text features, audio features, and fusion features, wherein the video classification model is used to classify the video to be classified.

7. The method according to claim 6, characterized in that, The video classification model trained based on the image features, text features, audio features, and fused features includes: The sample video is classified according to the image features to obtain first classification loss information; the sample video is classified according to the text features to obtain second classification loss information; the sample video is classified according to the audio features to obtain third classification loss information; and the sample video is classified according to the fusion features to obtain fourth classification loss information. A classification model for the video is trained based on the first classification loss information, the second classification loss information, the third classification loss information, and the fourth classification loss information.

8. A video processing apparatus, characterized in that, include: The first acquisition unit is used to acquire image features, text features, audio features, and text information of the video to be classified; The first determining unit is configured to determine a feature vector corresponding to the text information based on a pre-constructed vocabulary matrix, and to determine the feature vector as the text feature of the video to be classified; wherein, the vocabulary matrix is ​​used to represent the correspondence between historical text information and feature vector; the feature vector is obtained by feature extraction processing of historical text information, wherein the historical text information is text information representing the type attribute of historical videos, and the historical videos have the historical text information; The first fusion unit is used to perform fusion processing on the image features, the text features, and the audio features using a gated multi-mode unit to obtain the semantic features of the video to be classified. The first splicing unit is used to splice the semantic features and the text features to obtain the fusion features of the video to be classified. The second determining unit is used to determine the classification result of the video to be classified based on the fusion features.

9. A video processing apparatus, characterized in that, include: The third acquisition unit is used to acquire image features, text features, and audio features of the sample video; The second extraction unit is used to extract the sample text information of the sample video; The third determining unit is used to determine the feature vector corresponding to the sample text information based on a pre-constructed vocabulary matrix, and to determine the feature vector as the text feature of the sample video; wherein, the vocabulary matrix is ​​used to represent the correspondence between historical text information and feature vector; the feature vector is obtained by feature extraction processing of historical text information, wherein the historical text information is text information representing the type attribute of historical videos, and the historical videos have the historical text information; The second fusion unit is used to perform fusion processing on the image features, the text features, and the audio features using a gated multi-mode unit to obtain the semantic features of the sample video; The second splicing unit is used to splice the semantic features and the text features to obtain the fusion features of the sample video; The training unit is used to train a video classification model based on the image features, the text features, the audio features, and the fused features, wherein the video classification model is used to classify the video to be classified.

10. An electronic device, characterized in that, include: At least one processor and memory; The memory stores computer-executed instructions; The at least one processor executes computer execution instructions stored in the memory, causing the at least one processor to perform the method as described in any one of claims 1-5; or, causing the at least one processor to perform the method as described in claim 6 or 7.

11. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, implement the method as described in any one of claims 1-5; or, implement the method as described in claim 6 or 7.

Images