A video classification method and device, electronic equipment and storage medium

Abstract

The application discloses a video classification method and device, electronic equipment and a storage medium; an embodiment of the application obtains video content information of a target video in at least one mode; semantic extraction is performed on the video content information of the target video in at least one mode to obtain semantic feature information of the target video in at least one mode; feature enhancement processing is performed on the semantic feature information of the target video in at least one mode to obtain enhanced feature information of the target video; the enhanced feature information is fused with the semantic feature information of the target video in at least one mode to obtain target semantic feature information of the target video; and the target video is classified based on the target semantic feature information. The embodiment of the application can enhance the representation of the target semantic feature information by enhancing the features related to the content of the target video, and is beneficial to improving the accuracy of video classification.

Description

Technical Field

[0001] This application relates to the field of computer technology, and more specifically to a video classification method, apparatus, electronic device, and storage medium. Background Technology

[0002] With the development of computer technology, multimedia applications are becoming increasingly widespread, video types are becoming richer, and the number of videos is growing rapidly. The videos available to people are also becoming more diverse. To help users quickly find the videos they want to watch from this vast library, video playback platforms typically categorize their videos. Video categorization plays a crucial role in video management and interest-based recommendations. Furthermore, video categorization technology is widely used in fields such as surveillance, retrieval, and human-computer interaction.

[0003] Current related technologies typically involve first extracting video frames from the video to be classified, obtaining multiple target video images, then extracting image feature information from each target video image using a neural network, transforming the frame-level image feature information into video-level video feature information, and finally classifying the video based on the video feature information. However, the representativeness of the video feature information extracted in this way is relatively weak, resulting in relatively low accuracy of the video classification results. Summary of the Invention

[0004] This application provides a video classification method, apparatus, electronic device, and storage medium, which can enhance the representational power of target semantic feature information by strengthening features related to the target video content, thereby improving the accuracy of video classification.

[0005] This application provides a video classification method, including:

[0006] Obtain video content information of the target video in at least one modality;

[0007] Semantic extraction is performed on the video content information of the target video in at least one modality to obtain the semantic feature information of the target video in at least one modality;

[0008] The semantic feature information of the target video in at least one modality is subjected to feature enhancement processing to obtain the enhanced feature information of the target video;

[0009] The enhanced feature information is fused with the semantic feature information of the target video in at least one modality to obtain the target semantic feature information of the target video;

[0010] The target video is classified based on the target semantic feature information.

[0011] Accordingly, embodiments of this application provide a video classification device, including:

[0012] The acquisition unit is used to acquire video content information of the target video in at least one modality.

[0013] The extraction unit is used to perform semantic extraction on the video content information of the target video in at least one modality to obtain the semantic feature information of the target video in at least one modality;

[0014] The enhancement unit is used to perform feature enhancement processing on the semantic feature information of the target video in at least one modality to obtain the enhanced feature information of the target video.

[0015] The fusion unit is used to fuse the enhanced feature information with the semantic feature information of the target video in at least one modality to obtain the target semantic feature information of the target video;

[0016] A classification unit is used to classify the target video based on the target semantic feature information.

[0017] Optionally, in some embodiments of this application, the enhancement unit may be specifically used to perform logistic regression processing on the semantic feature information of the target video in at least one modality to obtain the enhanced feature information of the target video, wherein the feature values ​​of the features in the enhanced feature information are within a preset numerical range corresponding to the logistic regression.

[0018] Optionally, in some embodiments of this application, the strengthening unit may include a fusion subunit and a strengthening subunit, as follows:

[0019] The fusion subunit is used to fuse the semantic feature information of the target video in various modalities to obtain the fused semantic feature information of the target video.

[0020] The enhancement subunit is used to perform feature enhancement processing on the fused semantic feature information to obtain the enhanced feature information of the target video.

[0021] Optionally, in some embodiments of this application, the fused semantic feature information includes at least one feature;

[0022] Specifically, the enhancement subunit can be used to enhance the features in the fused semantic feature information when the feature value of the feature in the fused semantic feature information is greater than a first preset value; and to compress the features in the fused semantic feature information when the feature value of the feature in the fused semantic feature information is less than a second preset value; and to obtain the enhanced feature information of the target video based on the enhanced features and the compressed features.

[0023] Optionally, in some embodiments of this application, the at least one modality includes an image modality and a text modality;

[0024] The acquisition unit may include an image acquisition subunit and a text acquisition subunit, as follows:

[0025] The image acquisition subunit is used to perform image extraction processing on the target video to obtain an image sequence of the target video, wherein the image sequence is the video content information of the target video in the image mode.

[0026] The text acquisition subunit is used to perform text extraction processing on the target video to obtain a text sequence of the target video, wherein the text sequence is the video content information of the target video in text mode.

[0027] Optionally, in some embodiments of this application, the extraction unit may include an image extraction subunit, a processing subunit, an image fusion subunit, and a text extraction subunit, as follows:

[0028] The image extraction subunit is used to perform semantic extraction on each image in the image sequence to obtain the image semantic feature information of each image in the image sequence;

[0029] The processing subunit is used to process the image semantic feature information of each image based on the image semantic feature information of the preceding and following images of each image;

[0030] The image fusion subunit is used to fuse the image semantic feature information of each processed image to obtain the semantic feature information of the target video in the image modality;

[0031] The text extraction subunit is used to perform semantic extraction on the text sequence to obtain the semantic feature information of the target video in the text modality.

[0032] Optionally, in some embodiments of this application, the fusion unit may include a first fusion subunit, a second fusion subunit, and a determination subunit, as follows:

[0033] The first fusion subunit is used to fuse the enhanced feature information with the semantic feature information of the target video in the image modality to obtain the first feature information;

[0034] The second fusion subunit is used to fuse the enhanced feature information with the semantic feature information of the target video in the text modality to obtain the second feature information;

[0035] A subunit is defined for obtaining target semantic feature information of the target video based on the first feature information and the second feature information.

[0036] Optionally, in some embodiments of this application, the determining subunit may be specifically used to determine the weights corresponding to the first feature information and the second feature information; based on the weights, the first feature information and the second feature information are weighted and fused to obtain the target semantic feature information of the target video.

[0037] An electronic device provided in this application includes a processor and a memory. The memory stores multiple instructions, and the processor loads the instructions to execute the steps in the video classification method provided in this application.

[0038] Furthermore, this application embodiment also provides a storage medium storing a computer program thereon, wherein the computer program, when executed by a processor, implements the steps in the video classification method provided in this application embodiment.

[0039] This application provides a video classification method, apparatus, electronic device, and storage medium, which can acquire video content information of a target video in at least one modality; perform semantic extraction on the video content information of the target video in at least one modality to obtain semantic feature information of the target video in at least one modality; perform feature enhancement processing on the semantic feature information of the target video in at least one modality to obtain enhanced feature information of the target video; fuse the enhanced feature information with the semantic feature information of the target video in at least one modality to obtain target semantic feature information of the target video; and classify the target video based on the target semantic feature information. This application embodiment can enhance the representational power of target semantic feature information by strengthening features related to the target video content, which is beneficial to improving the accuracy of video classification. Attached Figure Description

[0040] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0041] Figure 1a This is a schematic diagram of a scenario for the video classification method provided in the embodiments of this application;

[0042] Figure 1b This is a flowchart of the video classification method provided in the embodiments of this application;

[0043] Figure 2a This is another flowchart of the video classification method provided in the embodiments of this application;

[0044] Figure 2b This is a model framework diagram of the video classification method provided in the embodiments of this application;

[0045] Figure 3a This is a schematic diagram of the structure of the video classification device provided in the embodiments of this application;

[0046] Figure 3b This is another structural schematic diagram of the video classification device provided in the embodiments of this application;

[0047] Figure 3c This is another structural schematic diagram of the video classification device provided in the embodiments of this application;

[0048] Figure 3d This is another structural schematic diagram of the video classification device provided in the embodiments of this application;

[0049] Figure 3e This is another structural schematic diagram of the video classification device provided in the embodiments of this application;

[0050] Figure 4 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0051] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0052] This application provides a video classification method, apparatus, electronic device, and storage medium. Specifically, the video classification apparatus can be integrated into an electronic device, which may be a terminal or server, etc.

[0053] It is understood that the video classification method of this embodiment can be executed on a terminal, on a server, or jointly by a terminal and a server. The above examples should not be construed as limiting this application.

[0054] like Figure 1a As shown, the video classification method is executed jointly by a terminal and a server as an example. The video classification system provided in this application includes a terminal 10 and a server 11, etc.; the terminal 10 and the server 11 are connected via a network, such as a wired or wireless network, etc., wherein the video classification device can be integrated into the server.

[0055] Server 11 can be used to: acquire video content information of a target video in at least one modality; perform semantic extraction on the video content information of the target video in at least one modality to obtain semantic feature information of the target video in at least one modality; perform feature enhancement processing on the semantic feature information of the target video in at least one modality to obtain enhanced feature information of the target video; fuse the enhanced feature information with the semantic feature information of the target video in at least one modality to obtain target semantic feature information of the target video; and classify the target video based on the target semantic feature information. Server 11 can be a single server, a server cluster composed of multiple servers, or a cloud server. In the video classification method or apparatus disclosed in this application, multiple servers can form a blockchain, and the servers are nodes on the blockchain.

[0056] The terminal 10 can receive the classification results of the target video sent by the server 11, and perform services such as recommendation and search based on the classification results. The terminal 10 can include a mobile phone, smart TV, tablet computer, laptop computer, or personal computer (PC). A client can also be set on the terminal 10, which can be an application client or a browser client, etc.

[0057] The step of classifying the target video by the server 11 can also be performed by the terminal 10.

[0058] The video classification method provided in this application relates to computer vision technology and natural language processing in the field of artificial intelligence. This application can enhance the representational power of target semantic feature information, which is beneficial to improving the accuracy of video classification.

[0059] Artificial intelligence (AI) is the theory, methods, technology, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that attempts to understand the essence of intelligence and produce a new kind of intelligent machine that can react in a way similar to human intelligence. AI studies the design principles and implementation methods of various intelligent machines, enabling them to have perception, reasoning, and decision-making capabilities. AI technology is a comprehensive discipline involving a wide range of fields, encompassing both hardware and software technologies. AI software technologies mainly include computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0060] Computer vision (CV) is a science that studies how to enable machines to "see." More specifically, it refers to machine vision, which uses cameras and computers to replace human eyes in recognizing, tracking, and measuring targets, and then performs image processing to create images more suitable for human observation or transmission to instruments. As a scientific discipline, computer vision studies related theories and technologies, attempting to build artificial intelligence systems capable of extracting information from images or multidimensional data. Computer vision technologies typically include image processing, image recognition, image semantic understanding, image retrieval, OCR, video processing, video semantic understanding, video content / behavior recognition, 3D object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous localization and mapping (SLAM), and common biometric recognition technologies such as facial recognition and fingerprint recognition.

[0061] Natural Language Processing (NLP) is an important area within computer science and artificial intelligence. It studies the theories and methods for enabling effective communication between humans and computers using natural language. NLP is a science that integrates linguistics, computer science, and mathematics. Therefore, research in this field involves natural language—the language people use in daily life—and thus it has a close connection with linguistic research. NLP technologies typically include text processing, semantic understanding, machine translation, question answering, and knowledge graphs.

[0062] The following sections provide detailed descriptions of each example. It should be noted that the order in which the embodiments are described is not intended to limit the preferred order of the embodiments.

[0063] This embodiment will be described from the perspective of a video classification device, which can be integrated into an electronic device, such as a server or a terminal.

[0064] The video classification method of this application can be applied to various scenarios that require video classification, such as video recommendation and video search. For example, a video platform needs to recommend videos to users based on their preferences. The video classification method provided in this embodiment can quickly classify massive amounts of videos and make recommendations based on the classification results. Furthermore, the video classification method provided in this embodiment enhances the representation power of target semantic feature information by strengthening features related to the target video content, which is beneficial to improving the accuracy of video recommendations.

[0065] like Figure 1b As shown, the specific process of this video classification method can be as follows:

[0066] 101. Obtain video content information of the target video in at least one modality.

[0067] In this embodiment, the target video is the video to be classified, and its type and duration are not limited. For example, it can be a news video, a historical video, or a variety show video, etc. In addition, the target video can also be a live video.

[0068] Among them, video content information refers to the content information contained in the video. Because the video contains rich visual, auditory and subtitle information, it has multiple modal information, such as image modality, audio modality (i.e., speech modality) and text modality. Based on the analysis and processing of information under each modality, the multimodal video content information of the video can be extracted.

[0069] Optionally, in this embodiment, the at least one modality includes an image modality and a text modality;

[0070] The step "obtain video content information of the target video in at least one modality" may include:

[0071] Image extraction processing is performed on the target video to obtain an image sequence of the target video, wherein the image sequence is the video content information of the target video in image mode;

[0072] Text extraction processing is performed on the target video to obtain a text sequence of the target video, which is the video content information of the target video in text mode.

[0073] The image sequence can be composed of extracted video frames from the target video. In some embodiments, the target video can be uniformly extracted, for example, by dividing the target video into video segments of equal length, such as dividing the target video into at least one 10-second video segment, and extracting one frame from each video segment. In other embodiments, keyframes can be extracted from the target video. Specifically, keyframes can be identified by the differences between the various video frames, and then extracted.

[0074] The text sequence of the target video can include the video title, video description, and subtitles. Subtitles can be divided into soft subtitles and hard subtitles. Soft subtitles are separately saved subtitle files that can be directly extracted from the video data stream; hard subtitles are subtitles embedded in the video. For hard subtitles, text recognition can be performed on the image sequence obtained after processing the target video to extract the subtitles; alternatively, speech recognition can be performed on the audio segments of the processed target video to translate the speech into subtitles.

[0075] Optionally, in other embodiments, the at least one modality includes an audio modality; the step "obtaining video content information of the target video in at least one modality" may further include: performing audio data extraction processing on the target video to obtain an audio sequence of the target video, wherein the audio sequence is the video content information of the target video in the audio modality.

[0076] 102. Perform semantic extraction on the video content information of the target video in at least one modality to obtain the semantic feature information of the target video in at least one modality.

[0077] Optionally, in this embodiment, the step "semantically extracting the video content information of the target video in at least one modality to obtain the semantic feature information of the target video in at least one modality" may include:

[0078] Semantic extraction is performed on each image in the image sequence to obtain the image semantic feature information of each image in the image sequence;

[0079] Based on the image semantic feature information of each image before and after it, the image semantic feature information of each image is processed;

[0080] The semantic feature information of the processed images is fused to obtain the semantic feature information of the target video in the image modality;

[0081] Semantic extraction is performed on the text sequence to obtain the semantic feature information of the target video in the text modality.

[0082] Specifically, the step "semantically extracting each image in the image sequence to obtain the image semantic feature information of each image in the image sequence" may include:

[0083] By using a neural network, convolution and pooling operations are performed on each image in the image sequence to obtain the semantic feature information of each image in the image sequence.

[0084] The neural network can be a residual network (ResNet) or a densely connected convolutional network (DenseNet), etc. The neural network in this embodiment is not limited to the types listed above.

[0085] In the step of "processing the image semantic feature information of each image based on the image semantic feature information of the preceding and following images", the image semantic feature information of each image can be processed by Long Short-Term Memory (LSTM) network, Bi-directional Long Short-Term Memory (BiLSTM) network, etc. This embodiment does not limit this.

[0086] In some embodiments, the step "fusing the image semantic feature information of the processed images to obtain the semantic feature information of the target video in the image modality" may include: performing a weighted transformation on the image semantic feature information of the processed images to obtain the semantic feature information of the target video in the image modality.

[0087] The semantic extraction model can be used to extract semantic information from the text sequence, thereby obtaining the semantic feature information of the target video in the text modality. This semantic extraction model can be a Text Convolutional Neural Network (TextCNN), a self-attention model, a Bidirectional Encoder Representations from Transformers (BERT), or an Alite Bert model, etc. However, it should be understood that the semantic extraction model in this embodiment is not limited to the types listed above.

[0088] In some embodiments, at least one modality includes an audio modality; the audio sequence of the target video can be extracted, and then semantic extraction can be performed on each audio frame in the audio sequence to obtain the audio content feature information of each audio frame in the audio sequence, and the audio content feature information of each audio frame can be fused to obtain the semantic feature information of the target video in the audio modality.

[0089] 103. Perform feature enhancement processing on the semantic feature information of the target video in at least one modality to obtain the enhanced feature information of the target video.

[0090] The semantic feature information of the target video in at least one modality may include features that are strongly related to the classification of the target video, or features that are weakly related to the classification of the target video. The weakly related features may be features that are unrelated to the topic information of the target video, because the topic information of the target video is closely related to its classification and the topic information can represent the category to which the target video belongs.

[0091] Feature enhancement processing can involve strengthening features in the semantic feature information that are strongly correlated with the target video classification; that is, strengthening features in the semantic feature information that are strongly correlated with the topic information of the target video. Specifically, in some embodiments, feature enhancement processing can also suppress features in the semantic feature information that are weakly correlated with the topic information of the target video.

[0092] Optionally, in this embodiment, the step "perform feature enhancement processing on the semantic feature information of the target video in at least one modality to obtain the enhanced feature information of the target video" may include:

[0093] Logistic regression processing is performed on the semantic feature information of the target video in at least one modality to obtain the enhanced feature information of the target video, wherein the feature values ​​of the features in the enhanced feature information are within a preset numerical range corresponding to the logistic regression.

[0094] Logistic regression, also known as logistic regression analysis, is commonly used in data mining. It can identify features that are highly correlated with the classification of a target video from the semantic feature information of at least one modality. The highly correlated features are then strengthened, specifically by giving them higher weights and lower weights to features that are less correlated.

[0095] The preset numerical range can be set according to the actual situation, and this embodiment does not limit it. For example, the preset numerical range can be from zero to one, that is, the feature values ​​of the features in the semantic feature information of the target video in at least one modality are mapped to a numerical range of zero to one. This mapping process can be non-linear or linear.

[0096] Specifically, the step "performing logistic regression on the semantic feature information of the target video in at least one modality to obtain the enhanced feature information of the target video" may include:

[0097] By using a preset threshold function, logistic regression processing is performed on the semantic feature information of the target video in at least one modality to obtain the enhanced feature information of the target video.

[0098] The preset gate function can be a sigmoid function, a tanh function, or something similar; this embodiment does not impose any restrictions on this.

[0099] The sigmoid function, also known as the S-shaped growth curve, can be used as an activation function in neural networks or in logistic regression to map variables to a value range of zero to one. The tanh function, or hyperbolic tangent, can be used as an activation function in neural networks within the field of deep learning.

[0100] Optionally, in this embodiment, the step "perform feature enhancement processing on the semantic feature information of the target video in at least one modality to obtain the enhanced feature information of the target video" may include:

[0101] The semantic feature information of the target video in each modality is fused to obtain the fused semantic feature information of the target video;

[0102] The fused semantic feature information is subjected to feature enhancement processing to obtain the enhanced feature information of the target video.

[0103] This involves fusing semantic feature information from various modalities of the target video. Specifically, this can be achieved by splicing the semantic feature information from various modalities of the target video together, with no restriction on the splicing order.

[0104] For example, if the semantic feature information of the target video in at least one modality includes the semantic feature information of the target video in the image modality and the semantic feature information in the text modality, then the step "fusing the semantic feature information of the target video in each modality to obtain the fused semantic feature information of the target video" may include:

[0105] The semantic feature information of the target video in the image mode and the semantic feature information in the text mode are fused to obtain the fused semantic feature information of the target video.

[0106] In some embodiments, the semantic feature information of the target video in at least one modality may further include the semantic feature information of the target video in the audio modality. Then, the semantic feature information of the target video in the image modality, text modality, and audio modality can be fused to obtain the fused semantic feature information of the target video.

[0107] Specifically, the semantic feature information of the target video in image, text, and audio modalities can be concatenated to obtain the fused semantic feature information of the target video. It should be noted that the concatenation order is not limited in this embodiment. For example, the semantic feature information of the text modal can be concatenated after the semantic feature information of the image modal, and then the semantic feature information of the audio modal can be concatenated after the semantic feature information of the text modal; alternatively, the semantic feature information of the audio modal can be concatenated after the semantic feature information of the text modal, and then the semantic feature information of the image modal can be concatenated after the semantic feature information of the audio modal.

[0108] Optionally, in this embodiment, the fused semantic feature information includes at least one feature;

[0109] The step "perform feature enhancement processing on the fused semantic feature information to obtain the enhanced feature information of the target video" may include:

[0110] When the feature value of a feature in the fused semantic feature information is greater than a first preset value, the feature in the fused semantic feature information is enhanced.

[0111] When the feature value of a feature in the fused semantic feature information is less than a second preset value, the feature in the fused semantic feature information is compressed.

[0112] Based on the enhanced features and the compressed features, the enhanced feature information of the target video is obtained.

[0113] The first preset value and the second preset value can be set according to the actual situation. This embodiment does not limit this. The second preset value is not greater than the first preset value. In some embodiments, the second preset value can be equal to the first preset value.

[0114] In this process, enhancement processing can correspond to a larger weight, while compression processing corresponds to a smaller weight. When the feature value of a feature in the fused semantic feature information is greater than a first preset value, the feature in the fused semantic feature information is enhanced based on the weight corresponding to the enhancement processing to obtain the enhanced feature; when the feature value of a feature in the fused semantic feature information is less than a second preset value, the feature in the fused semantic feature information is compressed based on the weight corresponding to the compression processing to obtain the compressed feature.

[0115] Optionally, in some embodiments, when the feature value of a feature in the fused semantic feature information is greater than a first preset value, the enhancement magnitude of the feature in the fused semantic feature information can be determined based on the difference between the feature value and the first preset value. Then, the feature in the fused semantic feature information is enhanced according to the enhancement magnitude. Specifically, the larger the difference, the higher the enhancement magnitude; conversely, the smaller the difference, the lower the enhancement magnitude. Similarly, when the feature value of a feature in the fused semantic feature information is less than a second preset value, the compression magnitude of the feature in the fused semantic feature information can be determined based on the difference between the feature value and the second preset value. Then, the feature in the fused semantic feature information is compressed according to the compression magnitude. Specifically, the larger the difference, the higher the compression magnitude; conversely, the smaller the difference, the lower the compression magnitude.

[0116] 104. The enhanced feature information is fused with the semantic feature information of the target video in at least one modality to obtain the target semantic feature information of the target video.

[0117] In this process, the enhanced feature information can be fused with the semantic feature information of each modality of the target video, and then the target semantic feature information of the target video can be determined based on the fusion results.

[0118] Optionally, in this embodiment, the step of "fusing the enhanced feature information with the semantic feature information of the target video in at least one modality to obtain the target semantic feature information of the target video" may include:

[0119] The enhanced feature information is fused with the semantic feature information of the target video in the image modality to obtain the first feature information;

[0120] The enhanced feature information is fused with the semantic feature information of the target video in text mode to obtain the second feature information;

[0121] Based on the first feature information and the second feature information, the target semantic feature information of the target video is obtained.

[0122] There can be various ways to fuse feature information, and this embodiment does not limit this. For example, the fusion method can be dot product operation, etc.

[0123] Specifically, the enhanced feature information can be multiplied by the semantic feature information of the target video in the image mode to obtain the first feature information; the enhanced feature information can be multiplied by the semantic feature information of the target video in the text mode to obtain the second feature information; and the target semantic feature information of the target video can be obtained based on the first feature information and the second feature information.

[0124] The step "obtaining target semantic feature information of the target video based on the first feature information and the second feature information" may include:

[0125] The first feature information and the second feature information are fused to obtain the target semantic feature information of the target video.

[0126] The method of fusing the first feature information and the second feature information is not limited. Specifically, the fusion method can be splicing, and the splicing order is not limited. For example, splicing the second feature information after the first feature information can obtain the target semantic feature information of the target video. Optionally, the fusion method can also be weighted operation, etc.

[0127] Optionally, in this embodiment, the step "obtaining target semantic feature information of the target video based on the first feature information and the second feature information" may include:

[0128] Determine the weights corresponding to the first feature information and the second feature information;

[0129] Based on the weights, the first feature information and the second feature information are weighted and fused to obtain the target semantic feature information of the target video.

[0130] 105. Based on the target semantic feature information, classify the target video.

[0131] Specifically, based on target semantic feature information, the probability of the target video belonging to each preset category can be predicted, thereby determining the target category of the target video. In particular, the category with the highest probability can be taken as the target category of the target video.

[0132] In this embodiment, a classifier can be used to predict the probability that the target video belongs to each preset category. Specifically, the classifier can be a support vector machine, a fully connected deep neural network, etc. This embodiment does not limit the specific classifier.

[0133] In this embodiment, a video classification model can be used to extract semantic information from the video content of the target video in at least one modality to obtain semantic feature information of the target video in at least one modality; feature enhancement processing can be performed on the semantic feature information of the target video in at least one modality to obtain enhanced feature information of the target video; the enhanced feature information can be fused with the semantic feature information of the target video in at least one modality to obtain target semantic feature information of the target video; and the target video can be classified based on the target semantic feature information.

[0134] It should be noted that this video classification model can be trained using multiple labeled sample videos. The labeling information can be the target category corresponding to the sample video, or the expected probability of the sample video belonging to each preset category; this embodiment does not impose any limitations on this.

[0135] The video classification model can be trained by other devices and then provided to the video classification device, or the video classification device can train it itself.

[0136] If the video classification device is trained automatically, then before the step "using a video classification model to perform semantic extraction on the video content information of the target video in at least one modality to obtain the semantic feature information of the target video in at least one modality" it may also include:

[0137] Acquire training data, which includes sample videos and the expected probability of the sample videos belonging to each preset category;

[0138] Using a video classification model, semantic extraction is performed on the video content information of the sample video in at least one modality to obtain semantic feature information of the sample video in at least one modality; feature enhancement processing is performed on the semantic feature information of the sample video in at least one modality to obtain enhanced feature information of the sample video; the enhanced feature information is fused with the semantic feature information of the sample video in at least one modality to obtain the target semantic feature information of the sample video.

[0139] Based on the target semantic feature information, the predicted probability of the sample video belonging to each preset category is determined;

[0140] Based on the predicted probability and expected probability corresponding to the sample video, the parameters in the video classification model are adjusted.

[0141] The training process can adjust the parameters of the video classification model based on the backpropagation algorithm so that the predicted probability of a sample video belonging to a preset category approaches the expected probability. Specifically, the difference between the predicted probability and the expected probability can be made smaller than a preset value, which can be set according to the actual situation.

[0142] In this embodiment, after classifying the target video based on the target semantic feature information, the classified target video can be applied to scenarios such as video search and video recommendation. Since the video classification method of this application enhances the representation power of the target semantic feature information by strengthening the features related to the target video content, its classification results are more accurate, thus improving the accuracy of video search and video recommendation.

[0143] As can be seen from the above, the electronic device in this embodiment can acquire video content information of a target video in at least one modality; perform semantic extraction on the video content information of the target video in at least one modality to obtain semantic feature information of the target video in at least one modality; perform feature enhancement processing on the semantic feature information of the target video in at least one modality to obtain enhanced feature information of the target video; fuse the enhanced feature information with the semantic feature information of the target video in at least one modality to obtain target semantic feature information of the target video; and classify the target video based on the target semantic feature information. This embodiment of the application can enhance the representational power of target semantic feature information by strengthening features related to the target video content, which is beneficial to improving the accuracy of video classification.

[0144] Based on the method described in the preceding embodiments, the following will provide a more detailed explanation by taking the specific integration of the video classification device into a server as an example.

[0145] This application provides a video classification method, such as... Figure 2a As shown, the specific process of this video classification method can be as follows:

[0146] 201. The server obtains video content information of the target video in at least one modality.

[0147] The step "obtain video content information of the target video in at least one modality" may include:

[0148] Image extraction processing is performed on the target video to obtain an image sequence of the target video, wherein the image sequence is the video content information of the target video in image mode;

[0149] Text extraction processing is performed on the target video to obtain a text sequence of the target video, which is the video content information of the target video in text mode.

[0150] Optionally, in other embodiments, the at least one modality includes an audio modality; the step "obtaining video content information of the target video in at least one modality" may further include: performing audio data extraction processing on the target video to obtain an audio sequence of the target video, wherein the audio sequence is the video content information of the target video in the audio modality.

[0151] 202. The server performs semantic extraction on the video content information of the target video in at least one modality to obtain the semantic feature information of the target video in at least one modality.

[0152] Optionally, in this embodiment, the step "semantically extracting the video content information of the target video in at least one modality to obtain the semantic feature information of the target video in at least one modality" may include:

[0153] Semantic extraction is performed on each image in the image sequence to obtain the image semantic feature information of each image in the image sequence;

[0154] Based on the image semantic feature information of each image before and after it, the image semantic feature information of each image is processed;

[0155] The semantic feature information of the processed images is fused to obtain the semantic feature information of the target video in the image modality;

[0156] Semantic extraction is performed on the text sequence to obtain the semantic feature information of the target video in the text modality.

[0157] In some embodiments, at least one modality includes an audio modality; the audio sequence of the target video can be extracted, and then semantic extraction can be performed on each audio frame in the audio sequence to obtain the audio content feature information of each audio frame in the audio sequence, and the audio content feature information of each audio frame can be fused to obtain the semantic feature information of the target video in the audio modality.

[0158] 203. The server fuses the semantic feature information of the target video in each modality to obtain the fused semantic feature information of the target video.

[0159] This involves fusing semantic feature information from various modalities of the target video. Specifically, this can be achieved by splicing the semantic feature information from various modalities of the target video together, with no restriction on the splicing order.

[0160] For example, if the semantic feature information of the target video in at least one modality includes the semantic feature information of the target video in the image modality and the semantic feature information in the text modality, then the step "fusing the semantic feature information of the target video in each modality to obtain the fused semantic feature information of the target video" may include:

[0161] The semantic feature information of the target video in the image mode and the semantic feature information in the text mode are fused to obtain the fused semantic feature information of the target video.

[0162] 204. The server performs feature enhancement processing on the fused semantic feature information to obtain the enhanced feature information of the target video.

[0163] Feature enhancement processing can involve strengthening features in the fused semantic feature information that are strongly correlated with the target video classification; in other words, strengthening features in the fused semantic feature information that are strongly correlated with the topic information of the target video. Specifically, in some embodiments, feature enhancement processing can also suppress features in the fused semantic feature information that are weakly correlated with the topic information of the target video.

[0164] Optionally, in this embodiment, the step "the server performs feature enhancement processing on the fused semantic feature information to obtain the enhanced feature information of the target video" may include:

[0165] The server performs logistic regression processing on the fused semantic feature information to obtain enhanced feature information of the target video. The feature values ​​of the enhanced feature information are within the preset numerical range corresponding to the logistic regression.

[0166] Specifically, the enhanced feature information of the target video can be obtained by performing logistic regression processing on the fused semantic feature information using functions such as sigmoid or tanh.

[0167] The step "perform feature enhancement processing on the fused semantic feature information to obtain the enhanced feature information of the target video" may include:

[0168] When the feature value of a feature in the fused semantic feature information is greater than a first preset value, the feature in the fused semantic feature information is enhanced.

[0169] When the feature value of a feature in the fused semantic feature information is less than a second preset value, the feature in the fused semantic feature information is compressed.

[0170] Based on the enhanced features and the compressed features, the enhanced feature information of the target video is obtained.

[0171] 205. The server fuses the enhanced feature information with the semantic feature information of the target video in at least one modality to obtain the target semantic feature information of the target video.

[0172] Optionally, in this embodiment, the step of "fusing the enhanced feature information with the semantic feature information of the target video in at least one modality to obtain the target semantic feature information of the target video" may include:

[0173] The enhanced feature information is fused with the semantic feature information of the target video in the image modality to obtain the first feature information;

[0174] The enhanced feature information is fused with the semantic feature information of the target video in text mode to obtain the second feature information;

[0175] Based on the first feature information and the second feature information, the target semantic feature information of the target video is obtained.

[0176] 206. The server classifies the target video based on the target semantic feature information.

[0177] Specifically, based on target semantic feature information, the probability of the target video belonging to each preset category can be predicted, thereby determining the target category of the target video. In particular, the category with the highest probability can be taken as the target category of the target video.

[0178] In this embodiment, after classifying the target video based on the target semantic feature information, the classified target video can be applied to scenarios such as video search and video recommendation. Since the video classification method of this application enhances the representation power of the target semantic feature information by strengthening the features related to the target video content, its classification results are more accurate, thus improving the accuracy of video search and video recommendation.

[0179] In one specific embodiment, such as Figure 2b As shown, the target video to be classified is processed by extracting video frames to obtain an image sequence (i.e., video frames). Then, an image semantic construction model (such as ResNet network) is used to extract features from each image in the image sequence to obtain the semantic feature information x1 of the target video in the image modality. At the same time, a text semantic construction model (such as BERT) is used to extract the semantics of the text content (such as the video title) of the target video to obtain the semantic feature information x2 of the target video in the text modality. x1 and x2 are used as inputs to a threshold function to perform feature enhancement processing on x1 and x2 to obtain the enhanced feature information of the target video. The enhanced feature information is then fused with x1 and x2 respectively to obtain the first feature information and the second feature information. Finally, the first feature information and the second feature information are concatenated to obtain the target semantic feature information of the target video. Based on the target semantic feature information, the probability distribution of the target video belonging to each preset category is predicted, and the target video is classified.

[0180] The threshold function can be specifically expressed by the formula σ(w) TThe expression [x1, x2]+b) is used, where w represents the vector mapping matrix, b represents the bias constant, T represents the transpose, σ can be a sigmoid function, x1 represents the semantic feature information of the target video in the image modality, and x2 represents the semantic feature information of the target video in the text modality. Using [x1, x2] as the input of the threshold function, it means that the enhanced feature information is jointly constructed by the feature information of the image modality and the text modality, and then the features in x1 and x2 are suppressed or enhanced based on the enhanced feature information.

[0181] Specifically, the step "using x1 and x2 as inputs to a threshold function, performing feature enhancement processing on x1 and x2 through the threshold function to obtain enhanced feature information of the target video" can involve first fusing x1 and x2 (e.g., splicing) to obtain fused semantic feature information x, and then substituting x as input into the threshold function σ(w) T x+b) is used to obtain enhanced feature information.

[0182] Specifically, the step "fusing the enhanced feature information with x1 and x2 respectively to obtain the first feature information and the second feature information" can be achieved by processing x1 and x2 separately through a fully connected layer, then performing a dot product operation between the enhanced feature information and the processed x1 to obtain the first feature information, and performing a dot product operation between the enhanced feature information and the processed x2 to obtain the second feature information.

[0183] The video classification method of this application can perform logistic regression processing on the semantic feature information of the target video in at least one modality using a threshold function to obtain the enhanced feature information of the target video. If the threshold function is sigmoid, the feature values ​​of the features in the enhanced feature information can be in the range of [0, 1]. The closer the feature value of a feature is to 0, the more severely the feature is suppressed, weakening its influence on the classification result; conversely, the closer the feature value of a feature is to 1, the more the feature is enhanced, increasing its influence on the classification result. Therefore, the threshold function suppresses feature information that is irrelevant to the topic content of the target video and enhances feature information that is more relevant to the topic content of the target video, which is beneficial to enhancing the semantic interaction of semantic feature information in each modality of the target video.

[0184] As can be seen from the above, this embodiment can obtain video content information of a target video in at least one modality through a server; perform semantic extraction on the video content information of the target video in at least one modality to obtain semantic feature information of the target video in at least one modality; fuse the semantic feature information of the target video in each modality to obtain fused semantic feature information of the target video; perform feature enhancement processing on the fused semantic feature information to obtain enhanced feature information of the target video; fuse the enhanced feature information with the semantic feature information of the target video in at least one modality to obtain target semantic feature information of the target video; and classify the target video based on the target semantic feature information. This embodiment can enhance the representational power of target semantic feature information by strengthening features related to the target video content, which is beneficial to improving the accuracy of video classification.

[0185] To better implement the above methods, embodiments of this application also provide a video classification device, such as... Figure 3a As shown, the video classification device may include an acquisition unit 301, an extraction unit 302, an enhancement unit 303, a fusion unit 304, and a classification unit 305, as follows:

[0186] (1) Obtain unit 301;

[0187] The acquisition unit 301 is used to acquire video content information of the target video in at least one modality.

[0188] Optionally, in some embodiments of this application, the at least one modality includes an image modality and a text modality;

[0189] The acquisition unit 301 may include an image acquisition subunit 3011 and a text acquisition subunit 3012, see [link to documentation] Figure 3b ,as follows:

[0190] The image acquisition subunit 3011 is used to perform image extraction processing on the target video to obtain an image sequence of the target video, wherein the image sequence is the video content information of the target video in the image mode.

[0191] The text acquisition subunit 3012 is used to perform text extraction processing on the target video to obtain a text sequence of the target video, wherein the text sequence is the video content information of the target video in text mode.

[0192] (2) Extraction unit 302;

[0193] Extraction unit 302 is used to perform semantic extraction on the video content information of the target video in at least one modality to obtain the semantic feature information of the target video in at least one modality.

[0194] Optionally, in some embodiments of this application, the extraction unit 302 may include an image extraction subunit 3021, a processing subunit 3022, an image fusion subunit 3023, and a text extraction subunit 3024, see [link to relevant documentation]. Figure 3c ,as follows:

[0195] The image extraction subunit 3021 is used to perform semantic extraction on each image in the image sequence to obtain the image semantic feature information of each image in the image sequence;

[0196] The processing subunit 3022 is used to process the image semantic feature information of each image based on the image semantic feature information of the preceding and following images of each image;

[0197] The image fusion subunit 3023 is used to fuse the image semantic feature information of each processed image to obtain the semantic feature information of the target video in the image modality;

[0198] The text extraction subunit 3024 is used to perform semantic extraction on the text sequence to obtain the semantic feature information of the target video in the text modality.

[0199] (3) Reinforcement Unit 303;

[0200] The enhancement unit 303 is used to perform feature enhancement processing on the semantic feature information of the target video in at least one modality to obtain the enhanced feature information of the target video.

[0201] Optionally, in some embodiments of this application, the enhancement unit 303 may be specifically used to perform logistic regression processing on the semantic feature information of the target video in at least one modality to obtain the enhanced feature information of the target video, wherein the feature values ​​of the features in the enhanced feature information are within a preset numerical range corresponding to the logistic regression.

[0202] Optionally, in some embodiments of this application, the strengthening unit 303 may include a fusion subunit 3031 and a strengthening subunit 3032, see [link to relevant documentation]. Figure 3d ,as follows:

[0203] The fusion subunit 3031 is used to fuse the semantic feature information of the target video in various modalities to obtain the fused semantic feature information of the target video.

[0204] The enhancement subunit 3032 is used to perform feature enhancement processing on the fused semantic feature information to obtain the enhanced feature information of the target video.

[0205] Optionally, in some embodiments of this application, the fused semantic feature information includes at least one feature;

[0206] The enhancement subunit 3032 can be specifically used to enhance the features in the fused semantic feature information when the feature value of the feature in the fused semantic feature information is greater than a first preset value; and to compress the features in the fused semantic feature information when the feature value of the feature in the fused semantic feature information is less than a second preset value; and to obtain the enhanced feature information of the target video based on the enhanced features and the compressed features.

[0207] (4) Fusion unit 304;

[0208] The fusion unit 304 is used to fuse the enhanced feature information with the semantic feature information of the target video in at least one modality to obtain the target semantic feature information of the target video.

[0209] Optionally, in some embodiments of this application, the fusion unit 304 may include a first fusion subunit 3041, a second fusion subunit 3042, and a determination subunit 3043, see [link to relevant documentation]. Figure 3e ,as follows:

[0210] The first fusion subunit 3041 is used to fuse the enhanced feature information with the semantic feature information of the target video in the image modality to obtain the first feature information;

[0211] The second fusion subunit 3042 is used to fuse the enhanced feature information with the semantic feature information of the target video in the text modality to obtain the second feature information;

[0212] The determination subunit 3043 is used to obtain the target semantic feature information of the target video based on the first feature information and the second feature information.

[0213] Optionally, in some embodiments of this application, the determining subunit 3043 may be specifically used to determine the weights corresponding to the first feature information and the second feature information; based on the weights, the first feature information and the second feature information are weighted and fused to obtain the target semantic feature information of the target video.

[0214] (5) Classification unit 305;

[0215] The classification unit 305 is used to classify the target video based on the target semantic feature information.

[0216] As can be seen from the above, in this embodiment, the acquisition unit 301 acquires video content information of the target video in at least one modality; the extraction unit 302 performs semantic extraction on the video content information of the target video in at least one modality to obtain semantic feature information of the target video in at least one modality; the enhancement unit 303 performs feature enhancement processing on the semantic feature information of the target video in at least one modality to obtain enhanced feature information of the target video; the fusion unit 304 fuses the enhanced feature information with the semantic feature information of the target video in at least one modality to obtain target semantic feature information of the target video; and the classification unit 305 classifies the target video based on the target semantic feature information. This embodiment can enhance the representational power of target semantic feature information by strengthening features related to the target video content, which is beneficial to improving the accuracy of video classification.

[0217] This application also provides an electronic device, such as... Figure 4 The diagram shows a schematic representation of the structure of an electronic device according to an embodiment of this application. This electronic device can be a terminal or a server, specifically:

[0218] The electronic device may include components such as a processor 401 with one or more processing cores, a memory 402 with one or more computer-readable storage media, a power supply 403, and an input unit 404. Those skilled in the art will understand that... Figure 4 The electronic device structure shown does not constitute a limitation on the electronic device and may include more or fewer components than shown, or combine certain components, or have different component arrangements. Wherein:

[0219] The processor 401 is the control center of the electronic device. It connects various parts of the electronic device via various interfaces and lines. By running or executing software programs and / or modules stored in the memory 402, and by calling data stored in the memory 402, it performs various functions and processes data, thereby providing overall monitoring of the electronic device. Optionally, the processor 401 may include one or more processing cores; preferably, the processor 401 may integrate an application processor and a modem processor, wherein the application processor mainly handles the operating system, user interface, and applications, and the modem processor mainly handles wireless communication. It is understood that the modem processor may not be integrated into the processor 401.

[0220] The memory 402 can be used to store software programs and modules. The processor 401 executes various functional applications and data processing by running the software programs and modules stored in the memory 402. The memory 402 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, application programs required for at least one function (such as sound playback function, image playback function, etc.), etc.; the data storage area may store data created according to the use of the electronic device, etc. In addition, the memory 402 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device. Accordingly, the memory 402 may also include a memory controller to provide the processor 401 with access to the memory 402.

[0221] The electronic device also includes a power supply 403 that supplies power to the various components. Preferably, the power supply 403 can be logically connected to the processor 401 through a power management system, thereby enabling functions such as charging, discharging, and power consumption management through the power management system. The power supply 403 may also include one or more DC or AC power supplies, recharging systems, power fault detection circuits, power converters or inverters, power status indicators, and other arbitrary components.

[0222] The electronic device may also include an input unit 404, which can be used to receive input digital or character information, and generate keyboard, mouse, joystick, optical or trackball signal inputs related to user settings and function control.

[0223] Although not shown, the electronic device may also include a display unit, etc., which will not be described in detail here. Specifically, in this embodiment, the processor 401 in the electronic device loads the executable files corresponding to the processes of one or more applications into the memory 402 according to the following instructions, and the processor 401 runs the applications stored in the memory 402 to realize various functions, as follows:

[0224] The process involves: acquiring video content information of a target video in at least one modality; performing semantic extraction on the video content information of the target video in at least one modality to obtain semantic feature information of the target video in at least one modality; performing feature enhancement processing on the semantic feature information of the target video in at least one modality to obtain enhanced feature information of the target video; fusing the enhanced feature information with the semantic feature information of the target video in at least one modality to obtain target semantic feature information of the target video; and classifying the target video based on the target semantic feature information.

[0225] For details on the implementation of each of the above operations, please refer to the previous examples, which will not be repeated here.

[0226] As can be seen from the above, this embodiment can acquire video content information of a target video in at least one modality; perform semantic extraction on the video content information of the target video in at least one modality to obtain semantic feature information of the target video in at least one modality; perform feature enhancement processing on the semantic feature information of the target video in at least one modality to obtain enhanced feature information of the target video; fuse the enhanced feature information with the semantic feature information of the target video in at least one modality to obtain target semantic feature information of the target video; and classify the target video based on the target semantic feature information. This embodiment can enhance the representational power of target semantic feature information by strengthening features related to the target video content, which is beneficial to improving the accuracy of video classification.

[0227] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be performed by instructions, or by instructions controlling related hardware. These instructions can be stored in a computer-readable storage medium and loaded and executed by a processor.

[0228] Therefore, embodiments of this application provide a storage medium storing a plurality of instructions that can be loaded by a processor to execute steps in any of the video classification methods provided in embodiments of this application. For example, the instructions can execute the following steps:

[0229] The process involves: acquiring video content information of a target video in at least one modality; performing semantic extraction on the video content information of the target video in at least one modality to obtain semantic feature information of the target video in at least one modality; performing feature enhancement processing on the semantic feature information of the target video in at least one modality to obtain enhanced feature information of the target video; fusing the enhanced feature information with the semantic feature information of the target video in at least one modality to obtain target semantic feature information of the target video; and classifying the target video based on the target semantic feature information.

[0230] For details on the implementation of each of the above operations, please refer to the previous examples, which will not be repeated here.

[0231] The storage medium may include: read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0232] Since the instructions stored in the storage medium can execute the steps of any of the video classification methods provided in the embodiments of this application, the beneficial effects that any of the video classification methods provided in the embodiments of this application can achieve can be realized, as detailed in the preceding embodiments, and will not be repeated here.

[0233] According to one aspect of this application, a computer program product or computer program is provided, comprising computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the methods provided in the various alternative implementations of the video classification described above.

[0234] The foregoing has provided a detailed description of a video classification method, apparatus, electronic device, and storage medium provided in the embodiments of this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the embodiments above are only for the purpose of helping to understand the method and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A video classification method, characterized in that, include: Acquire video content information of the target video in multiple modalities, including image, text, and audio modalities; Semantic extraction is performed on the video content information of the target video in multiple modalities to obtain the semantic feature information of the target video in multiple modalities. The semantic feature information of the target video in multiple modalities includes features that are strongly correlated with the classification of the target video and features that are weakly correlated with the classification of the target video. The weakly correlated features are specifically features that are unrelated to the topic information of the target video. The semantic feature information of the target video in multiple modalities is subjected to feature enhancement processing to obtain the enhanced feature information of the target video. The feature enhancement processing is to enhance the features in the semantic feature information that are strongly related to the theme information of the target video or to suppress the features in the semantic feature information that are weakly related to the theme information of the target video. The feature values ​​of the features in the enhanced feature information are within the preset numerical range corresponding to logistic regression. The enhanced feature information is fused with the semantic feature information of the target video in multiple modalities to obtain the target semantic feature information of the target video; Based on the target semantic feature information, the target video is classified; The step of performing feature enhancement processing on the semantic feature information of the target video in multiple modalities to obtain the enhanced feature information of the target video includes: By using a preset threshold function, logistic regression processing is performed on the semantic feature information of the target video in multiple modalities to obtain the enhanced feature information of the target video. or, The semantic feature information of the target video in image, text, and audio modalities is fused to obtain the fused semantic feature information of the target video. The fusion method includes splicing processing and weighted fusion. When the feature value of a feature in the fused semantic feature information is greater than a first preset value, the enhancement magnitude of the feature in the fused semantic feature information is determined based on the difference between the feature value of the feature in the fused semantic feature information and the first preset value. Then, the feature in the fused semantic feature information is enhanced according to the enhancement magnitude. When the feature value of a feature in the fused semantic feature information is less than a second preset value, the compression magnitude of the feature in the fused semantic feature information is determined based on the difference between the feature value of the feature in the fused semantic feature information and the second preset value. Then, the feature in the fused semantic feature information is compressed according to the compression magnitude. Based on the enhanced feature and the compressed feature, the enhanced feature information of the target video is obtained.

2. The method according to claim 1, characterized in that, If the preset threshold function is sigmoid, the feature value of the feature in the enhanced feature information can be in the range of [0, 1]. The closer the feature value of the feature in the enhanced feature information is to 0, the more severely the feature is suppressed; the closer the feature value of the feature in the enhanced feature information is to 1, the more the feature is enhanced. The preset threshold function suppresses feature information that is not related to the theme content of the target video and enhances feature information that is more related to the theme content of the target video.

3. The method according to claim 1, characterized in that, The acquisition of video content information of the target video in multiple modalities includes: Image extraction processing is performed on the target video to obtain an image sequence of the target video, wherein the image sequence is the video content information of the target video in image mode; Text extraction processing is performed on the target video to obtain a text sequence of the target video, which is the video content information of the target video in text mode.

4. The method according to claim 3, characterized in that, The step of semantically extracting the video content information of the target video in multiple modalities to obtain the semantic feature information of the target video in multiple modalities includes: Semantic extraction is performed on each image in the image sequence to obtain the image semantic feature information of each image in the image sequence; Based on the image semantic feature information of each image before and after it, the image semantic feature information of each image is processed; The semantic feature information of the processed images is fused to obtain the semantic feature information of the target video in the image modality; Semantic extraction is performed on the text sequence to obtain the semantic feature information of the target video in the text modality.

5. The method according to claim 4, characterized in that, The step of fusing the enhanced feature information with the semantic feature information of the target video in multiple modalities to obtain the target semantic feature information of the target video includes: The enhanced feature information is fused with the semantic feature information of the target video in the image modality to obtain the first feature information; The enhanced feature information is fused with the semantic feature information of the target video in text mode to obtain the second feature information; Based on the first feature information and the second feature information, the target semantic feature information of the target video is obtained.

6. The method according to claim 5, characterized in that, The step of obtaining the target semantic feature information of the target video based on the first feature information and the second feature information includes: Determine the weights corresponding to the first feature information and the second feature information; Based on the weights, the first feature information and the second feature information are weighted and fused to obtain the target semantic feature information of the target video.

7. A video classification device, characterized in that, include: The acquisition unit is used to acquire video content information of the target video in multiple modalities, including image, text, and audio modalities; The extraction unit is used to perform semantic extraction on the video content information of the target video in multiple modalities to obtain the semantic feature information of the target video in multiple modalities. The semantic feature information of the target video in multiple modalities includes features that are strongly correlated with the classification of the target video and features that are weakly correlated with the classification of the target video. The weakly correlated features are specifically features that are unrelated to the topic information of the target video. The enhancement unit is used to perform feature enhancement processing on the semantic feature information of the target video in multiple modalities to obtain the enhanced feature information of the target video. The feature enhancement processing is to enhance the features in the semantic feature information that are strongly related to the theme information of the target video or to suppress the features in the semantic feature information that are weakly related to the theme information of the target video. The feature values ​​of the features in the enhanced feature information are within a preset numerical range corresponding to logistic regression. The fusion unit is used to fuse the enhanced feature information with the semantic feature information of the target video in multiple modalities to obtain the target semantic feature information of the target video; A classification unit is used to classify the target video based on the target semantic feature information; The enhancement unit is specifically used to perform logistic regression processing on the semantic feature information of the target video in multiple modalities through a preset threshold function to obtain the enhanced feature information of the target video. or, The semantic feature information of the target video in image, text, and audio modalities is fused to obtain the fused semantic feature information of the target video. The fusion method includes splicing processing and weighted fusion. When the feature value of a feature in the fused semantic feature information is greater than a first preset value, the enhancement magnitude of the feature in the fused semantic feature information is determined based on the difference between the feature value of the feature in the fused semantic feature information and the first preset value. Then, the feature in the fused semantic feature information is enhanced according to the enhancement magnitude. When the feature value of a feature in the fused semantic feature information is less than a second preset value, the compression magnitude of the feature in the fused semantic feature information is determined based on the difference between the feature value of the feature in the fused semantic feature information and the second preset value. Then, the feature in the fused semantic feature information is compressed according to the compression magnitude. Based on the enhanced feature and the compressed feature, the enhanced feature information of the target video is obtained.

8. An electronic device, characterized in that, It includes a memory and a processor; the memory stores an application program, and the processor runs the application program within the memory to perform the operations in the video classification method according to any one of claims 1 to 6.

9. A storage medium, characterized in that, The storage medium stores a plurality of instructions adapted for loading by a processor to execute the steps of the video classification method according to any one of claims 1 to 6.

Images