A video classification method and device, electronic equipment and storage medium

Abstract

The application relates to the computer technical field, and particularly relates to a video classification method and device, electronic equipment and storage medium, to improve the accuracy of video classification. The method comprises the following steps: obtaining visual features of each target video frame in a to-be-classified video and semantic features corresponding to target description information; determining frame selection weights of the target video frames based on the semantic features and the visual features; determining key video frames in the to-be-classified video based on the frame selection weights, and determining visual enhanced features of the to-be-classified video based on the visual features of the key video frames; and determining category information of the to-be-classified video by using multimodal interactive features obtained by performing feature interaction based on the visual enhanced features and the semantic features. The application can effectively improve the accuracy of video classification by screening key video frames through frame selection weights, generating visual enhanced features based on the frame selection weights and the key video frames, and performing multimodal interaction on the visual enhanced features and the semantic features.

Description

Technical Field

[0001] This application relates to the field of computer technology, and more particularly to a video classification method, apparatus, electronic device, and storage medium. Background Technology

[0002] Video classification involves analyzing and understanding video information to categorize videos into predefined classes. It is one of the most fundamental and important tasks in computer vision. It plays a crucial role in numerous real-world applications, including video-based search, video recommendation, and the generation of video tags and titles.

[0003] Generally speaking, multimodal videos contain information from multiple modalities, such as video frame sequences, text descriptions, and audio. In order to combine information from each modality to improve the classification effect of the video, the relevant technologies mainly adopt the approach of "independent modeling of each modality + feature concatenation". For example, multiple single-modal video classification models are used to extract features from each single modality separately, and then the features of the single modality are concatenated and a linear neural network is used to learn and fuse the representation.

[0004] In the methods described above, independent modeling of visual modalities during video feature extraction easily introduces a large amount of redundant information and visual noise. Therefore, improving the accuracy of multimodal video classification is an urgent problem to be solved. Summary of the Invention

[0005] This application provides a video classification method, apparatus, electronic device, and storage medium to improve the accuracy of multimodal video classification.

[0006] This application provides a video classification method, including:

[0007] Obtain the visual features of each target video frame in the video to be classified, as well as the semantic features corresponding to the target description information of the video to be classified;

[0008] Based on the semantic features and visual features, the frame selection weights of each target video frame are determined.

[0009] Based on the selection weights of each frame, at least one key video frame is determined in the video to be classified, and based on the visual features of each of the at least one key video frame, the visual enhancement features of the video to be classified are determined.

[0010] Based on the visual enhancement features and the semantic features, feature interaction is performed to obtain corresponding multimodal interaction features, and based on the multimodal interaction features, the category information of the video to be classified is determined.

[0011] This application provides a video classification device, comprising:

[0012] The feature extraction unit is used to obtain the visual features of each target video frame in the video to be classified, as well as the semantic features corresponding to the target description information of the video to be classified.

[0013] The weight determination unit is used to determine the frame selection weight of each target video frame based on the semantic features and each visual feature;

[0014] A visual enhancement unit is used to select weights for each frame, determine at least one key video frame in the video to be classified, and determine the visual enhancement features of the video to be classified based on the visual features of each of the at least one key video frame.

[0015] The classification unit is used to perform feature interaction based on the visual enhancement features and the semantic features to obtain corresponding multimodal interaction features, and to determine the category information of the video to be classified based on the multimodal interaction features.

[0016] Optionally, the classification unit is specifically used for:

[0017] Visual transfer processing and semantic transfer processing are performed on the multimodal interaction features respectively, and the visual representation features and semantic representation features of the video to be classified are extracted from the multimodal interaction features.

[0018] Based on the visual representation features and the semantic representation features, category prediction is performed to obtain the category information of the video to be classified.

[0019] Optionally, the classification unit is specifically used for:

[0020] Based on the visual representation features, the class of the video to be classified is predicted to obtain a first probability value that the video to be classified belongs to each preset class; and based on the semantic representation features, the class of the video to be classified is predicted to obtain a second probability value that the video to be classified belongs to each preset class.

[0021] Based on each first probability value and each second probability value, the category information to which the video to be classified belongs is determined from each preset category.

[0022] Optionally, the classification unit is specifically used for:

[0023] For each preset category, perform the following operations: take the smaller of the first probability value and the second probability value corresponding to a preset category, or the average of the first probability value and the second probability value, as the predicted probability value of the video to be classified belonging to the preset category.

[0024] The preset category corresponding to the predicted probability value that is ranked in a specified order among the predicted probability values ​​is used as the category information to which the video to be classified belongs.

[0025] Optionally, the weight determination unit is specifically used for:

[0026] Based on the semantic features and the visual features, feature fusion is performed to obtain contextual features that match the semantic features;

[0027] Based on the attention mechanism, the semantic weights of each target video frame are obtained from the context features;

[0028] Perform any of the following operations:

[0029] The semantic weights of each target video frame are used as the corresponding frame selection weights.

[0030] Based on the category-related features and visual features of the video to be classified, the visual weights of each target video frame are determined; the sum of the semantic weights and visual weights of the same target video frame is used as the corresponding frame selection weights.

[0031] Optionally, the weight determination unit is specifically used for:

[0032] The semantic features are weighted, averaged, and transposed to obtain the auxiliary semantic features of the video to be classified.

[0033] The visual features and the auxiliary semantic features are fused to obtain the contextual features.

[0034] Optionally, the category-related features include: category-related features corresponding to each target video frame, determined by processing the visual features of each target video frame through a self-learning network;

[0035] The weight determination unit is specifically used for:

[0036] The category-related features corresponding to each target video frame are weighted and averaged to obtain reference features;

[0037] After concatenating the reference features with each visual feature, the concatenation result is linearly mapped to obtain the visual weights of each target video frame.

[0038] Optionally, the visual enhancement unit is specifically used for:

[0039] Based on the weights selected for each frame, video frames whose weight values ​​are within a specified order range in each target video frame are taken as key video frames.

[0040] Feature fusion is performed based on the visual features and corresponding weight values ​​of each key video frame to obtain the visual enhancement features of the video to be classified.

[0041] Optionally, the semantic features and visual features are obtained by inputting each target video frame and the target description information into a trained target video classification model, and then using the feature extraction network in the target video classification model.

[0042] The visual enhancement features are obtained by determining the frame selection weights of each target video frame based on the semantic features and the visual features through the key video frame selection network in the target video classification model.

[0043] The multimodal interaction features are obtained by performing feature interaction between the visual enhancement features and the semantic features based on the multimodal interaction network in the target video classification model.

[0044] Optionally, the device further includes:

[0045] The model training unit is used to obtain the target video classification model in the following ways:

[0046] Based on the training samples in the training sample set, the video classification model to be trained is iteratively trained to obtain the target video classification model; each training sample includes: each first sample video frame of the sample video, the first sample description information of the sample video, and the true class label; wherein, each iterative training cycle executes the following steps:

[0047] The first sample video frames and the first sample description information are input into the video classification model to obtain the first sample visual features of each first sample video frame and the first sample semantic features of the first sample description information.

[0048] The corresponding sample category information is determined by using the multimodal interaction features of the samples obtained based on the semantic features of the first sample and the visual features of each first sample.

[0049] The video classification model is adjusted by using a target loss function constructed based on the sample category information, the real category label, and the sample multimodal interaction features.

[0050] Optionally, the model training unit is further configured to obtain the multimodal interaction features of the samples in the following manner:

[0051] Based on the semantic features of the first sample and the visual features of each first sample, the sample frame selection weight of each first sample video frame is determined.

[0052] Based on the selection weights of each sample frame, at least one sample key video frame is determined in the sample video, and based on the first sample visual features of the at least one sample key video frame, the sample visual enhancement features of the sample video are determined.

[0053] Based on the visual enhancement features of the second sample and the semantic features of the first sample, feature interaction is performed to obtain the multimodal interaction features of the sample.

[0054] Optionally, the model training unit is specifically used for:

[0055] Based on the semantic features of the first sample and the visual features of each first sample, feature fusion is performed to obtain sample context features that match the semantic features of the first sample;

[0056] Based on the attention mechanism, the semantic weights of each first sample video frame are obtained from the sample context features;

[0057] Obtain the second sample visual features of each second sample video frame in the reference video, and determine the first sample visual weight of each first sample video frame based on each second sample visual feature and each first sample visual feature; the reference video and the sample video have the same true category label;

[0058] The sum of the semantic weight and visual weight of the same first sample video frame is used as the corresponding sample frame selection weight.

[0059] Optionally, the model training unit is specifically used for:

[0060] The visual features of each second sample are weighted and averaged to obtain the sample reference features.

[0061] After concatenating the sample reference features with the visual features of each first sample, the concatenation result is linearly mapped to obtain the first sample visual weight of each first sample video frame.

[0062] Optionally, the model training unit is further configured to:

[0063] Before inputting each first sample video frame and the first sample description information into the video classification model, a reference video matching the sample video is obtained, wherein the reference video has the same true category label as the sample video;

[0064] Each second sample video frame of the reference video and its corresponding second sample description information are input into the video classification model to obtain the second sample visual features of each second sample video frame and the second sample semantic features of the second sample description information.

[0065] Optionally, the model training unit is specifically used to construct the target loss function in the following ways:

[0066] Based on the difference between the sample category information and the true category label, a classification prediction loss function is constructed;

[0067] Based on the difference between the target dimension in the visual representation features of the sample and the corresponding dimension in the semantic representation features of the sample, a heterogeneous feature alignment loss function is determined; the visual representation features and the semantic representation features are extracted from the multimodal interaction features of the sample through feature transfer;

[0068] The target loss function is constructed based on the classification prediction loss function and the heterogeneous feature alignment loss function.

[0069] Optionally, the model training unit is specifically used for:

[0070] A constrained loss function is determined based on the differences between the sample video and the reference video that matches the sample video; the reference video has the same true category label as the sample video.

[0071] The target loss function is obtained by weighting the classification prediction loss function, the heterogeneous feature alignment loss function, and the constraint loss function.

[0072] Optionally, the sample category information includes first sample category information obtained based on the visual representation features of the sample, and second sample category information obtained based on the semantic representation features of the sample;

[0073] The model training unit is specifically used for:

[0074] Based on the difference between the first sample category information and the true category label, a visual classification loss function is determined; and based on the difference between the second sample category information and the true category label, a semantic classification loss function is determined.

[0075] Based on the visual classification loss function and the semantic classification loss function, the classification prediction loss function is constructed.

[0076] Optionally, the constraint loss function includes at least one of a semantic constraint loss function and a visual constraint loss function; the model training unit is further configured to determine the constraint loss function in the following ways:

[0077] A semantic constraint loss function is determined based on the difference between the semantic features of the first sample and the semantic features of the second sample; the semantic features of the second sample are determined based on the second sample description information of the reference video.

[0078] Based on the difference between the first sample visual features corresponding to the first sample constrained video frame in the sample video and the second sample visual features corresponding to the second sample constrained video frame in the reference video, a visual constraint loss function is determined.

[0079] Optionally, the model training unit is further configured to determine the first sample constrained video frame and the second sample constrained video frame in the following manner:

[0080] Based on the visual weights of each first sample, at least one first sample constrained video frame is determined in the sample video.

[0081] The visual features of each first sample are weighted and averaged. Based on the weighted average result and the visual features of each second sample, the second sample visual weight of each second sample video frame is determined. Based on the second sample visual weight, at least one second sample constrained video frame is determined in the reference video. An electronic device provided in this application includes a processor and a memory, wherein the memory stores a computer program that, when executed by the processor, causes the processor to perform the steps of any of the above-described video classification methods.

[0082] This application provides a computer-readable storage medium including a computer program. When the computer program is run on an electronic device, the computer program is used to cause the electronic device to perform the steps of any of the above-described video classification methods.

[0083] This application provides a computer program product, which includes a computer program stored in a computer-readable storage medium. When the processor of an electronic device reads the computer program from the computer-readable storage medium, the processor executes the computer program, causing the electronic device to perform the steps of any of the above-described video classification methods.

[0084] The beneficial effects of this application are as follows:

[0085] This application provides a video classification method, apparatus, electronic device, and storage medium. By acquiring visual and semantic features, this application analyzes the frame selection weights of each target video frame in the video to be classified. These weights are used to filter video frames, identifying key video frames—those that significantly influence the classification results. This improves the selection of key video frames and the learning of important features, effectively reducing redundant information and visual noise. Through the key video frames and their corresponding frame selection weights, visual features are enhanced to obtain visual enhancement features of the video to be classified. Furthermore, the enhanced visual features and semantic features are interacted to achieve information exchange and complementarity between different modalities. This fully utilizes multimodal video information, enabling the extraction and unified representation of key multimodal information, thereby improving video classification accuracy.

[0086] Other features and advantages of this application will be set forth in the following description and will be apparent in part from the description or may be learned by practicing the application. The objectives and other advantages of this application may be realized and obtained by means of the structures particularly pointed out in the written description, claims, and drawings. Attached Figure Description

[0087] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments of this application and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0088] Figure 1 This is an optional schematic diagram of an application scenario in an embodiment of this application;

[0089] Figure 2 This is an overall flowchart of a video product positioning method according to an embodiment of this application;

[0090] Figure 3 This is a flowchart illustrating a video classification method in an embodiment of this application;

[0091] Figure 4 This is a schematic diagram of a video frame and description information in an embodiment of this application;

[0092] Figure 5 This is a logical diagram illustrating how a target video classification model determines category information in an embodiment of this application.

[0093] Figure 6 This is a schematic diagram of a method for determining frame selection weight in an embodiment of this application;

[0094] Figure 7 This is a flowchart illustrating a model training method in an embodiment of this application.

[0095] Figure 8 This is a schematic diagram of a video classification model in an embodiment of this application;

[0096] Figure 9 This is a schematic diagram of a keyframe selection mechanism in an embodiment of this application;

[0097] Figure 10 This is a flowchart illustrating another model training method in an embodiment of this application;

[0098] Figure 11 This is a schematic diagram of yet another video classification model in the embodiments of this application;

[0099] Figure 12 This is a flowchart illustrating a method for classifying videos to be classified, as described in an embodiment of this application.

[0100] Figure 13A This is a schematic diagram of the first type of visual and textual information representation in the embodiments of this application;

[0101] Figure 13B This is a schematic diagram illustrating the second type of visual and textual information representation in an embodiment of this application;

[0102] Figure 13C This is a schematic diagram of the third type of visual and textual information representation in the embodiments of this application;

[0103] Figure 14 This is a schematic diagram of the composition structure of a video classification device according to an embodiment of this application;

[0104] Figure 15 This is a schematic diagram of the composition structure of an electronic device according to an embodiment of this application. Detailed Implementation

[0105] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings of the embodiments of this application. Obviously, the described embodiments are only some embodiments of the technical solutions of this application, and not all embodiments. Based on the embodiments recorded in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the technical solutions of this application.

[0106] The following describes some of the concepts involved in the embodiments of this application.

[0107] Multimodal video and heterogeneous features: Video data containing various types of information such as image sequences, audio, and descriptive text. Heterogeneous features refer to features that differ in value distribution and other aspects due to being extracted from different modalities. In the embodiments of this application, the video may contain multimodal video data, and the features corresponding to each modality will have certain differences in value distribution and other aspects. For example, visual features and semantic features are taken from different modalities and belong to heterogeneous features.

[0108] Attention mechanism: It is a method that enables neural networks to selectively focus on a subset of the input. The attention mechanism mainly has two aspects: deciding which part of the input needs to be focused on; and allocating limited information processing resources to the important parts.

[0109] Positive sample: refers to a sample that has consistent attributes with the target sample. In this embodiment, an original sample and its corresponding positive sample have consistency in video classification results, that is, they belong to the same video category, and the corresponding real category labels are also the same. Video classification refers to categorizing samples according to type, level, or nature. In this embodiment, the real category label is used to characterize the video category to which the sample belongs. For example, the preset categories corresponding to this application are: video, game, and education. The real category label can be represented by three binary digits, such as 010, which corresponds to the preset category of game, or 001, which corresponds to the preset category of education, and so on.

[0110] Frame selection weight: A weight value used to characterize the contribution of a video frame to the video classification result. A larger frame selection weight indicates a greater influence on the video classification result, and vice versa. In this embodiment, the frame selection weight can be set randomly, determined based on an attention mechanism, or determined through hierarchical causal reasoning, etc.

[0111] Visual enhancement features: The visual enhancement features in this application refer to the features obtained by visually enhancing the visual features of key video frames based on the visual features extracted from each video frame in the video. This process involves the removal of redundant information and visual noise. Therefore, the visual enhancement features obtained based on the enhanced representation can effectively improve the accuracy of video classification.

[0112] The embodiments of this application relate to artificial intelligence (AI), natural language processing (NLP), and machine learning (ML) technologies, and are designed based on computer vision technology and machine learning in artificial intelligence.

[0113] Artificial intelligence (AI) technology mainly includes computer vision, natural language processing, machine learning / deep learning, autonomous driving, and intelligent transportation. With the research and advancement of AI technology, it is being researched and applied in multiple fields, such as smart homes, intelligent customer service, virtual assistants, smart speakers, intelligent marketing, autonomous driving, robotics, and smart healthcare. It is believed that with further technological development, AI will be applied in even more fields and play an increasingly important role. The video classification method in this application embodiment can be applied to video search, video recommendation, and other fields. Combining AI with video classification in these fields can achieve accurate multimodal intelligent video classification.

[0114] Furthermore, in the process of video classification, it is necessary to obtain each target video frame in the video to be classified, as well as the target description information corresponding to the video to be classified, and input these data into the pre-trained target video classification model, and then predict the category of the video to be classified based on the target video classification model.

[0115] Furthermore, the target video classification model in this application embodiment is trained using machine learning or deep learning techniques. After training the target video classification model using the aforementioned techniques, it can be applied to achieve intelligent classification of multimodal videos, thereby effectively improving the accuracy of video classification.

[0116] The design concept of the embodiments of this application is briefly introduced below:

[0117] With the rapid development of internet technology, all kinds of videos have emerged, and videos have become ubiquitous on the internet. More and more people are finding it difficult to quickly find videos they like from the vast sea of ​​content. Therefore, it is crucial to narrow down the range of videos available to users so that they can quickly find their preferred content.

[0118] One method for narrowing down object selection is to search for videos within specific classification regions. Traditional video classification algorithms generally employ behavior recognition in simple scenes. With the advent of the short video era, a large number of videos are uploaded to video websites every minute, resulting in rich content and complex scenes. Related video analysis methods typically extract features from consecutive frames and then aggregate these features over time to represent the video. Due to the diversity of video content and the redundancy of video frames in complex scenes, the prediction results are significantly affected. Therefore, a robust key video frame selection method is urgently needed for classifying multimodal videos.

[0119] Video classification can be briefly described as pre-extracting video and visual features at the video and frame levels, with the aim of assigning them the correct video-level labels. However, this task is more difficult than image classification. Videos consist of a series of video frames and audio data, resulting in a large number of video features, which increases the difficulty of training the model as the number of features increases. Secondly, data redundancy is widespread in video data; there are small differences between consecutive frames, and significant scene differences between the same video, but very few labels are attached to a single video.

[0120] In related technologies, common multimodal video classification methods involve multimodal fusion and multi-frame image processing. However, these methods lack explicit constraints on the selection of key video frames and the extraction and collaborative learning of semantic information from different modalities, introducing redundant information. This can cause deviations in model fitting and feature fusion, resulting in weak scene generalization ability. Furthermore, heterogeneous modal features typically have different value ranges and distributions, so directly using neural networks for feature fusion can lead to biases in weighted feature fusion.

[0121] Therefore, how to use methods to select key video frames and solve the bias caused by heterogeneous modal feature fusion is an urgent problem to be solved.

[0122] In view of this, embodiments of this application propose a video classification method, apparatus, electronic device, and storage medium. This application analyzes the frame selection weights of each target video frame in the video to be classified using acquired visual and semantic features. These weights are then used to filter video frames, identifying key video frames—those that significantly influence the video classification results. This improves the selection of key video frames and the learning of important features, effectively reducing redundant information and visual noise. Visual features are enhanced using key video frames and their corresponding frame selection weights to obtain enhanced visual features of the video to be classified. Furthermore, the enhanced visual features and semantic features are interacted to achieve information exchange and complementarity between different modalities, fully utilizing multimodal video information to extract and uniformly represent key multimodal information, thereby improving video classification accuracy.

[0123] Furthermore, the video classification method in this application embodiment can be implemented based on a target video classification model obtained through machine learning. During the model training process, the model parameters are adjusted by combining a loss function that aligns some heterogeneous features. Based on this, the target video classification model can effectively solve the bias caused by the fusion of heterogeneous modal features and further improve the accuracy of video classification.

[0124] The preferred embodiments of this application are described below with reference to the accompanying drawings. It should be understood that the preferred embodiments described herein are for illustration and explanation only and are not intended to limit this application. Furthermore, the embodiments and features in the embodiments of this application can be combined with each other without conflict.

[0125] like Figure 1 The diagram shown is an application scenario illustration of an embodiment of this application. The application scenario diagram includes two terminal devices 110 and one server 120.

[0126] In this embodiment, the terminal device 110 includes, but is not limited to, mobile phones, tablets, laptops, desktop computers, e-book readers, smart voice interaction devices, smart home appliances, and in-vehicle terminals. The terminal device may have a video classification-related client installed. This client can be software (such as a browser, video software, etc.), or a webpage, mini-program, etc. The server 120 is the backend server corresponding to the software, webpage, mini-program, etc., or a server specifically used for video classification; this application does not impose specific limitations. The server 120 can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms.

[0127] It should be noted that the video classification methods in the various embodiments of this application can be executed by an electronic device, which can be a terminal device 110 or a server 120. That is, the method can be executed by the terminal device 110 or the server 120 alone, or by both the terminal device 110 and the server 120. For example, when the terminal device 110 and the server 120 execute the method together, the terminal device 110 sends each target video frame in the video to be classified, along with the corresponding target description information, to the server 120. The server 120 then obtains the corresponding visual features and semantic features, and uses the semantic features and visual features to determine the frame selection weights of each target video frame. Furthermore, based on the frame selection weights, the server determines the key video frames in the video to be classified, removes redundant information, and generates visual enhancement features based on the visual features of the key video frames and the corresponding frame selection weights. Afterward, the server 120 performs classification prediction on the video to be classified based on the multimodal interaction features obtained by feature interaction between the visual enhancement features and the semantic features, thereby obtaining the corresponding category information. Finally, the server 120 can feed back the obtained category information to the terminal device 110, which can then display the videos to be categorized or recommend videos to the target audience based on the category information.

[0128] In one alternative implementation, the terminal device 110 and the server 120 can communicate via a communication network.

[0129] In one alternative implementation, the communication network is a wired network or a wireless network.

[0130] It should be noted that, Figure 1 The examples shown are merely illustrative; in reality, the number of terminal devices and servers is unlimited and is not specifically limited in the embodiments of this application.

[0131] In this embodiment of the application, when there are multiple servers, the multiple servers can form a blockchain, and the servers are nodes on the blockchain; as disclosed in the video classification method of this application, the video data and related features, category information, etc. involved can be stored on the blockchain, such as the target video frame and target description information in the video to be classified, or the first sample video frame and first sample description information in the sample video.

[0132] It should be noted that the video classification method in this application embodiment can be applied to various downstream tasks, such as video search, video recommendation, object detection, scene graph generation, optical flow tracing, product positioning, etc.

[0133] See Figure 2 This document presents an overall flowchart of a video product localization method based on the video classification method described in this application. The method primarily calculates the similarity between multimodal video frames and products to achieve product matching and search, ultimately outputting a product ranking and displaying the product ID and target bounding box of the searched products. Figure 2 As shown in the image.

[0134] Specifically, by combining the target video classification model in the search model to extract key regions of the video, and receiving video frames and audio text feature inputs, the target video classification model automatically learns the selection of key video frames and the extraction of important features. Based on this, the target product in the video can be accurately detected. Furthermore, by combining the image and corresponding title text features of each product contained in the product data, the product corresponding to the target product in the input video is matched through similarity calculation, so as to output a more accurate product classification prediction.

[0135] When applied to video product positioning, this application employs a pre-trained target video classification model. Based on this model, hierarchical causal reasoning (a method for studying causal relationships and their reasoning rules) is used to enhance the representation of visual features, effectively improving recognition accuracy. Furthermore, this application uses heterogeneous modality alignment and contrastive learning mechanisms when training the video classification model. The target classification model trained in this way can effectively improve the complementary effect between modalities during multimodal reasoning and strengthen the fusion effect of heterogeneous features (the model training process will be described in detail below). Based on this, the accuracy of video classification can be effectively improved, thereby further enhancing the accuracy of video product positioning.

[0136] Furthermore, the embodiments of this application can be applied to various scenarios, including but not limited to cloud technology, artificial intelligence, smart transportation, and assisted driving.

[0137] The video classification method provided by the exemplary embodiments of this application will be described below with reference to the accompanying drawings and the application scenarios described above. It should be noted that the application scenarios described above are only shown to facilitate understanding of the spirit and principles of this application, and the embodiments of this application are not limited in any way in this respect.

[0138] See Figure 3 The diagram shown is an implementation flowchart of a video classification method provided in this application embodiment. Taking the server as the execution entity as an example, the specific implementation flow of the method is as follows:

[0139] S31: The server obtains the visual features of each target video frame in the video to be classified, as well as the semantic features corresponding to the target description information of the video to be classified.

[0140] The target video frame can be any video frame in the video to be classified, a video frame within a certain time period, or a video frame containing the target object, etc. Optionally, the number of target video frames is m.

[0141] The target description information refers to the semantic description of the video to be classified. This can be a text description of the video (such as the video title) or an audio description. The audio description needs to be converted into text information using methods such as speech recognition before semantic feature extraction. Optionally, the number of words in the target description information can also be m. If the number of words in the semantic description exceeds m after segmentation, filtering can be performed, such as removing meaningless interjections or words that have little impact on the overall semantic meaning. If the number of words is less than m, it can be supplemented with meaningless characters, etc. This paper does not impose specific limitations on this.

[0142] like Figure 4 As shown, it is a schematic diagram of a video frame and description information listed in the embodiments of this application. Figure 4Only one target video frame is listed here. In reality, the video to be classified is a sequence of many video frames. Some or all of the video frames in this sequence can be used as target video frames. The title of the video can serve as the target description information, such as "A dog and a cat are sharing food".

[0143] It should be noted that the title can be in any language and can contain any characters, such as English letters, numbers, Chinese characters, etc. This article does not impose any specific restrictions.

[0144] Furthermore, the video classification method in this application embodiment can be implemented through machine learning, see [link / reference]. Figure 5 As shown, this is a logical schematic diagram of determining category information through a target video classification model in an embodiment of this application. Specifically, the steps of any video classification method in the embodiments of this application can be executed based on the target video classification model. For example, when extracting semantic features and various visual features based on the target video classification model, an optional implementation is to input each target video frame and target description information into the trained target video classification model, and then extract features based on the feature extraction network in the target video classification model. Figure 5 As shown, the feature extraction network in the target video classification model is divided into two parts: a visual feature extraction part and a semantic feature extraction part. Each of these two parts can correspond to a pre-trained large model (ViLT), also known as a pre-trained network. That is, visual features and semantic features of target description information of each target video frame are extracted in different pre-trained networks. The semantic features of target description information can specifically include the semantic features of each word segment in the target description information.

[0145] S32: The server determines the frame selection weight of each target video frame based on semantic features and various visual features.

[0146] In this embodiment of the application, in order to better select key video frames (hereinafter referred to as key frames) from the video, it is necessary to establish a key frame selection mechanism. The key frame selection mechanism established in this embodiment of the application is mainly based on semantic features and visual features. In this process, it is necessary to combine the above features to calculate the frame selection weight of each target video frame, also known as causal frame weight. If the weight value corresponding to a video frame is larger, it indicates that the video frame has a greater impact on the video classification result. Therefore, key video frames can be screened from each target video frame based on the weight value.

[0147] Optionally, the frame selection weights can be determined in S32 in ways including but not limited to the following two:

[0148] Method 1: First, feature fusion is performed based on semantic features and various visual features to obtain contextual features that match the semantic features; then, based on the attention mechanism, the semantic weights of each target video frame are obtained from the contextual features; and the semantic weights of each target video frame are used as the corresponding frame selection weights.

[0149] This method calculates frame selection weights based on an attention mechanism. Specifically, semantic features and various visual features are fused to obtain a fused result, which is then used as a context feature (also known as a context vector). Furthermore, based on the attention mechanism, the importance of each target video frame for video classification is learned from the context vector to obtain the weight corresponding to each target video frame. This weight is specifically learned through semantic information and can be called semantic weight.

[0150] like Figure 6 The diagram illustrates a method for determining frame selection weights in an embodiment of this application. In this method, the semantic weights of each target video frame can be directly used as the corresponding frame selection weights. For example, the semantic context vector can be input into the attention mechanism module to obtain the semantic causal frame weights (referred to as semantic weights), and the semantic weights of each target video frame can be used as the frame selection weights of the corresponding frames.

[0151] Optionally, when determining contextual features, the semantic features can first be weighted, averaged, and transposed to obtain auxiliary semantic features of the video to be classified; then, the visual features and auxiliary semantic features can be fused to obtain the contextual features.

[0152] For example, the semantic features of a video to be classified are a matrix composed of m semantic vectors. By taking a weighted average of these m semantic vectors and then transposing the result, we can obtain the auxiliary semantic features of the video to be classified.

[0153] Furthermore, when fusing the visual features of the video to be classified with the aforementioned auxiliary semantic features, the specific method involves performing a BMM operation on the visual features and auxiliary semantic features of the video to be classified to obtain context features (also known as context vectors). The visual features of the video to be classified, similar to the semantic features, are matrices composed of the visual features (also known as visual vectors) of each target video frame (a total of m video frames). The BMM operation is a special type of tensor multiplication operation, i.e., matrix multiplication, which achieves feature fusion.

[0154] For example, if m = 50, the semantic weights of the 50 target video frames in the video to be classified are calculated using method one, and are denoted as w. 1-1 w 1-2 w 1-3w 1-4 w 1-5 w 1-6 w 1-7 w 1-8 w 1-9 w 1-10 ,…,w 1-50 The corresponding frame selection weight is also: w 1-1 w 1-2 w 1-3 w 1-4 w 1-5 w 1-6 w 1-7 w 1-8 w 1-9 w 1-10 ,…,w 1-50 .

[0155] Method 2: Feature fusion is performed based on semantic features and various visual features to obtain contextual features that match the semantic features; based on the attention mechanism, the semantic weights of each target video frame are obtained from the contextual features; based on the category-related features and various visual features of the video to be classified, the visual weights of each target video frame are determined; the sum of the semantic weights and visual weights of the same target video frame is used as the corresponding frame selection weights.

[0156] Unlike the first method, this method does not solely rely on attention mechanisms to determine frame selection weights. It further incorporates causal frame selection based on visual information. The keyframe selection mechanism in this method employs a two-layer causal reasoning approach to explicitly constrain key video frames. This involves selecting frame selection weights at the video frame level and learning causal features at the video content level to achieve the learning of video causal features. In the video classification task of this embodiment, causal frames refer to video frames that influence the video classification result, and causal features are similarly defined, referring to features that influence the video classification result.

[0157] Still Figure 6 As shown, the frame selection weights in this method are divided into two parts. One part is the visual causal frame weighting, which is the visual weight in this paper. This weight is determined based on the category-related features of the video to be classified and each visual feature. The category-related features of the video to be classified are features that can characterize the video category of the video to be classified. The other part is the semantic causal frame weighting, which is the semantic weight in this paper. The specific calculation method of this weight can be found in Method 1. Repeated parts will not be repeated.

[0158] The calculation process of visual weights is explained in detail below. The specific steps are as follows:

[0159] An optional implementation method is that the category-related features of the video to be classified can be obtained through self-learning. Specifically, the visual features of each target video frame are learned by the self-learning vector in the self-learning network to obtain the category-related features corresponding to the video to be classified. The category-related features of the video are a matrix composed of m vectors, that is, composed of the category-related features corresponding to each target video frame.

[0160] Based on this, when calculating the visual weight of each target video frame, the category-related features corresponding to each target video frame are first weighted and averaged to obtain reference features. Then, the reference features are concatenated with each visual feature, and the concatenation result is linearly mapped to obtain the visual weight of each target video frame.

[0161] Specifically, by utilizing the category-related features of each target video frame, reference features are obtained by weighting at the frame level. These reference features are then copied into m copies and concatenated with the visual features of each target video frame to obtain m concatenated vectors. These m concatenated vectors are then processed by a linear mapping layer and normalized by an activation function (softmax) to finally obtain the visual weights of each target video frame.

[0162] In Method 2, in addition to using visual causal frame weighting guided by category-related features, semantic causal frame weighting can also be used to guide the selection of frame selection weights in order to further constrain key video frames. Specifically, the semantic constraint on key video frames uses semantic information fused with an attention mechanism for key video frame selection; the specific operation can be found in Method 1, and will not be repeated here.

[0163] For example, if m = 50, the semantic weights of the 50 target video frames in the video to be classified are calculated in the same way, and denoted as w. 1-1 w 1-2 w 1-3 w 1-4 w 1-5 w 1-6 w 1-7 w 1-8 w 1-9 w 1-10 ,…,w 1-50 The visual weights of these 50 target video frames are calculated using the second method described above, and are denoted as w. 2-1 w 2-2 w 2-3 w 2-4 w 2-5 w 2-6 w 2-7 w 2-8 w2-9 w 2-10 ,…,w 2-50 The corresponding frame selection weights are: w 1-1 +w 2-1 w 1-2 +w 2-2 w 1-3 +w 2-3 w 1-4 +w 2-4 w 1-5 +w 2-5 w 1-6 +w 2-6 w 1-7 +w 2-7 w 1-8 +w 2-8 w 1-9 +w 2-9 w 1-10 +w 2-10 ,…,w 1-50 +w 2-50 .

[0164] It should be noted that, in addition to the two frame selection methods listed above, random selection is also possible, that is, randomly determining the weight values ​​corresponding to each target video frame, thereby generating visual enhancement features based on the frame selection weights, etc. This article does not make specific limitations.

[0165] S33: The server selects weights for each frame, determines at least one key video frame in the video to be classified, and determines the visual enhancement features of the video to be classified based on the visual features of each of the at least one key video frame.

[0166] In the embodiments of this application, after determining the frame selection weight of each target video frame using any of the above methods, at least one target video frame can be selected from the video to be classified as the key video frame based on the weight values, while other video frames are removed to reduce information redundancy.

[0167] One possible method for selecting key video frames is to select video frames whose weight values ​​fall within a specified order range based on the weight selection of each frame, and then use these video frames as key video frames.

[0168] The specified order range refers to the range of sorting results when sorting parameters according to their weight values. The parameters here refer to the selection weights of each frame, that is, sorting the selection weights of each frame according to their weight values. For example, when sorting from largest to smallest, the order is the first k (i.e., top k), or when sorting from smallest to largest, the order is the last k, etc. This article does not make specific limitations.

[0169] Similarly, the specified order in the following text is also similar, specifying the order of a sorting result, such as sorting first.

[0170] Taking m=50 as an example, these 50 target video frames are sorted from largest to smallest according to their respective frame selection weights, and the top k are selected as key video frames. For example, if k=10, that is, from these 50 target video frames, the 10 target video frames with the largest corresponding frame selection weights are selected as key video frames.

[0171] Based on the key video frames obtained through screening, feature fusion can be performed on the visual features and corresponding weight values ​​of each key video frame to obtain the visual enhancement features of the video to be classified.

[0172] Specifically, the matrix composed of the weights of the top k frames and the matrix composed of the visual features of the corresponding top k key video frames are fused by dot product, and the final result is used as the visual enhancement feature of the video to be classified.

[0173] Based on the above implementation methods, alignment of video content of the same category can be achieved, a unified visual representation can be learned, causal features of causal frame selection of video can be obtained, and explicit constraints on the selection of key video frames in the video can be realized.

[0174] Optionally, this step can also be implemented using machine learning, such as when obtaining visual enhancement features based on a target video classification model. Figure 5 As shown, the first step is to determine the frame selection weights of each target video frame using the key video frame selection network in the target video classification model. Then, by combining these frame selection weights, the visual features of each key video frame and its corresponding weight value are fused to obtain the visual enhancement features of the video to be classified. The method for determining the frame selection weights using the key video frame selection network can also refer to any of the methods listed above, and will not be repeated here.

[0175] S34: The server performs feature interaction based on visual enhancement features and semantic features to obtain corresponding multimodal interaction features, and determines the category information of the video to be classified based on the multimodal interaction features.

[0176] In this embodiment, a multimodal information interaction method is employed to allow features from heterogeneous modalities to interact, reducing the differences between these features and obtaining multimodal interactive features. Furthermore, based on these multimodal interactive features, category prediction is performed to obtain the video classification result for the video to be classified.

[0177] Optionally, when obtaining category information based on the target video classification model, multimodal interaction features can be transferred to obtain visual transfer representations and semantic transfer representations based on downstream task transfer. Then, category prediction is performed on the representations, and the video classification result is calculated. One optional implementation method is to determine the category information of the video to be classified in the following way:

[0178] First, visual transfer processing and semantic transfer processing are performed on the multimodal interaction features to extract the visual and semantic representation features of the video to be classified. Then, category prediction is performed based on the visual and semantic representation features to obtain the category information of the video to be classified.

[0179] For example, E is obtained by mapping multimodal interaction features and then processing them through a multi-head attention mechanism. vs Then, the visual features E after the interaction are extracted from it. v semantic features E after interaction s After visual perception mapping Aggregate content representation information and generate visual representation features F vt After semantic information mapping Aggregate content representation information and generate semantic representation features F st This enables the extraction of fine-grained heterogeneous features.

[0180] Based on the above implementation, visual perception mapping and semantic perception mapping can be mapped to different feature spaces, realizing interactive feature filtering that enhances multimodal interactive features for downstream tasks.

[0181] Optionally, the process for category prediction based on visual and semantic representation features is as follows:

[0182] First, the category of the video to be classified is predicted based on visual representation features to obtain the first probability value of the video to be classified belonging to each preset category; and second, the category of the video to be classified is predicted based on semantic representation features to obtain the second probability value of the video to be classified belonging to each preset category.

[0183] Then, based on each first probability value and each second probability value, the category information of the video to be classified is determined from each preset category.

[0184] For example, there are three preset categories: games, sports, and education. The first probability values ​​determined based on visual representation features are: games 0.3, sports 0.5, and education 0.2; the second probability values ​​determined based on semantic representation features are: games 0.25, sports 0.55, and education 0.2.

[0185] Therefore, based on the aforementioned first probability values ​​and second probability values, the preset category to which the video to be classified belongs can be determined from the preset categories.

[0186] Optionally, since category prediction was performed based on visual representation features and semantic representation features respectively in the above process, and corresponding prediction results were obtained respectively, namely each first probability value and each second probability value, the category information of the video to be classified can be predicted in the following ways, including but not limited to:

[0187] Prediction Method 1: For each preset category, the average of the first probability value and the second probability value corresponding to the preset category is used as the predicted probability value of the video to be classified belonging to the preset category; then, the preset category corresponding to the predicted probability value whose sorting result is in the specified order among the predicted probability values ​​is used as the category information to which the video to be classified belongs.

[0188] As listed above, for the three preset categories, the predicted probability value for "Games" is the average of 0.3 and 0.25, which is 0.275. Similarly, the predicted probability value for "Sports" is 0.525, and for "Education" it is 0.2. Furthermore, by sorting these predicted probability values ​​in descending order, assuming the specified order is Top 1, the preset category corresponding to 0.525, "Sports," can be used as the category to which the video to be classified belongs.

[0189] Prediction Method 2: For each preset category, the smaller of the first probability value and the second probability value corresponding to the preset category is taken as the predicted probability value of the video to be classified belonging to the preset category; then, the preset category corresponding to the predicted probability value whose sorting result is in the specified order among the predicted probability values ​​is taken as the category information to which the video to be classified belongs.

[0190] As listed above, for the three preset categories, the predicted probability value for "games" is the smaller of 0.3 and 0.25, i.e., 0.25. Similarly, the predicted probability value for "sports" is 0.5, and for "education" it is 0.2. Furthermore, by sorting these predicted probability values ​​in descending order, assuming the specified order is Top 1, the preset category corresponding to 0.5, "sports," can be used as the category to which the video to be classified belongs.

[0191] It should be noted that the preset categories listed above are merely illustrative examples, and any preset category is applicable to the embodiments of this application. These preset categories can be coarse-grained, such as games, sports, education, and live streaming as listed above, or fine-grained, such as game live streaming, e-commerce live streaming, and online education live streaming. This application does not impose specific limitations; the specific category depends on actual needs.

[0192] Optionally, this step can also be implemented using machine learning, such as when obtaining multimodal interaction features based on a target video classification model. Figure 5 As shown, based on the multimodal interaction network in the target video classification model, visual enhancement features and semantic features are interacted to obtain multimodal interaction features. Then, based on these multimodal interaction features, category prediction is performed to obtain the final category information.

[0193] The model training process in the embodiments of this application will be described in detail below:

[0194] One possible implementation is to iteratively train the video classification model to be trained based on the training samples in the training sample set to obtain the target video classification model; wherein each training sample includes: each first sample video frame of the sample video, the first sample description information of the sample video, and the true category label.

[0195] For example, for a given dataset Where v i This represents the i-th video information, i.e., the video frame, s i This represents the i-th semantic description, which can be a text description, an audio description, etc. Category labels There are M preset categories, c i It is one of these M preset categories.

[0196] Based on the above construction of the training sample set, each training sample in the training sample set contains a sample video, which corresponds to multiple first sample video frames. (referred to as video information) ), and the corresponding first description information (abbreviated as semantic description) ), corresponding to the real category label c i Among them, s i It can represent the i-th word in the first description information corresponding to the sample video.

[0197] See Figure 7 As shown, this is a flowchart illustrating a model training method in an embodiment of this application. Each iteration of training executes the following steps:

[0198] S70: Obtain a reference video that matches the sample video.

[0199] The reference video and the sample video have the same real category label, meaning that the reference video and the sample video belong to the same category.

[0200] It should be noted that the reference video in the embodiments of this application can also be referred to as a positive sample video. For a sample video, the true category label is c. i Positive sample matching can be used. Videos and semantic description Matching positive samples and

[0201] in, and This refers to the video information (i.e., each second sample video frame) and semantic description (i.e., the second sample description information) of the positive sample video.

[0202] S71: Input each first sample video frame and the first sample description information into the video classification model to obtain the first sample visual features of each first sample video frame and the first sample semantic features of the first sample description information; input each second sample video frame of the reference video and the corresponding second sample description information into the video classification model to obtain the second sample visual features of each second sample video frame and the second sample semantic features of the second sample description information.

[0203] That is, in step S71, not only do we need to input each first sample video frame and the corresponding first sample description information of the sample video into the video classification model to be trained, but we can also input each second sample video frame and the corresponding second sample description information of the sample video into the video classification model to be trained.

[0204] See Figure 8 The diagram shown illustrates a video classification model according to an embodiment of this application. This model is a multimodal video classification model based on hierarchical causal reasoning, comprising four technical modules: a feature extraction module, a hierarchical causal reasoning module, a causal feature learning module, and a multimodal information interaction module. The feature extraction module is... Figure 5 The feature extraction network, hierarchical causal inference module, and causal feature learning module combination in the target video classification model shown are equivalent to the key video frame selection network in the target video classification model. The multimodal information interaction module is the multimodal interaction network in the target video classification model.

[0205] Specifically, by matching the visual and semantic information of the input video with positive samples, the visual and semantic features of the video are extracted in the feature extraction module.

[0206] like Figure 8 As shown, based on the feature extraction module, from the video... and semantic description Extracting visual features (i.e., the visual features and semantic features of the first sample video frame) (i.e., the semantic features of the first sample description information of each first sample), where and This represents a pre-trained large-scale model (Vision-and-Language Transformer, ViLT) that represents the visual feature extraction part and the semantic feature extraction part, respectively. Furthermore, it can simultaneously obtain the visual features corresponding to positive sample videos. (i.e., the visual features and semantic features of the second sample video frames) (i.e., the semantic features of the second sample description information).

[0207] In terms of semantic features, semantic contrastive learning can be used to constrain the first sample semantic features F of the sample video. s (i.e.) the semantic features of the second sample from the positive sample video That is, based on the difference between the semantic features of the first sample and the semantic features of the second sample, the semantic constraint loss function is determined, and the specific definition is as follows:

[0208]

[0209] in, This is the infoNEC loss function in contrastive learning. By optimizing the loss function to maximize the complementary information of positive samples, samples move closer together in the feature space, while negative samples move further apart. This is the semantic constraint loss function in the embodiments of this application. This semantic constraint loss function can be used to construct the target loss function in step S73.

[0210] S72: The corresponding sample category information is determined by using the multimodal interaction features of the samples obtained based on the semantic features of the first sample and the visual features of each first sample.

[0211] It should be noted that the specific implementation of step S72 is the same as described above. Figure 3 The process of steps S32-S33 is similar to that described above. Figure 3 This refers to the model application phase, where the target video classification model is used to predict the video category to which the video to be classified belongs. S72, on the other hand, is a summary of the model training phase, where the video classification model to be trained is used to predict the video category to which the sample video belongs.

[0212] Specifically, S72 is implemented based on a hierarchical causal reasoning module, a causal feature learning module, and a multimodal information interaction module:

[0213] The hierarchical causal reasoning module is divided into two parts: causal frame weight selection (i.e., frame selection weight selection) and causal feature learning. In the causal frame weight selection part, a weighting mechanism is used to re-weight the co-occurrence probabilities of video frames, while simultaneously employing an attention mechanism that fuses semantic and visual information to guide the weighting of video frame selection. The resulting causal frames are used to measure the importance of each frame in the video, initially filtering out low-quality video frames. In the causal feature learning module, by utilizing the alignment of feature content across videos of the same category, a unified visual representation of samples is learned to obtain the causal features for selecting causal frames in the video, thus achieving explicit constraints on the selection of keyframes in the video.

[0214] Furthermore, in the multimodal information interaction module, features from heterogeneous modalities are interacted to reduce the differences between them, resulting in multimodal interaction features. In the downstream task transfer, these multimodal interaction features are transferred to obtain visual transfer representations and semantic transfer representations. Category prediction is then performed on these representations, and the video classification results are calculated.

[0215] The following section provides a detailed introduction to the hierarchical causal inference module and the causal feature learning module (these two modules are the key video frame selection network):

[0216] An optional implementation includes the following sub-steps S721-S724 in step S72:

[0217] S721: Based on the semantic features of the first sample and the visual features of each first sample, determine the sample frame selection weight of each first sample video frame.

[0218] In this step, the frame selection weight selection is divided into two parts: one part is visual causal frame weighting, i.e., sample visual weight, and the other part is semantic causal frame weighting, i.e., sample semantic weight. An optional implementation is that the calculation method of step S721 is similar to the calculation process of the second method for determining the frame selection weight of each target video frame in the video to be classified, as listed above. Specifically, it can also be divided into the following sub-steps Sa-Sd( Figure 7 (Not shown in the image), that is, in the causal frame weight selection part, the frame selection weight (i.e., causal frame weight) of each first video frame in the sample video can be determined in the following way:

[0219] Sa. Based on the semantic features of the first sample and the visual features of each first sample, feature fusion is performed to obtain sample context features that match the semantic features of the first sample.

[0220] Specifically, the calculation method for sample context features is similar to that for the calculation method for context features of the video to be classified, which will be explained in detail below.

[0221] Sb. Based on the attention mechanism, the semantic weights of each first sample video frame are obtained from the sample context features.

[0222] Steps Sa-Sb are similar to the weight calculation process for the target video frames listed above, and the repetitions will not be repeated. The difference lies in step Sc. During model training, each sample video has a matching positive sample video of the same category, i.e., a reference video. Therefore, the reference features corresponding to the sample video are determined based on the reference video and do not require self-learning. The specific process is as follows:

[0223] Sc. Obtain the second sample visual features of each second sample video frame in the reference video, and determine the first sample visual weight of each first sample video frame based on each second sample visual feature and each first sample visual feature.

[0224] That is, the sample reference features are determined based on the second sample visual features of each second sample video frame in the reference video, and then the first sample visual weights of each first sample video frame are obtained based on the sample reference features and each first sample visual feature.

[0225] For step Sc, one optional implementation is that step Sc can be further divided into the following sub-steps:

[0226] Sc1: The visual features of each second sample are weighted and averaged to obtain the sample reference features.

[0227] Sc2: After concatenating the sample reference features with the visual features of each first sample, perform linear mapping on the concatenation result to obtain the first sample visual weight of each first sample video frame.

[0228] Sd. The sum of the semantic weight and visual weight of the first sample video frame is used as the corresponding sample frame selection weight.

[0229] The following example, using an original sample and its corresponding positive sample, illustrates the calculation process of step Sa-Sd:

[0230] like Figure 8 As shown, the first sample visual feature F of the original sample is obtained from the feature extraction module. v ={v i |i=1,…,m} and the visual features of the second sample corresponding to the positive sample. One of them, v i That is, a first sample visual feature, one That is, a second sample visual feature, and then, in step Sc1, the second sample visual feature of the positive sample is used. The weights for each video frame in the positive samples are obtained by weighting at the frame level, and then extended to the dimensions corresponding to the frames to obtain the weighted visual features of the positive samples. Denote as sample reference features

[0231] Then, in step Sc2, Visual features F of the first sample respectively v The last dimension is concatenated together, so that each frame contains the weighted visual features of the original sample and the positive sample, which is the concatenation result obtained by concatenating the sample reference features with the visual features of each first sample respectively; then the concatenation result is processed by a linear mapping layer to obtain the visual features F of the first sample. v The first sample visual weight w is obtained based on the positive samples. v The specific definition is as follows:

[0232]

[0233] Here, G(.,.) represents concatenating the last dimension of the features from the two inputs, then passing it through a linear mapping layer and normalizing it with softmax to finally obtain the first sample visual weight w of the original sample. v .

[0234] Similarly, the visual weights of the second sample for positive samples can be obtained. The specific definitions are as follows:

[0235]

[0236] That is, the visual features of each first sample are weighted and averaged to obtain the weighted visual features M of the original samples. v And based on the weighted average result M v and visual features of each second sample Determine the second sample visual weights for each second sample video frame. For details, please refer to the above w. v The calculation process will not be repeated here.

[0237] In this embodiment, in addition to using visual causal frame weighting guided by positive samples, the selection of causal frames further constrains key video frames by employing semantic causal frame weighting to guide the selection of frame selection weights. The semantic constraint on key video frames uses semantic information fused with an attention mechanism for key video frame selection, corresponding to steps Sa-Sb above, and the specific operation is as follows:

[0238] In Sa, the first sample semantic features F of the original samples extracted from the feature extraction module are obtained. s ={s i |i=1,…,m}, for F s First, perform a weighted average and transpose to obtain F′. s F, fused with the visual features of the first samplev This assists in semantic feature F′ s And through BMM operation, the sample context features C = {c i |i=1,…,m}, specifically defined as follows:

[0239] C = BMM(F) v ,F′ s ).

[0240] In step Sb, the semantic sample context features C are input into the attention mechanism module to obtain the sample semantic weights w. s This yields the semantic causal frame weighting. The first sample visual weight w of the visual information is then... v The semantic weights w of the sample semantic information s The features are added together to obtain the visual features F of the first sample. v Sample frame selection weight w c The specific definition is as follows:

[0241] w c =w v +w s .

[0242] Therefore, the weights w of the causal frames from both the visual and semantic perspectives can be utilized. c The input is fed into the causal feature learning module, and weighted from the causal frames w c Select the k largest weights w k This guides the selection of causal frames (i.e., key video frames of the samples).

[0243] S722: Based on the weights selected for each sample frame, at least one key sample video frame is determined in the sample video, and based on the first sample visual features of the at least one key sample video frame, the sample visual enhancement features of the sample video are determined.

[0244] The method for determining key video frames in sample videos is similar to the method for determining key video frames in videos to be classified, as listed above. That is, based on the sample frame selection weights corresponding to each first sample video frame, the first sample video frame whose weight value is within the specified order range is selected as the key video frame. Repeated parts will not be described again.

[0245] Specifically, after obtaining the sample frame selection weights, the causal feature learning module is needed to further align the content of videos of the same category. By comparing positive samples, the causal feature F corresponding to each video is learned. c The specific process is as follows:

[0246] like Figure 8 As shown, the first sample visual feature F of the original sample vInput into causal feature learning network In the middle, learn the causal feature F c The specific definition is as follows:

[0247]

[0248] in, It contains two linear mapping layers and one Rule activation function layer.

[0249] In this embodiment of the application, the causal feature learning network Constraints need to be learned using causal features (That is, the visual constraint loss function). One possible implementation is to determine the visual constraint loss function based on the difference between the first sample visual features corresponding to the first sample constraint video frame in the sample video and the second sample visual features corresponding to the second sample constraint video frame in the reference video.

[0250] The first sample constrained video frame is a first sample video frame selected from the sample video whose corresponding weight value is within a specified order range based on the visual weight of each first sample; similarly, the second sample constrained video frame is a second sample constrained video frame selected from the reference video whose corresponding weight value is within a specified order range based on the visual weight of each second sample.

[0251] Based on the above, the visual constraint loss function The calculation process is as follows:

[0252] The first sample visual weight w of the original sample obtained from the visual causal frame weighting v Visual weights of the second sample and positive samples In the process, the visual features corresponding to the top k causal frames with the largest weights (i.e., the matrix composed of the k visual features) are selected, thus obtaining F. k and Among them, F k Includes w v The visual features of the first sample corresponding to the first k values, Include The visual features of the second sample corresponding to the top k values. Then, through... Calculate the loss function between the two. Unconstrained Causal Feature Network The learning process yields causal features learned from positive samples. The specific definition is as follows:

[0253]

[0254] in, The specific operation involves taking the features F of the original input sample frames. kFeatures of positive sample frames First, the corresponding features are obtained by passing them through a linear mapping layer and then weighted and averaged. Finally, the MSE loss between the weighted average features is calculated.

[0255] Finally, weight w is selected based on the sample frames. c The top k largest frames are selected, and their weights are multiplied with the corresponding first sample visual features to obtain the final sample visual enhancement feature F′. v .like Figure 9 As shown, it is a schematic diagram of a keyframe selection mechanism in an embodiment of this application. Figure 9 It refers to Figure 8 Building upon this foundation, further examples of causal frame selection and causal feature learning are provided. Since consecutive video frames have redundancy and the duration differences between different video frames of the same video can cause noise, this application proposes... Figure 9 The keyframe selection mechanism shown solves the information redundancy of invalid frames in video frames by simultaneously modeling causal factors at two levels: causal frame reasoning and causal feature reasoning. It highlights the key information in video frames and filters out low-quality video frames.

[0256] By introducing a cutting-edge multimodal pre-trained large model framework, the fusion of heterogeneous modal features and the interaction between information are enhanced, reducing the distribution differences of heterogeneous features in the feature space and enhancing the representation learning ability between different modalities. The hierarchical causal reasoning-based multimodal video classification algorithm combines the complementarity of fine-grained modal information with the semantic consistency of multimodal information, effectively alleviating the redundancy of video frame information and enabling the extraction of key information from video frames and the identification of video content of the same category.

[0257] The following is a detailed introduction to the multimodal information interaction module (multimodal interaction network):

[0258] S723: Based on the visual enhancement features of the second sample and the semantic features of the first sample, feature interaction is performed to obtain multimodal interaction features of the samples.

[0259] S724: Based on the obtained multimodal interaction features of the samples, determine the corresponding sample category information.

[0260] like Figure 8 As shown, the visual enhancement feature F′ of the sample is... v With the semantic features F of the first sample s The data is fused and fed into a multimodal interaction network. In the interaction feature mapping part, intermodal attention encoding information is learned based on pre-trained knowledge, and the visual enhancement features F′ of the samples obtained from two layers of causal inference are used. v With the semantic features F of the first sample sAs input, it promotes the fusion of multimodal interaction features and the interaction between information, thereby reducing the distributional differences of heterogeneous features in the feature space and generating sample multimodal interaction features E. vs This module facilitates collaborative learning between features from different modalities and allows for transfer to downstream tasks. The specific formula is as follows:

[0261] E vs =ViLT(F′) v ,F s );

[0262] The ViLT(.,.) network can be a 12-layer interactive network, where each layer first incorporates the visual enhancement features F′ of the samples. v and the semantic features F of the first sample s The system employs a multimodal information fusion regularization method (LayerNorm, or LN for short), followed by multi-head attention mechanism (MSA(.)) operation, then LayerNorm normalization, and finally two fully connected networks, including a multi-layer MLP perceptron. This part contains two layers of Norm functions and GELU nonlinear activation function.

[0263] In the downstream task transfer section, visual perception mapping and semantic perception mapping can be mapped to different feature spaces, realizing interaction feature filtering for downstream task enhancement of sample multimodal interaction features. After the sample multimodal interaction features are mapped, E is obtained through multi-head attention mechanism. vs Then, the visual features E after the interaction are extracted from it. v semantic features E after interaction s After visual perception mapping Aggregate content representation information and generate sample visual representation features F vt After semantic information mapping Aggregate content representation information and generate sample semantic representation features F st This enables the extraction of fine-grained heterogeneous features. The specific definition is as follows:

[0264]

[0265]

[0266] Among them, semantic features are simpler than visual features, and therefore the corresponding mapping process is also simpler. See the two formulas above. However, the process of semantic information mapping can also refer to the visual perception mapping process, and MSA and LN processing can be added. Repeated parts will not be repeated.

[0267] During downstream task transfer, the partially heterogeneous mapping f(.) operation can be used to modify the visual representation features F of the samples. vt and sample semantic representation features F st Perform partial heterogeneous mapping and use KL(.,.) to calculate the alignment loss function representing heterogeneous features. The calculation formula is defined as follows:

[0268]

[0269] That is, heterogeneous feature alignment loss function It is determined based on the difference between the target dimension in the visual representation features of the sample and the corresponding dimension in the semantic representation features of the sample. For example, F... vt If the dimension is 100, then the target dimension can refer to the first 60 dimensions, f(F vt That is, 60 dimensions, f(F) st It is also 60-dimensional.

[0270] Based on this, when predicting the category of a video to be classified, each first probability value can be determined based on the first 60 dimensions of the visual representation features of the video to be classified. Similarly, each second probability value can be determined based on the first 60 dimensions of the semantic representation features of the video to be classified.

[0271] In the above implementation, the model's feature extraction and selection capabilities are enhanced by acquiring visual and semantic features from multimodal interactive feature interactions. These features are then used to predict the visual and semantic representations extracted by the model. The representations are mapped through a nonlinear layer to output category prediction information. Cross-entropy (CE) loss and partial heterogeneous mapping loss are used to calculate the classification loss in the single-label classification task. By fully utilizing multimodal video information based on multimodal interactive networks and partial heterogeneous alignment methods, the biases in model fitting and feature fusion of existing algorithms can be effectively overcome, achieving the extraction and unified representation of key multimodal information and improving video classification accuracy.

[0272] The calculation process of the target loss function in the embodiments of this application is described in detail below:

[0273] S73: The video classification model is adjusted by using a target loss function constructed based on sample category information, real category labels and sample multimodal interaction features.

[0274] Specifically, based on model training using positive samples, S73 can be: using a target loss function constructed based on sample category information, real category labels and sample multimodal interaction features, as well as second sample visual features and second sample semantic features, to adjust the parameters of the video classification model.

[0275] Optionally, the target loss function can be constructed in the following manner, including the following steps S731-S733 ( Figure 7 (Not shown in the image):

[0276] S731: Construct a classification prediction loss function based on the difference between sample category information and the true category label.

[0277] In other words, a classification prediction loss function is constructed based on the difference between the final predicted result and the true result. Specifically, considering that sample category information may include first sample category information obtained by prediction based on sample visual representation features, and second sample category information obtained by prediction based on sample semantic representation features, the classification prediction loss function can be: the sum of the visual classification loss function determined based on the difference between the first sample category information and the true category label, and the semantic classification loss function determined based on the difference between the second sample category information and the true category label.

[0278] S732: Determine the heterogeneous feature alignment loss function based on the difference between the target dimension in the visual representation features of the sample and the corresponding dimension in the semantic representation features of the sample.

[0279] This step corresponds to the steps listed above. Repeated points will not be elaborated upon.

[0280] S733: Construct a target loss function based on the classification prediction loss function and the heterogeneous feature alignment loss function.

[0281] Optionally, in S733, a constraint loss function can be further determined based on the difference between the sample video and the reference video matched with the sample video; then, the target loss function is obtained by weighting the classification prediction loss function, the heterogeneous feature alignment loss function, and the constraint loss function.

[0282] Optionally, the constraint loss function in the embodiments of this application includes a semantic constraint loss function. and visual constraint loss function At least one of the following. The specific calculation methods for these two constraint loss functions can be found in the examples; repeated details will not be elaborated further.

[0283] In this embodiment, an optional training strategy is as follows: Since videos contain information across multiple modalities, including semantic and visual information, and the visual information in videos is closely related to the temporal structure, while semantic information can help to better understand the video, this application needs to try a method to fully utilize this information, that is, to find an optimal way to fuse different modalities. Addressing the difficulty of multimodal video classification tasks, the (target) video classification model CasCausal-Net in this embodiment iteratively trains the model by summing five parts: the cross-entropy (CE) loss in the classification prediction (including semantic classification prediction loss and visual classification prediction loss in this paper), the loss used to constrain causal feature learning, the visual and semantic feature alignment loss, and the contrastive loss of semantic information. Based on this, the task classification has two stages: single-modal training and multimodal interactive feature fusion.

[0284] A visual classification prediction loss function is used in the multimodal video stage based on hierarchical causal reasoning. and semantic classification prediction loss function Visual constraint loss function for constrained causal features Semantic Constraint Loss Function for Semantic Information Comparative Learning and heterogeneous feature alignment loss function sum To train the model, the specific definition is as follows:

[0285]

[0286] Where α is The weighting coefficient, β is The weighting coefficient, γ is The weighting coefficients.

[0287] It should be noted that the model training process listed above mainly corresponds to the calculation method of frame selection weights listed in Method 1, which requires positive samples as a reference.

[0288] Accordingly, the method for determining frame selection weights based on the second method does not require positive samples as references, and the corresponding model training process is as follows:

[0289] See Figure 10 As shown, this is a flowchart illustrating another model training method in this application embodiment. Each iteration of training executes the following steps:

[0290] S101: Input each first sample video frame and the first sample description information into the video classification model to obtain the first sample visual features of each first sample video frame and the first sample semantic features of the first sample description information.

[0291] S102: Using the multimodal interaction features of the samples obtained based on the semantic features of the first sample and the visual features of each first sample, the corresponding sample category information is determined.

[0292] S102 is the same as S72, and can be divided into the following sub-steps S1021-S1024:

[0293] S1021: Based on the semantic features of the first sample and the visual features of each first sample, determine the sample frame selection weight of each first sample video frame.

[0294] Optionally, the calculation method of step S1021 is similar to the calculation process of method one for determining the frame selection weight of each target video frame in the video to be classified, as listed above. Specifically, it can also be divided into the following sub-steps Sa-Sc( Figure 10 (Not shown in the image):

[0295] Sa. Based on the semantic features of the first sample and the visual features of each first sample, feature fusion is performed to obtain sample context features that match the semantic features of the first sample;

[0296] Sb. Based on the attention mechanism, obtain the sample semantic weights of each first sample video frame from the sample context features;

[0297] Sc. The semantic weights of the same first sample video frame are used as the corresponding sample frame selection weights.

[0298] S1022: Based on the weights selected for each sample frame, at least one key sample video frame is determined in the sample video, and based on the first sample visual features of the at least one key sample video frame, the sample visual enhancement features of the sample video are determined.

[0299] S1023: Based on the visual enhancement features of the second sample and the semantic features of the first sample, feature interaction is performed to obtain multimodal interaction features of the samples.

[0300] S1024: Based on the obtained multimodal interaction features of the samples, determine the corresponding sample category information.

[0301] S103: The video classification model is adjusted by using a target loss function constructed based on sample category information, real category labels and sample multimodal interaction features.

[0302] It should be noted that in step S103, only two types of loss functions need to be constructed: a classification prediction loss function (including a visual classification prediction loss function and a semantic classification prediction loss function) and a heterogeneous feature alignment loss function, to determine the target loss function, such as... The specific definitions are as follows:

[0303]

[0304] It should be noted that, Figure 10 The calculation process listed can be found in the above embodiments; repeated parts will not be described again.

[0305] like Figure 11 As shown, this is a schematic diagram of another video classification model in the embodiments of this application, which corresponds to a method for calculating sample frame selection weights based on an attention mechanism to generate sample visual enhancement features.

[0306] Specifically, based on the feature extraction network, from the video... and semantic description Extracting visual features from the first sample and semantic features of the first sample In the key video frame selection network, an attention mechanism is used to select from F... v Select key video frames F′ of the sample v In a multimodal interactive network, the first sample visual feature F′ corresponding to the key video frames of the sample is selected. v and the semantic features F of the first sample s The feature interaction mapping section enables interaction between modules of heterogeneous features, resulting in deep-level multimodal interaction features E of the samples. vs In downstream task transfer, the multimodal interaction features E of the samples will be transferred. vs The visual representation features e of the sample are obtained after the visual feature transfer layer. v The semantic feature transfer layer is used to obtain the semantic representation features e of the sample. s Then, the visual representation features e of the sample. v and sample semantic representation features e s Predict and perform heterogeneous alignment to predict video classification labels. For specific implementation methods, please refer to the above embodiments; repeated details will not be repeated here.

[0307] Furthermore, it should be noted that the multimodal interaction networks listed in the embodiments of this application can be replaced by time-series networks such as Long Short-Term Memory (LSTM) and Gate Recurrent Unit (GRU), and this document does not impose any specific limitations.

[0308] See Figure 12 The diagram shown is a flowchart of a method for classifying videos to be classified according to an embodiment of this application. Taking a server as the execution entity as an example, the specific implementation process of this method is as follows:

[0309] Step S1200: The server obtains each target video frame and target description information of the video to be classified;

[0310] Step S1201: The server inputs each target video frame and target description information into the target video classification model, and obtains the visual features of each target video frame and the semantic features corresponding to the target description information based on the feature extraction network in the model.

[0311] Step S1202: The server inputs each visual feature and semantic feature into the key video frame selection network in the model. Based on the key video frame selection network, the semantic features and each visual feature are fused to obtain context features that match the semantic features.

[0312] Step S1203: The server obtains the semantic weights of each target video frame from the context features based on the attention mechanism;

[0313] Step S1204: The server determines the visual weight of each target video frame based on the category-related features and various visual features of the video.

[0314] Step S1205: The server uses the sum of the semantic weight and visual weight of the same target video frame as the corresponding frame selection weight;

[0315] Step S1206: The server selects the k video frames with the largest corresponding weight values ​​from each target video frame based on the weight of each frame, and uses them as key video frames.

[0316] Step S1207: The server performs feature fusion based on the visual features and corresponding weight values ​​of each key video frame to obtain the visual enhancement features of the video.

[0317] Step S1208: The server inputs the visual enhancement features and semantic features into the multimodal interaction network in the model, and performs feature interaction on the visual enhancement features and semantic features based on the multimodal interaction network to obtain the corresponding multimodal interaction features;

[0318] Step S1209: The server performs visual transfer processing and semantic transfer processing on the multimodal interaction features respectively, and extracts the visual representation features and semantic representation features of the video from the multimodal interaction features.

[0319] Step S1210: The server performs category prediction on the video based on visual representation features to obtain the first probability value of the video belonging to each preset category; and performs category prediction on the video based on semantic representation features to obtain the second probability value of the video belonging to each preset category.

[0320] Step S1211: For each preset category, the server takes the smaller of the first probability value and the second probability value corresponding to the preset category as the predicted probability value of the video belonging to a preset category;

[0321] Step S1212: The server uses the preset category corresponding to the highest predicted probability value among all predicted probability values ​​as the category information to which the video belongs.

[0322] It should be noted that this application also provides relevant implementations for the model training listed above. Specifically, this application conducted experiments on two real-world datasets, MSR-VTT and ActivityNet Captions. The dataset statistics are shown in Table 1.

[0323] Table 1

[0324]

[0325]

[0326] The MSR-VTT dataset contains 10,000 unique video clips. Each of these is annotated with 20 different text captions, resulting in a total of 200,000 video caption pairs. This application splits the MSR-VTT dataset into 9,000 and 1,000 videos for training and testing, respectively.

[0327] The ActivityNet Captions dataset contains 20,000 captioned videos, totaling 849 video hours and 100,000 segments. Each video has a unique start and end time. On average, each 20,000 videos contains 3.65 temporally localized sentences, for a total of 100,000 sentences. The sentences per segment follow a relatively normal distribution. Furthermore, the number of sentences increases with video duration. The average sentence length is 13.48 words, also exhibiting a normal distribution. Since some videos were not officially labeled, the labeled sample data was split into 10,009 and 4,515 videos for training and testing, respectively.

[0328] Furthermore, in experiments on the MSR-VTT and ActivityNet Captions datasets, this application uses accuracy to evaluate the model's prediction performance in single-label classification, as shown in the following formula:

[0329] Accuracy formula:

[0330]

[0331] Wherein, TP represents the number of positive samples, FP represents the number of negative samples, TN represents the number of positive samples, and FN represents the number of negative samples. For the above indicators, this application calculates the average of TOP-1 and TOP-5.

[0332] In terms of data preprocessing, the input raw video dataset needs to be processed by different feature extraction networks to extract corresponding features. For the MSR-VTT dataset, visual motion features and text features corresponding to the text describing the video are extracted. Motion features are extracted using a pre-trained S3D (Sensaura 3D, an audio technology) network with a feature dimension of 1024; text features are extracted using a Cloud Speech to Text API with a feature dimension of 768. For the ActivityNet Captions dataset, visual motion features and audio features corresponding to the audio in the video are extracted. Motion features are also extracted using a pre-trained S3D network with a feature dimension of 1024; audio features are extracted using a VGGish network pre-trained on the YouTube-8M dataset with a feature dimension of 128. Since the task is video classification, corresponding label text is obtained based on the division of the test set and the training set to test the accuracy of the classification results. Therefore, the MSR-VTT dataset contains visual motion features and semantic text features, while the ActivityNet Captions dataset contains visual motion features and semantic audio features. Through dataset preprocessing, the multimodal information corresponding to the two datasets was obtained.

[0333] Regarding the details of model implementation, in the experiments, this application followed the pre-frontal training model VILT feature dimension setting, setting the feature dimension to 768. The experimental model was optimized using the Adam optimizer during training, with the learning rate selected from 1e-6 to 5e-3, and the optimizer's learning rate decayed to 0.1 times its original value every 4 epochs. The training sample size was set to 64. α represents the constraint causal features. The weighting coefficients, where β is the loss for aligning partially heterogeneous features. The weight coefficients, where γ is the semantic information contrastive learning loss. The weighting coefficients α, β, and γ are set in the range of 0.1 to 0.5.

[0334] In terms of comparative experimental results, this application demonstrates the performance of a multimodal video classification algorithm based on hierarchical causal inference on two datasets, comparing the performance of the basic visual model GRU backbone network. Video classification experiments based on visual information, semantic information, and multimodal information were conducted on both models. Furthermore, several key video frame selection mechanisms were employed: one based on random frame selection, one based on attention fusion, and one based on a multimodal video classification algorithm using hierarchical causal inference. Simultaneously, the performance of different modal features in early fusion (EF) and late fusion (LF) for downstream classification tasks was compared on the GRU model. As shown in Table 2, this application makes the following observations:

[0335] Table 2: Comparative Experimental Results—Video Classification Algorithm Based on Multimodal Information Fusion of Key Feature Frame Selection

[0336]

[0337]

[0338] Here, Stochastic represents random selection, Attention corresponds to the first method determined in this paper, and CasCausal corresponds to the second method determined in this paper. Based on the above, it can be seen that after multimodal information fusion, the video classification performance is higher than that using only single-modal features. Among the various key video frame selection mechanisms established, the multimodal video classification algorithm using hierarchical causal inference performs best, indicating that two layers of causal inference at the video frame level and video content level can effectively constrain key video frames in the video and achieve content alignment of videos of the same category. The video classification algorithm based on attention-based key video frame selection performs second best, indicating that utilizing the complementarity between visual and semantic features can effectively constrain key video frames in the video. The Stochastic video classification algorithm with random frame selection mechanism, due to its randomness, introduces more noise than using all visual feature frames, thus resulting in poor classification performance.

[0339] In the GRU model, comparing the video classification performance of pre-fusion and post-fusion under three modality fusion methods reveals that the classification performance of features from different modalities that are fused through an interactive feature network and then directly transferred to the downstream classification task is higher than that of features directly fused and sent to the interactive network. This indicates that the different value ranges and distributions of heterogeneous features can cause fusion bias. The CasCausal video classification method can effectively overcome this bias.

[0340] The multimodal video classification algorithm based on hierarchical causal reasoning showed significant performance improvement when applied to different backbone networks, demonstrating its model independence.

[0341] The pre-trained large model ViLT performed worse than the basic backbone network GRU on the experimental dataset, indicating that the pre-trained large model has poor generalization ability due to issues such as data distribution bias during transfer learning. CasCausal-Net reduces the bias caused by visual noise interference through causal frame selection and improves the transfer performance through the alignment of heterogeneous features, effectively enhancing the ability of the pre-trained large model to transfer downstream tasks.

[0342] In addition, to investigate the effectiveness of the proposed framework, this application also conducted ablation experiments, the results of which are shown in Table 3.

[0343] Table 3: Impact of each module of the algorithm on prediction performance

[0344]

[0345] In this application embodiment, the experiment selected a video classification algorithm S based on single-modal visual information and a video classification algorithm V based on single-modal semantic information as baselines. The different classification results of V and S indicate that there are differences in the value range and distribution between different modalities. However, due to the consistency between multimodal information and the complementary data characteristics between fine-grained modalities, the algorithm using multimodal information fusion (V+S) performs better in downstream tasks than the direct classification performance of single-modal semantic features (S) and visual features (V). Since continuous video frames have redundancy and the difference in duration between different video frames of the same video can cause noise, a key video frame selection mechanism is established based on the above algorithms. At the same time, in the two-layer causal modeling of causal frame inference and causal feature inference, this application explored the causal feature inference network and divided it into self-learning causal feature inference CasCausal (S), direct causal feature inference CasCausal (F), and multi-layer causal feature inference CasCausal (M) according to the different causal feature learning in the second-level causal inference. Experiments show that V+S+CasCausal(M) indicates that features learned from causal features are more beneficial for downstream task classification. By implementing weighting at the video frame level through causal frames and learning causal features through multi-layer mapping, content-aligned causal features are obtained. Combining causal frames from first-level inference with weighted causal features can further robustly improve the accuracy of video classification.

[0346] In summary, this application analyzes the visual and textual information representations of a video classification algorithm based on an attention mechanism and a frame selection mechanism at different stages to explore the fusion of heterogeneous features at different stages. Figures 13A-13C This section presents the visual and textual information representations of a keyframe-selected video classification algorithm at different stages, showcasing the visual and semantic information visualizations of 20 randomly selected test samples. Figure 13A (a) in the figure is the shallow representation obtained in the feature extraction network; Figure 13B (b) in the text refers to the features of the attention mechanism extracted by the key video frame selection mechanism module based on the attention mechanism. Figure 13C (c) in the interaction feature mapping section extracts multimodal interaction features.

[0347] from Figures 13A-13C It can be seen that the representation distribution of different modalities of visual and textual information varies significantly in the embedding space of t-SNE. As the frame selection algorithm based on the attention mechanism is executed, the representation distribution of visual and semantic information from the same multimodal video gradually becomes consistent.

[0348] In detail, after passing through the shallow features obtained directly by the feature extraction network, the distribution of shallow visual and semantic representations of the same video shows a clear distinction; that is, for the same digital nodes representing the same visual-semantic information pair, they are distributed in relatively far apart positions. After passing through the attention-based frame selection mechanism, the distribution of visual and semantic information in the feature space shows a trend of convergence, indicating that the attention-based selection mechanism can achieve high semantic value in frame selection and extract key video frames. After passing through the multimodal interaction mapping part, the distribution of the interaction representations between the two modalities in the feature space gradually becomes consistent, indicating that the ViLT model can alleviate the problem of inconsistent distribution of heterogeneous features in the feature space. These observations demonstrate that the attention-based frame selection mechanism can effectively select key video frames and promote the model's representation learning ability, making the distribution of visual-semantic information pairs from the same video similar in the embedding space of t-SNE, which may be one reason for the improved video classification performance.

[0349] In summary, this application proposes a general multimodal video classification algorithm based on hierarchical causal reasoning. It leverages the complementary nature of fine-grained intermodal information and the semantic consistency among multimodal information, along with causal modeling to establish causal frame selection at the video frame level and causal feature selection at the video content level. This effectively removes video frame information detrimental to classification, achieves adaptive keyframe selection, and improves video classification accuracy. Furthermore, it employs the cutting-edge multimodal pre-trained large-scale model ViLT, utilizing a pre-trained interactive network to promote multimodal feature fusion and information interaction, thereby reducing the differences in the distribution of heterogeneous features in the feature space. By selecting positive samples for each video, alignment of video content is achieved. Through the extraction and unified representation of multimodal key information, the algorithm enables the extraction and collaborative learning of common features beneficial to classification across modalities.

[0350] Based on the same inventive concept, embodiments of this application also provide a video classification device. For example... Figure 14As shown, this is a structural schematic diagram of the video classification device 1400, which may include:

[0351] The feature extraction unit 1401 is used to obtain the visual features of each target video frame in the video to be classified, as well as the semantic features corresponding to the target description information of the video to be classified.

[0352] The weight determination unit 1402 is used to determine the frame selection weight of each target video frame based on semantic features and various visual features.

[0353] The visual enhancement unit 1403 is used to select weights for each frame, determine at least one key video frame in the video to be classified, and determine the visual enhancement features of the video to be classified based on the visual features of each of the at least one key video frame.

[0354] The classification unit 1404 is used to perform feature interaction based on visual enhancement features and semantic features to obtain corresponding multimodal interaction features, and to determine the category information of the video to be classified based on the multimodal interaction features.

[0355] Optionally, classification unit 1404 is specifically used for:

[0356] Visual transfer processing and semantic transfer processing are performed on the multimodal interaction features respectively, and the visual representation features and semantic representation features of the video to be classified are extracted from the multimodal interaction features.

[0357] Category prediction is performed based on visual and semantic representation features to obtain category information for the video to be classified.

[0358] Optionally, classification unit 1404 is specifically used for:

[0359] Based on visual representation features, the class of the video to be classified is predicted to obtain a first probability value that the video to be classified belongs to each preset class; and based on semantic representation features, the class of the video to be classified is predicted to obtain a second probability value that the video to be classified belongs to each preset class.

[0360] Based on each first probability value and each second probability value, the category information of the video to be classified is determined from each preset category.

[0361] Optionally, classification unit 1404 is specifically used for:

[0362] For each preset category, perform the following operations: take the smaller of the first probability value and the second probability value corresponding to a preset category, or the average of the first probability value and the second probability value, as the predicted probability value of the video to be classified belonging to a preset category;

[0363] The preset category corresponding to the predicted probability value that is ranked in a specified order among the predicted probability values ​​is used as the category information to which the video to be classified belongs.

[0364] Optionally, the weight determination unit 1402 is specifically used for:

[0365] Feature fusion is performed based on semantic features and various visual features to obtain contextual features that match the semantic features;

[0366] Based on the attention mechanism, the semantic weights of each target video frame are obtained from the context features;

[0367] Perform any of the following operations:

[0368] The semantic weights of each target video frame are used as the corresponding frame selection weights.

[0369] Based on the category-related features and visual features of the video to be classified, the visual weight of each target video frame is determined; the sum of the semantic weight and visual weight of the same target video frame is used as the corresponding frame selection weight.

[0370] Optionally, the weight determination unit 1402 is specifically used for:

[0371] After weighted averaging and transposing the semantic features, auxiliary semantic features of the video to be classified are obtained.

[0372] By fusing visual features with auxiliary semantic features, contextual features are obtained.

[0373] Optionally, category-related features include: category-related features corresponding to each target video frame, determined by a self-learning network based on the visual features of each target video frame.

[0374] The weight determination unit 1402 is specifically used for:

[0375] The reference features are obtained by weighted averaging the category-related features corresponding to each target video frame.

[0376] After concatenating the reference features with each visual feature, the concatenation result is linearly mapped to obtain the visual weights of each target video frame.

[0377] Optionally, the visual enhancement unit 1403 is specifically used for:

[0378] Based on the weight selection of each frame, the video frames whose corresponding weight values ​​are within the specified order range in each target video frame are taken as key video frames.

[0379] Feature fusion is performed based on the visual features and corresponding weight values ​​of each key video frame to obtain the visual enhancement features of the video to be classified.

[0380] Optionally, semantic features and visual features are obtained by inputting each target video frame and target description information into a trained target video classification model, and then using the feature extraction network in the target video classification model.

[0381] Visual enhancement features are obtained by determining the frame selection weights of each target video frame based on semantic features and various visual features through the key video frame selection network in the target video classification model.

[0382] Multimodal interaction features are obtained by performing feature interaction between visual enhancement features and semantic features based on the multimodal interaction network in the target video classification model.

[0383] Optionally, the device also includes:

[0384] Model training unit 1405 is used to obtain the target video classification model in the following ways:

[0385] Based on the training samples in the training sample set, the video classification model to be trained is trained iteratively to obtain the target video classification model. Each training sample contains: the first sample video frame of the sample video, the first sample description information of the sample video, and the true class label. Each iterative training iteration executes the following steps:

[0386] Each first sample video frame and the first sample description information are input into the video classification model to obtain the first sample visual features of each first sample video frame and the first sample semantic features of the first sample description information.

[0387] The corresponding sample category information is determined by using the multimodal interaction features of the samples obtained based on the semantic features of the first sample and the visual features of each first sample.

[0388] The video classification model is parameter-tuned by employing a target loss function constructed based on sample category information, real category labels, and multimodal interaction features of samples.

[0389] Optionally, the model training unit 1405 is also used to obtain multimodal interaction features of the samples in the following ways:

[0390] Based on the semantic features of the first sample and the visual features of each first sample, the sample frame selection weight of each first sample video frame is determined.

[0391] Based on the selection weights of each sample frame, at least one sample key video frame is determined in the sample video, and based on the first sample visual features of the at least one sample key video frame, the sample visual enhancement features of the sample video are determined.

[0392] Based on the visual enhancement features of the second sample and the semantic features of the first sample, feature interaction is performed to obtain multimodal interaction features of the samples.

[0393] Optionally, model training unit 1405 is specifically used for:

[0394] Feature fusion is performed based on the semantic features of the first sample and the visual features of each first sample to obtain sample context features that match the semantic features of the first sample.

[0395] Based on the attention mechanism, the semantic weights of each first sample video frame are obtained from the sample context features;

[0396] Obtain the second sample visual features of each second sample video frame in the reference video, and determine the first sample visual weight of each first sample video frame based on each second sample visual feature and each first sample visual feature; the real category label of the reference video and the sample video is the same.

[0397] The sum of the semantic weight and visual weight of the same first sample video frame is used as the corresponding sample frame selection weight.

[0398] Optionally, model training unit 1405 is specifically used for:

[0399] The visual features of each second sample are weighted and averaged to obtain the sample reference features.

[0400] After concatenating the sample reference features with the visual features of each first sample, the concatenation results are linearly mapped to obtain the first sample visual weights of each first sample video frame.

[0401] Optionally, model training unit 1405 is also used for:

[0402] Before inputting each first sample video frame and the first sample description information into the video classification model, a reference video matching the sample video is obtained. The reference video has the same true category label as the sample video.

[0403] Input each second sample video frame of the reference video and the corresponding second sample description information into the video classification model to obtain the second sample visual features of each second sample video frame and the second sample semantic features of the second sample description information.

[0404] Optionally, model training unit 1405 is specifically used to construct the target loss function in the following ways:

[0405] Based on the difference between sample category information and true category labels, a classification prediction loss function is constructed;

[0406] Based on the difference between the target dimension in the visual representation features of the samples and the corresponding dimension in the semantic representation features of the samples, a heterogeneous feature alignment loss function is determined; the visual representation features and semantic representation features are extracted from the multimodal interaction features of the samples through feature transfer.

[0407] A target loss function is constructed based on the classification prediction loss function and the heterogeneous feature alignment loss function.

[0408] Optionally, model training unit 1405 is specifically used for:

[0409] The constrained loss function is determined based on the differences between the sample video and the reference video that matches the sample video; the true category labels of the reference video and the sample video are the same.

[0410] The target loss function is obtained by weighting the classification prediction loss function, the heterogeneous feature alignment loss function, and the constraint loss function.

[0411] Optionally, the sample category information includes first sample category information obtained based on sample visual representation features, and second sample category information obtained based on sample semantic representation features;

[0412] Model training unit 1405 is specifically used for:

[0413] Based on the difference between the category information of the first sample and the true category label, a visual classification loss function is determined; and based on the difference between the category information of the second sample and the true category label, a semantic classification loss function is determined.

[0414] A classification prediction loss function is constructed based on the visual classification loss function and the semantic classification loss function.

[0415] Optionally, the constraint loss function includes at least one of a semantic constraint loss function and a visual constraint loss function; the model training unit 1405 is also used to determine the constraint loss function in the following ways:

[0416] The semantic constraint loss function is determined based on the difference between the semantic features of the first sample and the semantic features of the second sample; the semantic features of the second sample are determined based on the second sample description information of the reference video.

[0417] Based on the difference between the first sample visual features corresponding to the first sample constrained video frame in the sample video and the second sample visual features corresponding to the second sample constrained video frame in the reference video, the visual constraint loss function is determined.

[0418] Optionally, the model training unit 1405 is also used to determine the first sample constrained video frame and the second sample constrained video frame in the following manner:

[0419] Based on the visual weights of each first sample, at least one first sample constrained video frame is determined in the sample video.

[0420] The visual features of each first sample are weighted and averaged. Based on the weighted average and the visual features of each second sample, the visual weight of each second sample video frame is determined. Based on the visual weight of each second sample, at least one second sample constrained video frame is determined in the reference video.

[0421] Based on the aforementioned video classification device, this application analyzes the frame selection weights of each target video frame in the video to be classified by acquiring visual and semantic features. These weights are then used to filter video frames, identifying key video frames that significantly impact the classification results. This improves the selection of key video frames and the learning of important features, effectively reducing redundant information and visual noise. The visual features are then enhanced using the key video frames and their corresponding frame selection weights to obtain the visual enhancement features of the video to be classified. Furthermore, the enhanced visual features and semantic features are interacted to achieve information exchange and complementarity between different modalities. This fully utilizes multimodal video information, enabling the extraction and unified representation of key multimodal information, thereby improving video classification accuracy.

[0422] For ease of description, the above sections are divided into modules (or units) according to their functions and described separately. Of course, in implementing this application, the functions of each module (or unit) can be implemented in one or more software or hardware components.

[0423] Having introduced the video classification method and apparatus according to exemplary embodiments of this application, we will now introduce an electronic device according to another exemplary embodiment of this application.

[0424] Those skilled in the art will understand that various aspects of this application can be implemented as a system, method, or program product. Therefore, various aspects of this application can be specifically implemented in the following forms: a completely hardware implementation, a completely software implementation (including firmware, microcode, etc.), or a combination of hardware and software implementations, collectively referred to herein as a "circuit," "module," or "system."

[0425] Based on the same inventive concept as the above-described method embodiments, this application also provides an electronic device. In this embodiment, the structure of the electronic device can be as follows: Figure 15 As shown, it includes a memory 1501, a communication module 1503, and one or more processors 1502.

[0426] The memory 1501 is used to store computer programs executed by the processor 1502. The memory 1501 may mainly include a program storage area and a data storage area. The program storage area may store the operating system and programs required to run instant messaging functions, etc.; the data storage area may store various instant messaging information and operation instruction sets, etc.

[0427] Memory 1501 may be volatile memory, such as random-access memory (RAM); memory 1501 may also be non-volatile memory, such as read-only memory, flash memory, hard disk drive (HDD), or solid-state drive (SSD); or memory 1501 may be any other medium capable of carrying or storing a desired computer program having the form of instructions or data structures and accessible by a computer, but is not limited thereto. Memory 1501 may be a combination of the above-described memories.

[0428] Processor 1502 may include one or more central processing units (CPUs) or digital processing units, etc. Processor 1502 is used to implement the above-described video classification method when it calls the computer program stored in memory 1501.

[0429] The communication module 1503 is used to communicate with terminal devices and other servers.

[0430] This application embodiment does not limit the specific connection medium between the memory 1501, communication module 1503, and processor 1502. This application embodiment... Figure 15 The memory 1501 and the processor 1502 are connected via a bus 1504, and the bus 1504 is in Figure 15 The diagram uses thick lines to describe the connections between other components; these are for illustrative purposes only and should not be considered limiting. The 1504 bus can be divided into address bus, data bus, control bus, etc. For ease of description, Figure 15 It is described using only a thick line, but does not indicate that there is only one bus or one type of bus.

[0431] The memory 1501 stores a computer storage medium, which stores computer-executable instructions for implementing the video classification method of this application embodiment. The processor 1502 is used to execute the above-described video classification method, such as... Figure 3 As shown.

[0432] In some possible implementations, various aspects of the video classification method provided in this application can also be implemented as a program product, which includes a computer program. When the program product is run on an electronic device, the computer program causes the electronic device to perform the steps of the video classification method according to the various exemplary embodiments of this application described above. For example, the electronic device can perform actions such as... Figure 3 The steps are shown in the figure.

[0433] The program product may employ any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples (a non-exhaustive list) of readable storage media include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0434] The program product of the embodiments of this application may employ a portable compact disc read-only memory (CD-ROM) and include a computer program, and may run on an electronic device. However, the program product of this application is not limited thereto. In this application, the readable storage medium may be any tangible medium that contains or stores a program that may be used by or in conjunction with a command execution system, apparatus, or device.

[0435] A readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying a readable computer program. This propagated data signal may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable signal medium may also be any readable medium other than a readable storage medium, capable of sending, propagating, or transmitting a program for use by or in conjunction with a command execution system, apparatus, or device.

[0436] Computer programs contained on readable media may be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RF, etc., or any suitable combination thereof.

[0437] Computer programs for performing the operations of this application can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java and C++, and conventional procedural programming languages ​​such as C or similar languages. The computer program can execute entirely on the user's electronic device, partially on the user's electronic device, as a standalone software package, partially on the user's electronic device and partially on a remote electronic device, or entirely on a remote electronic device or server. In cases involving remote electronic devices, the remote electronic device can be connected to the user's electronic device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external electronic device (e.g., via the Internet using an Internet service provider).

[0438] It should be noted that although several units or sub-units of the device have been mentioned in the detailed description above, this division is merely exemplary and not mandatory. In fact, according to embodiments of this application, the features and functions of two or more units described above can be embodied in one unit. Conversely, the features and functions of one unit described above can be further divided and embodied by multiple units.

[0439] Furthermore, although the operations of the method of this application are described in a specific order in the accompanying drawings, this does not require or imply that these operations must be performed in that specific order, or that all the operations shown must be performed to achieve the desired result. Additionally or alternatively, certain steps may be omitted, multiple steps may be combined into one step, and / or one step may be broken down into multiple steps.

[0440] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing a computer-usable computer program.

[0441] Although preferred embodiments of this application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of this application.

[0442] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A method of video classification, characterized by, The method comprises: obtaining visual features of each target video frame in a video to be classified and semantic features corresponding to target description information of the video to be classified; performing feature fusion based on the semantic features and the visual features to obtain context features matched with the semantic features; obtaining semantic weights of the target video frames from the context features based on an attention mechanism; and taking the semantic weights of the target video frames as respective frame selection weights; determining at least one key video frame in the video to be classified based on the frame selection weights and determining visual enhanced features of the video to be classified based on visual features of the at least one key video frame; performing feature interaction based on the visual enhanced features and the semantic features to obtain corresponding multi-modal interaction features and determining category information of the video to be classified based on the multi-modal interaction features.

2. The method of claim 1, wherein, The determining of the category information of the video to be classified based on the multi-modal interaction features comprises: performing visual migration processing and semantic migration processing on the multi-modal interaction features to extract visual representation features and semantic representation features of the video to be classified from the multi-modal interaction features; performing category prediction based on the visual representation features and the semantic representation features to obtain the category information of the video to be classified.

3. The method of claim 2, wherein, The determining of the category information of the video to be classified based on the visual representation features and the semantic representation features comprises: performing category prediction on the video to be classified based on the visual representation features to obtain first probability values of the video to be classified belonging to each preset category and performing category prediction on the video to be classified based on the semantic representation features to obtain second probability values of the video to be classified belonging to each preset category; determining category information of the video to be classified from the preset categories based on the first probability values and the second probability values.

4. The method of claim 3, wherein, The determining of the category information of the video to be classified from the preset categories based on the first probability values and the second probability values comprises: for each preset category, performing the following operations: taking a smaller value of the first probability value and the second probability value corresponding to the preset category or a mean value of the first probability value and the second probability value as a prediction probability value of the video to be classified belonging to the preset category; and taking a preset category corresponding to a prediction probability value with a specified order in the sorted prediction probability values as the category information of the video to be classified.

5. The method of claim 1, wherein, After the semantic weights of the target video frames are obtained from the context features based on the attention mechanism, the frame selection weights can also be determined in the following manner: determining visual weights of the target video frames based on category-related features of the video to be classified and the visual features; and taking a sum of the semantic weight and the visual weight of the same target video frame as the corresponding frame selection weight.

6. The method of claim 1, wherein, The feature fusion based on the semantic features and the visual features to obtain context features matched with the semantic feature comprises: The semantic features are weighted and averaged and transposed to obtain auxiliary semantic features of the video to be classified; The visual features and the auxiliary semantic features are fused to obtain the context features.

7. The method of claim 5, wherein, The category-related features include category-related features corresponding to the target video frames respectively, which are determined based on the self-learning network of the visual features of the target video frames respectively; The visual weights of the target video frames are determined based on the category-related features and the visual features respectively, including: The category-related features corresponding to the target video frames are weighted and averaged to obtain reference features; The reference features are respectively spliced with the visual features, and linear mapping is performed on the spliced results to obtain the visual weights of the target video frames.

8. The method of claim 1, wherein, The visual enhancement features of the video to be classified are determined based on the visual features of the at least one key video frame, including: The corresponding weight values of the target video frames in the video frames in the specified order range are selected as key video frames based on the frame selection weights; The visual enhancement features of the video to be classified are obtained by fusing the visual features and the corresponding weight values of the key video frames.

9. The method according to any one of claims 1 to 8, wherein The semantic features and the visual features are obtained by inputting the target video frames and the target description information into the trained target video classification model respectively based on the feature extraction network in the target video classification model; The visual enhancement features are obtained by the key video frame selection network in the target video classification model based on the frame selection weights of the target video frames determined by the semantic features and the visual features; The multi-modal interaction features are obtained by the multi-modal interaction network in the target video classification model based on the feature interaction of the visual enhancement features and the semantic features.

10. The method of claim 9, wherein, The target video classification model is obtained by the following method: The video classification model to be trained is trained by cyclic iteration based on the training samples in the training sample set to obtain the target video classification model; each training sample includes: each first sample video frame of a sample video, first sample description information and a real category label of the sample video; wherein, each cyclic iteration training performs the following steps: The first sample video frames and the first sample description information are input into the video classification model to obtain the first sample visual features of the first sample video frames and the first sample semantic features of the first sample description information respectively; The sample multi-modal interaction features are obtained based on the first sample semantic features and the first sample visual features to determine the corresponding sample category information; The video classification model is adjusted in parameters based on the target loss function constructed based on the sample category information, the real category label and the sample multi-modal interaction features.

11. The method of claim 10, wherein, The sample multi-modal interaction features are obtained by the following method: determine sample frame selection weights of the first sample video frames based on the first sample semantic features and the first sample visual features; determine at least one sample key video frame in the sample video based on the sample frame selection weights, and determine sample visual enhanced features of the sample video based on first sample visual features of the at least one sample key video frame; perform feature interaction based on the sample visual enhanced features and the first sample semantic features to obtain the sample multi-modal interaction features.

12. The method of claim 11, wherein, The determining of the sample frame selection weights of the first sample video frames based on the first sample semantic features and the first sample visual features comprises: perform feature fusion based on the first sample semantic features and the first sample visual features to obtain sample context features matched with the first sample semantic features; obtain sample semantic weights of the first sample video frames from the sample context features based on an attention mechanism; obtain second sample visual features of second sample video frames in a reference video, and determine first sample visual weights of the first sample video frames based on the second sample visual features and the first sample visual features; the reference video has the same real category label as the sample video; sum the sample semantic weights and the first sample visual weights of the same first sample video frame to obtain corresponding sample frame selection weights.

13. The method of claim 12, wherein, The determining of the first sample visual weights of the first sample video frames based on the second sample visual features and the first sample visual features comprises: perform weighted average on the second sample visual features to obtain sample reference features; perform linear mapping processing on the sample reference features and the first sample visual features to obtain the first sample visual weights of the first sample video frames.

14. The method of claim 10, wherein, Before the inputting of the first sample video frames and the first sample description information into the video classification model, the method further comprises: obtaining a reference video matched with the sample video, the reference video having the same real category label as the sample video; The inputting of the first sample video frames and the first sample description information into the video classification model to obtain the first sample visual features of the first sample video frames and the first sample semantic features of the first sample description information further comprises: inputting second sample video frames of the reference video and corresponding second sample description information into the video classification model to obtain second sample visual features of the second sample video frames and second sample semantic features of the second sample description information.

15. The method of claim 10, wherein, The target loss function is constructed in the following manner: constructing a classification prediction loss function based on a difference between the sample category information and the real category label; determining a heterogeneous feature alignment loss function based on a difference between a target dimension in a sample visual representation feature and a corresponding dimension in a sample semantic representation feature; the sample visual representation feature and the sample semantic representation feature are extracted from the sample multi-modal interaction features through feature migration; construct the target loss function based on the classification prediction loss function and the heterogeneous feature alignment loss function.

16. The method of claim 15, wherein, The sample category information includes first sample category information obtained based on the sample visual representation feature and second sample category information obtained based on the sample semantic representation feature. The classification prediction loss function is constructed based on the difference between the sample category information and the real category label, including: a visual classification loss function is determined based on the difference between the first sample category information and the real category label, and a semantic classification loss function is determined based on the difference between the second sample category information and the real category label; The classification prediction loss function is constructed based on the visual classification loss function and the semantic classification loss function.

17. The method of claim 15, wherein, The target loss function is constructed based on the classification prediction loss function and the heterogeneous feature alignment loss function, including: a constraint loss function is determined based on the difference between the sample video and the reference video matched with the sample video; the reference video has the same real category label as the sample video; The target loss function is obtained by weighting processing based on the classification prediction loss function, the heterogeneous feature alignment loss function, and the constraint loss function.

18. The method of claim 17, wherein, The constraint loss function includes at least one of a semantic constraint loss function and a visual constraint loss function; the constraint loss function is determined in the following manner: a semantic constraint loss function is determined based on the difference between the first sample semantic feature and the second sample semantic feature; the second sample semantic feature is determined based on the second sample description information of the reference video; a visual constraint loss function is determined based on the difference between the first sample visual feature corresponding to the first sample constraint video frame in the sample video and the second sample visual feature corresponding to the second sample constraint video frame in the reference video.

19. The method of claim 18, wherein, The first sample constraint video frame and the second sample constraint video frame are determined in the following manner: at least one first sample constraint video frame is determined in the sample video based on each first sample visual weight; second sample visual weights of each second sample video frame are determined based on the weighted average result and each second sample visual feature; at least one second sample constraint video frame is determined in the reference video based on each second sample visual weight.

20. A video classification apparatus, comprising: It includes: a feature extraction unit configured to obtain respective visual features of each target video frame in a to-be-classified video and a semantic feature corresponding to target description information of the to-be-classified video; a weight determination unit configured to perform feature fusion based on the semantic feature and each visual feature to obtain a context feature matched with the semantic feature; a semantic weight of each target video frame is obtained from the context feature based on an attention mechanism; and the semantic weight of each target video frame is taken as a corresponding frame selection weight. a visual enhancement unit configured to determine at least one key video frame in the video to be classified based on a selection weight of each frame, and determine a visual enhancement feature of the video to be classified based on respective visual features of the at least one key video frame; a classification unit configured to perform feature interaction based on the visual enhancement feature and the semantic feature, obtain a corresponding multi-modal interaction feature, and determine class information of the video to be classified based on the multi-modal interaction feature.

21. An electronic device, comprising: The computer program is stored in the computer readable storage medium, and when the processor of the electronic device reads the computer program from the computer readable storage medium, the processor executes the computer program, so that the electronic device executes the steps of the method in any one of claims 1-19.

22. A computer-readable storage medium, characterized in that, The computer program is stored in the computer readable storage medium, and when the processor of the electronic device reads the computer program from the computer readable storage medium, the processor executes the computer program, so that the electronic device executes the steps of the method in any one of claims 1-19.

23. A computer program product, characterised in that, The computer program is stored in the computer readable storage medium, and when the processor of the electronic device reads the computer program from the computer readable storage medium, the processor executes the computer program, so that the electronic device executes the steps of the method in any one of claims 1-19.

Images