A short video multi-label classification method based on double-layer prompt modeling

Abstract

This invention provides a short video multi-label classification method based on two-layer cue modeling, belonging to the field of multimodal video understanding and multi-label classification technology. This method improves classification performance through collaborative semantic modeling at both the label level and the label group level. At the label level, attribute cue words are used to structurally model the visual and audio features of each label, generating attribute description text, which is then retrieved and fused with video and audio features at the attribute level to form an initial label representation. At the label group level, relationship cue words are used to model the relationships between labels, generating relationship centers and updating relationship constraints based on label roles. The complementary relationship modeling at the label and label group levels solves the problem of the separation between label semantic modeling and inter-label relationship modeling, mitigating the challenge of decreased discriminative power caused by relationship enhancement. By performing multi-label classification based on the updated label representation, the accuracy, stability, and interpretability of multi-label recognition in multimodal videos can be significantly improved.

Description

Technical Field

[0001] This invention relates to the field of multimodal video understanding and multi-label classification technology. Specifically, it relates to a hierarchical modeling method based on cue words for multimodal video multi-label classification. This method combines visual, audio, and inter-label relationship information, and improves classification performance through hierarchical cue modeling. It is particularly suitable for multi-label video analysis scenarios such as video understanding, behavior recognition, intelligent surveillance, and media recommendation. Background Technology

[0002] With the continuous development of computer vision and multimodal learning technologies, multimodal video classification has demonstrated significant value in various practical applications such as intelligent surveillance, autonomous driving, and media recommendation.

[0003] However, existing multimodal video multi-label classification methods still have significant shortcomings. On the one hand, existing methods typically focus on learning unified representations from global visual and audio features, lacking structured modeling of the semantics of individual labels, making it difficult to provide stable and interpretable intra-class discrimination criteria for each label. On the other hand, the modeling of relationships between labels is usually separated from the modeling of the semantics of the labels themselves, lacking a collaborative mechanism, and making it difficult to simultaneously support the effective discrimination of label semantics and inter-label relationships.

[0004] Especially in multi-label scenarios, labels often have relationships such as conditional dependencies, functional cooperation, and temporal correlations. When introducing label relationships, existing methods often update label representations through feature propagation and representation fusion. However, due to the lack of sufficient modeling of single-label content, relationship information can easily interfere with label discrimination, causing label representations to tend to be similar, thereby weakening discriminability and causing the problem of "relationship enhancement leading to dilution of discriminability".

[0005] Therefore, there is an urgent need for a hierarchical modeling method based on prompt words, which can perform structured modeling of single-label semantics at the label level and model the relationships between labels at the label group level, so that label semantic modeling and inter-label relationship modeling can work together to support sample discrimination, thereby effectively solving the problems of semantic fragmentation and relationship enhancement weakening discriminability in existing multi-label methods. Summary of the Invention

[0006] To address the technical problems in existing short video multi-tag classification methods, such as insufficient semantic modeling of tags themselves, inadequate coordination between visual and audio modalities, disconnect between tag relationship modeling and tag discrimination processes, and the tendency for tag representations to converge and discrimination ability to decline during relationship enhancement, this invention proposes a short video multi-tag classification method based on two-layer cue modeling. This method aims to construct a multimodal multi-tag classification scheme that balances the semantic expression within tags with the constraints of relationships between tags. By constructing two layers—tag-level attribute cue modeling and tag group-level relationship cue modeling—this method improves tag interpretability while enhancing the reasoning ability between tag relationships. It is particularly suitable for applications such as short video understanding, behavior recognition, content moderation, intelligent monitoring, media recommendation, and multimodal content analysis.

[0007] In a first aspect, the present invention provides a short video multi-label classification method based on two-layer cue modeling, comprising the following steps: S1. For each category label, construct visual attribute cue words and audio attribute cue words, and for label groups, construct relationship cue words. Based on the preset template, generate label-level attribute description text and label group-level candidate relationship description text. S2, obtain the input short video and its corresponding audio, and extract the intermediate layer visual features, intermediate layer audio features, and final layer visual features and final layer audio features; S3 encodes the tag-level attribute description text and maps it to the attribute query vector. It then performs attribute-level retrieval and fusion on the intermediate layer visual features and intermediate layer audio features to generate the initial representation of each tag. S4, the candidate relation description text at the label group level is encoded and mapped to the relation semantic space. Combined with the global semantic representation of the input sample, the candidate relations are selected based on relevance and relation centers are constructed. S5. Based on the role information of the tag in each tag group, construct the source role message and the target role message, update the relation constraints of the tag initialization representation, and obtain the updated tag representation; S6 takes the updated label representation as input to the multi-label classification head, outputs the prediction results of each label, and performs joint optimization through classification loss and attribute diversity loss during the training phase.

[0008] Secondly, the present invention provides a short video multi-label classification system based on two-layer cue modeling, characterized in that it includes: The attribute cue and description text generation module is configured to generate visual attribute cue and audio attribute cue for each tag, and generate a corresponding set of attribute description text based on a preset template; The relation prompt and candidate relation description generation module is configured to generate relation prompt templates for tag groups and generate a set of candidate relation description texts; The multimodal feature extraction module is configured to extract intermediate layer visual features, intermediate layer audio features, and final layer visual features and final layer audio features from the input short video and its audio. The attribute-level evidence retrieval and fusion module is configured to encode attribute description text into attribute query vectors, perform attribute-level retrieval on intermediate-layer visual features and intermediate-layer audio features, and fuse visual and audio attribute responses to generate a tag initialization representation. The relation description selection and relation center construction module is configured to encode, map, and filter candidate relation description texts, and construct source-side and target-side relation centers by combining the global semantic representation of the input samples. The role-aware relationship update module is configured to construct relationship messages based on the role information of the tag in the tag group and update the tag representation; The multi-label classification module is configured to output the prediction results of each label based on the updated label representation, and to perform joint optimization of classification loss and attribute diversity loss during the training phase.

[0009] This invention has the following technical advantages: (1) Tag-level semantic modeling. By constructing visual attribute cue words and audio attribute cue words for each tag and generating a set of attribute description text, fine-grained evidence related to the tag can be retrieved in the visual and audio modalities, thereby forming a tag initialization representation with intra-class discrimination criteria, improving the stability and distinguishability of the tag representation; (2) Attribute-level multimodal evidence retrieval. Attribute-level cross-attention retrieval is performed on intermediate-layer visual features and intermediate-layer audio features, which can accurately locate local evidence corresponding to the label attributes, improve the utilization rate of multimodal information, and reduce intermodal redundancy interference; (3) Label group-level relation semantic modeling. A candidate relation description text set is generated by relation prompt words, and the relation constraints of the label representation are updated by combining the source-end directional relation center, the target-end directional relation center and the role messages within the label group, thereby providing inter-class relation constraints for the label representation and improving the final multi-label classification accuracy; (4) Attribute filtering and diversity constraint mechanism. Attribute responses are filtered and weighted through geometric consistency and global semantic consistency, and attribute diversity constraints are used to avoid multiple attributes under the same label from focusing on the same evidence region, thereby maintaining the division of labor in attribute queries and enhancing the ability to express fine-grained features; (5) Two-layer modular system design. The method can be implemented through the collaborative implementation of a tag-level prompt modeling module and a tag group-level prompt modeling module. The system structure is clear and easy to deploy, and it is suitable for short video understanding, multimodal content analysis and related application scenarios. Attached Figure Description

[0010] Figure 1 This is a flowchart of the method of the present invention. Detailed Implementation

[0011] To make the objectives, technical solutions, and beneficial effects of this invention clearer, the invention will be further described below, but the scope of protection of this invention is not limited thereto.

[0012] Example 1: This embodiment provides a multi-label classification method for short videos based on two-layer cue modeling, suitable for multi-label recognition of short video samples containing both visual and audio information. This method aims to improve classification accuracy, stability, and interpretability by fully modeling the internal semantics of the labels and further incorporating constraints between the relationships between the labels. The overall process includes the following steps: S1: Construct a two-layer prompt word system; S2: Extract multimodal features for two-layer modeling; S3: Perform tag-level attribute retrieval and generate tag initialization representations; S4: Perform tag group-level relationship modeling and generate relationship centers; S5: Perform tag group-level relationship update; S6: Perform multi-tag classification based on the updated tag representations.

[0013] Furthermore, the specific method of S1 is as follows: S11: For each category label, construct visual attribute cue words and audio attribute cue words respectively.

[0014] Among them, visual attribute cues are used to describe the appearance, actions, state and scene features of the target in the video, while audio attribute cues are used to describe the timbre, rhythm, intensity and event acoustic features of the target in the audio.

[0015] S12: Generate a set of visual attribute description text and a set of audio attribute description text using preset templates and a large language model.

[0016] Each tag can correspond to multiple attribute description texts, which are used to characterize the multimodal semantics of the tag from multiple dimensions.

[0017] S13: Build relation prompt word templates for semantically related tag groups.

[0018] The relation prompt word template is used to describe the conditional dependencies, functional cooperation, temporal associations, co-occurrence constraints, or role division relationships between tags within a tag group, and a set of candidate relation description texts is generated based on the relation prompt word template.

[0019] Furthermore, the specific method of S2 is as follows: S21: Obtain short video samples to be classified, extract video frames according to a preset sampling strategy, and input them into the visual feature extraction network to obtain intermediate layer visual features and final layer visual features.

[0020] S22: Perform time-frequency transformation on the audio corresponding to the short video and input it into the audio feature extraction network to obtain the intermediate layer audio features and the final layer audio features.

[0021] S23: Among them, the intermediate layer visual features and intermediate layer audio features contain rich local spatiotemporal details and temporal information, which are used for semantic retrieval within the label; the final layer visual features and final layer audio features contain higher-level semantic information, which are used to represent the global semantic information of the input sample and support the learning of relationships between labels. Furthermore, the final layer visual features and final layer audio features can be globally pooled or globally encoded to obtain the global semantic representation g of the input sample.

[0022] Furthermore, the specific method of S3 is as follows: S31: Encode the attribute description text obtained in step S1 into a text embedding and map it into an attribute query vector Q.

[0023] S32: Using the attribute query vector Q, perform attribute-level cross-attention retrieval on the intermediate layer visual features and intermediate layer audio features respectively to obtain visual attribute responses and audio attribute responses, thereby locating the local evidence most relevant to the attribute in the visual modality and audio modality.

[0024] S33: Fuse the visual attribute response and audio attribute response under the same label to obtain an attribute-level fused representation.

[0025] S34: In order to improve the quality of attribute responses, the responses of different attributes under the same label are scored in two ways. On the one hand, the geometric consistency score is calculated based on the consistency between different attributes. On the other hand, the global semantic score is calculated based on the degree of matching between the attribute response and the global semantic representation of the input sample.

[0026] S35: Calculate the attribute weights based on the above scoring results, and perform a weighted summation of the fusion representations of each attribute to generate the initial representation of the label, which serves as the input for subsequent label group-level relationship modeling.

[0027] Furthermore, the specific method of S4 is as follows: S41: Encode the candidate relation description text generated in step S1 to obtain the relation text embedding.

[0028] S42: Map the relational text embeddings to the source-side relational semantic space and the target-side relational semantic space respectively to distinguish the different roles of the tags in the relation.

[0029] S43: Combining the global semantic representation g of the input sample, perform relevance selection on each candidate relation description, select the relation description most relevant to the current sample, and generate source-side directional relation centers respectively. Center of Relationship with Target Direction These relationship centers provide contextual information for subsequent tag relationship updates.

[0030] Furthermore, the specific method of S5 is as follows: S51: Based on the source-end direction relationship center and the target-end direction relationship center obtained in step S4, calculate the role weight of label n in label group g, and construct the source-end role message and the target-end role message.

[0031] S52: Aggregate the relationship messages of a tag in each tag group to obtain an enhanced relationship representation of the tag. If a tag belongs to multiple tag groups, then aggregate the relationship messages from multiple tag groups.

[0032] S53: Update the initial label representation using residuals after aggregating the relational messages to obtain a label representation containing the relational constraints between labels. Through this step, the conditional dependencies and functional relationships between labels can be dynamically modeled.

[0033] Furthermore, the specific method of S6 is as follows: S61: Input the updated label representation obtained in step S5 into the multi-label classification head, output the predicted score of each label, and obtain the final multi-label classification result according to the set threshold.

[0034] S62: During the training phase, the model is optimized by constructing a total loss function that includes the final multi-label classification loss and the attribute diversity loss. The final multi-label classification loss is used to constrain the classification ability of the updated label representation, and the attribute diversity loss is used to maintain the diversity of interest regions among different attributes under the same label, thereby enhancing the interpretability of the label representation and the stability of the classification results.

[0035] S63: During the inference phase, the short video to be classified is input into the trained model, and the tag-level attribute retrieval, tag group-level relationship modeling, and multi-tag classification process are executed in sequence, finally outputting the multi-tag classification result.

[0036] Example 2: To further illustrate the short video multi-label classification method based on two-layer cue modeling proposed in this invention, the steps described in Embodiment 1 are explained in detail below with reference to mathematical expressions and accompanying drawings. The following formulas and symbol definitions are consistent with the original technical approach in this application specification.

[0037] Step 201: For the input short video sample obtained in step S2, extract intermediate layer visual features through the visual feature extraction network and the audio feature extraction network respectively. Intermediate layer audio features Final layer visual features and final layer audio features The intermediate layer features are used for attribute-level retrieval, and the final layer features are used for relation learning. Furthermore, the final layer visual and audio features are globally pooled or globally encoded to obtain a global semantic representation g of the input sample, which is used for subsequent attribute filtering and relation modeling.

[0038] Step 202: For the attribute query vector Q obtained in step S1, in the intermediate layer visual features and intermediate layer audio features Perform attribute-level cross-attention retrieval to obtain visual attribute responses. and audio property response The formula for calculating cross-attention is as follows: in, For query vector, For key vectors, It is a scaling factor. Used to normalize attention weights, and These represent the intermediate layer features of the visual and audio modalities, respectively. This step is used to locate the local evidence most relevant to the attribute from the visual and audio modalities.

[0039] Step 203: Fuse the visual attribute responses and audio attribute responses under the same tag to obtain an attribute-level fused representation. Where n represents the nth tag, and k represents the kth attribute under that tag. The attribute response fusion method is as follows: in, The matrix is ​​a learnable fusion matrix, and [;] denotes a concatenation operation. To improve the quality of attribute responses, responses to different attributes under the same label are double-scored. First, a geometric consistency score is calculated based on the consistency between different attributes. The calculation formula is as follows: in, Represents the similarity function. This indicates the number of attributes corresponding to the tag. is the scaling factor. Further, a global semantic score is calculated based on the degree of matching between the attribute response and the global semantic representation g of the input sample. The calculation formula is as follows: Attribute weights are calculated based on the two types of scoring mentioned above. The initial representation of the tag is obtained by weighted summation of the fused representations of each attribute. : This step yields a stable and distinguishable initial representation for each label, which serves as input for subsequent label group-level relationship modeling.

[0040] Step 204: For the candidate relation description text generated in Step S1, first encode it to obtain relation text embeddings; then map the relation text embeddings to the source-side relation semantic space and the target-side relation semantic space respectively to distinguish the different roles of the labels in the relation. Combined with the global semantic representation g of the input sample, select the relevance of each candidate relation description, select the relation description most relevant to the current sample, and generate source-side relation centers respectively. Center of Relationship with Target Direction These relationship centers provide contextual information for subsequent tag relationship updates.

[0041] Step 205: In this step, based on the source-end direction relationship center and the target-end direction relationship center obtained in step 204, calculate the position of label n in the label group. The role weights in the data, and the relationship messages of the tags are constructed. It is represented as follows: in, and Representing tags respectively In the tag group The source-side role weight and the target-side role weight in the data. This represents element-wise multiplication. Then, the relationship messages of multiple tag groups to which a tag belongs are aggregated, and the tag representation is updated using residuals. : in, Indicates label All tag groups in which it is located This is to update the coefficients. Through this step, the conditional dependencies and functional relationships between labels are dynamically modeled, and the label representation is updated to include the constraints of the relationships between labels.

[0042] Step 206: Input the updated label representation obtained in Step 205 into the multi-label classification head, output the predicted score for each label, and obtain the final multi-label classification result based on a set threshold. During the training phase, the model is optimized by constructing a total loss function that includes the final multi-label classification loss and attribute diversity loss. Final multi-label classification loss. The calculation formula is: Final multi-label classification loss The calculation formula is: in For the final output logits, This is for multi-dimensional labeling of tags.

[0043] Attribute diversity loss The calculation formula is: in, This represents attribute attention after normalization.

[0044] Final joint loss function for: in, and These are hyperparameters. During the inference phase, the short video to be classified is input into the trained model, which sequentially performs tag-level attribute retrieval, tag group-level relationship modeling, and multi-tag classification, ultimately outputting the multi-tag classification result.

[0045] Example 3: This embodiment provides a short video multi-label classification system based on two-layer prompt modeling, applicable to multi-label recognition tasks in application scenarios such as short video understanding, multimodal content analysis, behavior recognition, and intelligent monitoring. The system includes at least one processor and a memory communicatively connected to it. The memory stores computer program instructions executable by the processor to implement the following functional modules: A tag-level attribute prompt and description text generation module: This module generates visual and audio attribute prompts for each tag, and generates visual and audio attribute description texts based on templates and a large language model, providing textual priors for tag-level semantic modeling. A tag group-level relationship prompt and candidate relationship description generation module: This module generates relationship prompt templates for tag groups and generates a set of candidate relationship description texts to characterize the dependencies, co-occurrence relationships, temporal relationships, and role division relationships between tags within the tag group. A multimodal feature extraction module: This module extracts intermediate and final layer visual and audio features from the input short video and its audio, and generates a global semantic representation of the input sample to support attribute retrieval and relationship learning, respectively. **Attribute-Level Evidence Retrieval and Fusion Module:** This module performs attribute-level cross-attention retrieval in both visual and audio modalities based on attribute query vectors, and fuses visual and audio attribute responses to generate an initial label representation. **Relationship Description Selection and Relationship Center Construction Module:** This module encodes, maps, and filters candidate relationship description texts, combining global semantic representation to generate source-side and target-side relationship centers. **Role-Aware Relationship Update Module:** This module constructs relationship messages based on the role information of labels within a label group and updates the label representation through aggregation and residual updates. **Multi-Label Classification Training and Inference Module:** This module trains the model and outputs multi-label predictions based on the updated label representations.

[0046] The above description is merely a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A multi-label classification method for short videos based on two-layer cue modeling, characterized in that, Includes the following steps: S1. Design visual attribute cue words and audio attribute cue words for each tag, construct relationship cue words for each tag group, and generate corresponding attribute description text sets and candidate relationship description text sets based on preset templates and large language models respectively; S2 extracts intermediate-layer visual and auditory features and final-layer visual and auditory features from the input video and audio. The intermediate-layer visual and auditory features are used for semantic retrieval within tags, and the final-layer visual and auditory features are used for learning relationships between tags. S3 encodes the tag-level attribute description text generated in S1 into a text embedding and maps it into an attribute query vector. It then performs attribute retrieval on the visual and audio features extracted in S2, filters and fuses them, and generates the initial tag representation. S4 encodes the label group-level candidate relation description text generated in S1 into a text embedding and maps it to the source-side directional relation semantic space and the target-side directional relation semantic space; and constructs the source-side directional relation center and the target-side directional relation center based on the global semantic representation; S5, based on the relationship description selected in S4, construct the source role message and target role message of the tag, and update the tag representation according to the role weight; S6 performs multi-label classification based on the updated label representation, including constructing the total loss function for model training during the training phase and generating prediction results through forward computation during the inference phase.

2. The short video multi-label classification method based on two-layer cue modeling according to claim 1, characterized in that, In S1, visual attribute description text and audio attribute description text are generated independently. Multiple attribute description texts are generated for each label, preferably about 15. The text generation is completed through a large language model to provide specific attribute representations for subsequent multimodal feature retrieval.

3. The short video multi-label classification method based on two-layer cue modeling according to claim 1, characterized in that, In S2, the input short video is sampled into a video frame sequence and input into a pre-trained visual feature extraction network to obtain intermediate layer visual features and final layer visual features. The corresponding audio signal is converted into a time-frequency representation and input into an audio feature extraction network to obtain intermediate-layer and final-layer audio features. The intermediate-layer visual and audio features provide the fine-grained multimodal base representation required for attribute-level cross-attention retrieval, while the final-layer visual and audio features characterize the global semantic information of the input samples to support the learning of relationships between labels.

4. The short video multi-label classification method based on two-layer cue modeling according to claim 1, characterized in that, In S3, the label-level attribute description text is encoded into a text embedding and then mapped to an attribute query vector. Cross-attention calculation is performed with the intermediate layer visual features and intermediate layer audio features respectively to obtain the corresponding attribute response. Then, based on the consistency between the attribute response and the global semantics of the input sample, as well as the geometric consistency between the attribute responses, the attribute responses are filtered and weighted to retain reliable attributes and fuse multiple attribute responses to form the initial label representation.

5. The short video multi-label classification method based on two-layer cue modeling according to claim 1, characterized in that, In S4, the candidate relation description text set is generated by combining relation prompt word templates with a large language model. The candidate relation description texts are mapped to the source-side relation semantic space and the target-side relation semantic space, respectively. Based on the global semantic representation of the input sample, the candidate relation descriptions are relevance scored, selected, and aggregated to generate the source-side relation center and the target-side relation center, providing sample-related relation context information for subsequent label representation updates.

6. The short video multi-label classification method based on two-layer cue modeling according to claim 1, characterized in that, In S5, the role of a tag in each tag group is divided into source role and target role, and corresponding role messages are constructed based on the source direction relationship center and the target direction relationship center, respectively. The role messages are weighted according to the role weight of the tag within the group, and the role messages of multiple tag groups to which the tag belongs are aggregated to update the tag representation, thereby realizing tag group-level relationship constraint modeling.

7. The short video multi-label classification method based on two-layer cue modeling according to claim 1, characterized in that, In S6, during the training phase, the model is jointly optimized using a total loss function consisting of the final multi-label classification loss and the attribute diversity loss. The final multi-label classification loss constrains the classification results of the label representations after the label group-level relation update, while the attribute diversity loss constrains the attention distribution of different attributes under the same label to different evidence regions in the input sample, so as to avoid multiple attribute responses concentrating on the same evidence region and maintain the specialization of attribute queries. During the inference phase, the short video to be classified is input into the trained model, and attribute-level retrieval, label initialization representation construction, relation description selection, label representation update, and multi-label classification are executed in sequence, outputting the prediction results corresponding to each label.

8. A short video multi-label classification system based on two-layer cue modeling, characterized in that, This system integrates multiple functional modules to implement a multi-label classification process for short videos. It is applicable to short video understanding, multimodal content analysis, and related scenarios. Specifically, it includes: a module for generating tag-level attribute prompts and descriptive text, used to generate visual and audio attribute descriptions; a multimodal feature extraction module, used to extract visual and audio features from intermediate and final layers; an attribute-level evidence retrieval and fusion module, used to perform attribute-level cross-attention retrieval and generate initial tag representations; a tag group-level relationship description generation and selection module, used to generate candidate relationship descriptions and construct relationship centers; a role-aware relationship update module, used to update tag representations based on role messages; and a multi-label classification training and inference module, used to output multi-label prediction results.

Images