Public opinion video tag aggregation method and system based on artificial intelligence

Abstract

The invention provides a public opinion video tag aggregation method and system based on artificial intelligence, and relates to the technical field of artificial intelligence. Comprising the following steps: acquiring pictures and text information in a short video, and performing semantic alignment; different large language models are adopted to generate preliminary labels for the pictures and the text information after semantic alignment; clustering the pictures and the texts after semantic alignment to obtain clusters; calculating the labeling probability of each primary label type in the current cluster by each large language model, and selecting the primary label with the highest probability sum as a clustering label of the current cluster; calculating the reliability weight of each large language model in the current cluster based on the clustering label of the current cluster; and based on the reliability weight, calculating the weighted support degree of all the large language models to different preliminary label types of each piece of data in the current cluster, calculating the weighted label of the current data, and further determining a final label. According to the method, the condition of few labels or no labels can be effectively processed, and the manual workload is greatly reduced.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence technology, and in particular relates to a method and system for aggregating public opinion video tags based on artificial intelligence. Background Technology

[0002] In recent years, short videos have become the main battleground for the fermentation of online public opinion. A short video of a few seconds to a dozen seconds can often go viral across the entire internet within hours. This has led not only to an explosive growth in the amount of data, but also to serious challenges posed to public opinion monitoring. To quickly grasp the core key points of an event from a large number of public opinion videos, it is necessary to use tag aggregation methods to integrate information and obtain semantically accurate tags to replace complex content for decision-making.

[0003] Relying on manual tagging is not only costly but also highly susceptible to subjective bias, leading to a redundant and chaotic tagging system. Traditional tag aggregation methods rely solely on the statistical patterns of the tags themselves, neglecting the video's own features and structure, and potentially ignoring information from highly reliable sources. Furthermore, existing AI algorithms for tag aggregation largely depend on single-modal information; the incompleteness of this information and noise interference severely limit the practical effectiveness of deep learning. Summary of the Invention

[0004] To overcome the shortcomings of the existing technology, the present invention provides an artificial intelligence-based method and system for aggregating public opinion video tags, which can effectively handle situations with few or no tags and significantly reduce manual workload.

[0005] To achieve the above objectives, one or more embodiments of the present invention provide the following technical solutions: The first aspect of this invention provides a method for aggregating public opinion video tags based on artificial intelligence.

[0006] The AI-based method for aggregating public opinion video tags includes the following steps: Extract image and text information from short videos and perform semantic alignment; Different large language models were used to generate preliminary labels for semantically aligned images and text information; Cluster the semantically aligned images and text to obtain clusters; Calculate the labeling probability of each large language model for each initial label type in the current cluster, sum the labeling probabilities of all large language models for each initial label type, and select the initial label with the highest probability sum as the clustering label for the current cluster. Based on the clustering label of the current cluster, calculate the reliability weight of each large language model within the current cluster; Based on reliability weights, calculate the weighted support of all large language models for each data point in the current cluster for different initial label categories, and select the initial label corresponding to the maximum weighted support as the weighted label for the current data. When the weighted label of the current data is greater than the preset threshold and the weighted label is consistent with the clustering label of the current cluster, the weighted label of the current data is determined to be the final label.

[0007] The second aspect of this invention provides an artificial intelligence-based public opinion video tag aggregation system.

[0008] An AI-based public opinion video tagging aggregation system includes: The semantic alignment module is configured to: acquire image and text information from short videos and perform semantic alignment; The initial label generation module is configured to generate initial labels for semantically aligned images and text information using different large language models. The clustering module is configured to cluster semantically aligned images and text to obtain clusters. The cluster label determination module is configured to: calculate the labeling probability of each large language model for each preliminary label type in the current cluster, sum the labeling probabilities of all large language models for each preliminary label type, and select the preliminary label with the highest probability sum as the clustering label of the current cluster. The reliability weight calculation module is configured to calculate the reliability weight of each large language model within the current cluster based on the clustering label of the current cluster. The data weighted label calculation module is configured to: calculate the weighted support of all large language models for each data in the current cluster with different initial label types based on reliability weights, and select the initial label corresponding to the maximum weighted support as the weighted label of the current data; The data final label determination module is configured to determine the final label when the weighted label of the current data is greater than a preset threshold and the weighted label is consistent with the clustering label of the current cluster. A third aspect of the present invention provides a computer-readable storage medium having a program stored thereon, which, when executed by a processor, implements the steps of the artificial intelligence-based public opinion video tag aggregation method described in the first aspect of the present invention.

[0009] The fourth aspect of the present invention provides an electronic device, including a memory, a processor, and a program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the artificial intelligence-based public opinion video tag aggregation method as described in the first aspect of the present invention.

[0010] The above one or more technical solutions have the following beneficial effects: This invention provides an artificial intelligence-based method and system for aggregating public opinion video tags. First, cluster tags are calculated based on statistical laws. Then, weighted tags are obtained through weighted calculation using a large language model. Finally, fixed tags are determined by comparing the weighted tags with the cluster tags of their respective clusters. By using a combination of "large model tags + multimodal clustering + weighted aggregation," the method considers both the statistical laws of the model-annotated tags and the characteristics and structure of the short video content itself. This approach not only effectively handles situations with few or no tags but also more accurately identifies "samples that truly require human intervention," significantly reducing the amount of manual work.

[0011] By capturing multi-source information from public opinion video data using a cyclic multimodal autoencoder, this method can better utilize the characteristics and structure of the video content itself, emphasizing information from highly reliable sources. This approach balances manual costs with aggregation quality, achieving low-cost, high-quality, and scalable tag aggregation in public opinion analysis scenarios.

[0012] Advantages of additional aspects of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0013] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0014] Figure 1 This is a flowchart of the method in Example 1.

[0015] Figure 2 This is a flowchart of the data processing of the multimodal autoencoder in Example 1. Detailed Implementation

[0016] It should be noted that the following detailed descriptions are exemplary and intended to provide further illustration of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0017] It should be noted that the terminology used herein is for the purpose of describing particular implementations only and is not intended to limit the exemplary implementations of the present invention.

[0018] Where there is no conflict, the embodiments and features in the embodiments of the present invention can be combined with each other.

[0019] Example 1 This invention provides an artificial intelligence-based method for aggregating public opinion video tags, aiming to solve the following technical problems: (1) There are a large number of short videos on public opinion, and manual annotation is costly, inefficient, and subject to subjective bias. (2) Traditional tag aggregation methods (such as majority voting) are based only on the statistical regularity of the tags themselves, without considering the characteristics and structure of the short video content itself, and may ignore information from highly reliable sources.

[0020] To address the two technical challenges mentioned above, this embodiment integrates multimodal information for clustering and introduces different large models to generate initial tags for short videos, guiding the clustering process. Tag aggregation is then performed based on this initial model. This approach effectively reduces manual labor costs by leveraging statistical patterns and sample similarity, while targeted manual review ensures the quality of key samples. This "large model generation - multimodal clustering - manual verification" method provides a powerful solution to the challenge of aggregating tags for public opinion videos. Furthermore, compared to single data features, multimodal data offers richer information dimensions.

[0021] like Figure 1 As shown, the AI-based method for aggregating public opinion video tags includes the following steps: Extract image and text information from short videos and perform semantic alignment; Different large language models were used to generate preliminary labels for semantically aligned images and text information; Cluster the semantically aligned images and text to obtain clusters; Calculate the labeling probability of each large language model for each initial label type in the current cluster, sum the labeling probabilities of all large language models for each initial label type, and select the initial label with the highest probability sum as the clustering label for the current cluster. Based on the clustering label of the current cluster, calculate the reliability weight of each large language model within the current cluster; Based on reliability weights, calculate the weighted support of all large language models for each data point in the current cluster for different initial label categories, and select the initial label corresponding to the maximum weighted support as the weighted label for the current data. When the weighted label of the current data is greater than the preset threshold and the weighted label is consistent with the clustering label of the current cluster, the weighted label of the current data is determined to be the final label.

[0022] The technical solution of this embodiment will be explained in detail below.

[0023] (1) The images and text information of the short video are obtained and preprocessed as the input of the multimodal autoencoder.

[0024] (2) Construct a multimodal autoencoder. The core objective of this autoencoder is to align two types of vectors in the semantic space through cross-modal constraints, that is, the image vector and text vector describing the same content are close in space, while preserving their respective modal specificity.

[0025] like Figure 2 As shown, the main process of multimodal autoencoder data processing is as follows: First, the model adopts a two-stream network architecture, including two encoders with independent weights: an image stream and a text stream. Next, the extracted image and text embedding vectors are processed by a multimodal recurrent unification module to enhance the semantic association between different modalities in a common feature space, ultimately outputting the feature vectors of the image and text respectively.

[0026] Overall, after acquiring image and text information from short videos, semantic alignment is performed using a multimodal autoencoder, specifically including: Construct a multimodal autoencoder with a dual-stream network architecture for image and text streams; Image and text information are input into a multimodal autoencoder, and independent encoders are used to map the image and text information to a high-dimensional space to obtain high-dimensional image feature vectors and high-dimensional semantic feature vectors. The high-dimensional image feature vector and the high-dimensional semantic feature vector are input into the multimodal cyclic unification module. First, cross-modal alignment is performed. The high-dimensional image feature vector and the high-dimensional semantic feature vector are projected into a common subspace through a multilayer perceptron to achieve alignment mapping. Then, cross-modal cyclic consistency is checked. The decoder obtains the reconstructed image and text.

[0027] The main architecture of a multimodal autoencoder includes an encoder, a multimodal cyclic unified module, and a decoder. The data processing for each module is as follows: ① Encoder Using CNN to process image data Transform into high-dimensional image feature vectors Preserve spatial layout and semantic information of objects; use the Transformer encoder to convert text... Transform into high-dimensional semantic feature vectors It captures the contextual relationships and overall meaning between words.

[0028] ② Multimodal Loop Unified Module This module mainly consists of the following two parts: First, cross-modal alignment: alignment mapping is achieved through a multilayer perceptron (MLP). , That is, by using a 3-layer fully connected network and ReLU, the high-dimensional feature vector is transformed... and Projecting onto a low-dimensional common subspace yields... and : , , in, Alignment mapping representing image features; Alignment mapping representing text features; Represents image features after cross-modal alignment; This represents the text features after cross-modal alignment.

[0029] This part eliminates modal differences and achieves semantic alignment.

[0030] Second, cross-modal cycle consistency check: This part enforces consistency by inversely mapping the common subspace vectors back to the high-dimensional feature space of the original mode. and Located in the same common subspace, this method further enhances cross-modal semantic consistency. Furthermore, it requires no category labels or predefined semantic bridges, making it a purer unsupervised cross-modal alignment mechanism. The specific process is as follows: First pass Will Map back to the high-dimensional feature space of the image, then decode back into an image. The formula is: . Secondly, through Will Map back to the high-dimensional feature space of the text, then decode back into text. The formula is: , in, express The inverse mapping; express The inverse mapping; The image represents the reconstructed image; The text indicating reconstruction; Represents an image decoder; This indicates a text decoder.

[0031] This part only requires inverse mapping to complete the transformation, and no additional module needs to be generated.

[0032] ③ Decoder Reconstruct the original modes from independent latent high-dimensional vectors, ensuring that the vectors retain sufficient modal information.

[0033] ④ Loss Construction The loss function mainly includes the following parts: Reconstruction loss: Measures the difference between the original data and the reconstruction result from the autoencoder, ensuring the effectiveness of the encoder and decoder. , in, N The total number of samples in the dataset. The one-hot vector sequence of the real text has a length of L ; For the decoder in the t Predicted probability distribution over each token.

[0034] Cross-modal cycle consistency check loss: guarantees cross-modal semantic alignment. . Cluster alignment loss: guides alignment vectors to cluster towards cluster centers, enhancing cluster separability. , in, For soft-assignment probabilities, we use Student's probability with 1 degree of freedom. t The similarity of distributions is measured by the following formula: , ( M (where is the number of clusters) is the cluster center. Indicates the cluster center of the j-th cluster; This represents the cluster center of the g-th cluster; .

[0035] For the target distribution, the calculation formula is: . This represents the soft assignment probability, which is the probability that sample i is softly assigned to the center of the g-th cluster.

[0036] Regularization loss: To avoid overfitting, a regularization loss is added here:

[0037] in, , The weight matrix of the MLP. , For bias.

[0038] The overall loss function is:

[0039] in, These are all hyperparameters used to balance the weights of various parameters.

[0040] (3) Input the processed images and text information of the public opinion video into different large language models. By optimizing the prompt, the large model can generate preliminary labels more accurately. These labels are used as pre-labels for subsequent cluster label allocation.

[0041] (4) Define the data vector aligned by the autoencoder as multi-view data, and input it into the multi-view spectral clustering to obtain clusters. This step is existing technology and will not be described in detail here.

[0042] Then, combining the initial labels obtained from the large language model, class label mapping is performed on the clusters. The specific process is as follows: Model annotator i In cluster C Internal tag label is k probability for:

[0043] in, M The total number of clusters. C ( C =1,2,..., M ) is represented as a cluster, TC This represents the total number of data points within the cluster. lij Indicates model annotator i Data j ( j= 1,2,..., TC The label marked with ) For Kronek The function, its formula is:

[0044] So, cluster C Chinese label k The total proportion of occurrences is:

[0045] in, Indicates cluster C The set of model annotators who annotate the data in the middle.

[0046] The model labels with the highest probability are mapped to clusters. C Clustering tags:

[0047] If multiple labels have the same probability, then the global label frequency is taken into account (i.e., the label that appears most frequently in the dataset is selected).

[0048] After obtaining the cluster labels, in order to assign a specific label to each sample data, it is first necessary to calculate the model annotator. i In cluster C Reliability weight W ic The specific formula is as follows: ; in, W ic For clusters C Chinese model annotators i Reliability weights; , All of these are smoothing terms, which makes the reliability estimate stable even when the data is sparse.

[0049] In cluster C In the middle, for sample data j ( j= 1,2,..., TC ), its candidate tags are k Weighted support for:

[0050] For each sample data j ( j= 1,2,..., TC ), its weighted label for:

[0051] when hour, T For a preset threshold and T =0.5, cluster C Samples in j Its weighted label with cluster C Label Consistent, then For sample data j The final tag.

[0052] When it exists k 1≠ k 2 and Then refer to the label of the cluster to which the sample belongs. ,Right now k 1 and k 2. Whose weighted label corresponds to the cluster label to which the sample belongs? If the weighted label is the final label of the sample data, then the weighted label of that sample data is the final label of that sample data.

[0053] when When, then label the sample. jThe status is "Pending review" and requires manual confirmation.

[0054] Any other special cases, such as belonging to a cluster C samples j Its weighted label with cluster C Label Inconsistencies and other issues will also result in samples being marked as "pending verification" for manual confirmation.

[0055] In this step, besides fixing the label by comparing the weighted label with the label of the cluster to which it belongs, there is a broader method: that is, when When the weighted label is set to the sample label, there is no need to compare it with the cluster label; when At that time, the sample is marked as "pending verification" and confirmed manually. This process may require less manpower.

[0056] Example 2 This embodiment discloses an artificial intelligence-based public opinion video tag aggregation system.

[0057] An AI-based public opinion video tagging aggregation system includes: The semantic alignment module is configured to: acquire image and text information from short videos and perform semantic alignment; The initial label generation module is configured to generate initial labels for semantically aligned images and text information using different large language models. The clustering module is configured to cluster semantically aligned images and text to obtain clusters. The cluster label determination module is configured to: calculate the labeling probability of each large language model for each preliminary label type in the current cluster, sum the labeling probabilities of all large language models for each preliminary label type, and select the preliminary label with the highest probability sum as the clustering label of the current cluster. The reliability weight calculation module is configured to calculate the reliability weight of each large language model within the current cluster based on the clustering label of the current cluster. The data weighted label calculation module is configured to: calculate the weighted support of all large language models for each data in the current cluster with different initial label types based on reliability weights, and select the initial label corresponding to the maximum weighted support as the weighted label of the current data; The data final label determination module is configured to determine the final label when the weighted label of the current data is greater than a preset threshold and the weighted label is consistent with the clustering label of the current cluster. Example 3 The purpose of this embodiment is to provide a computer-readable storage medium.

[0058] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the artificial intelligence-based public opinion video tag aggregation method as described in Embodiment 1 of this disclosure.

[0059] Example 4 The purpose of this embodiment is to provide an electronic device.

[0060] An electronic device includes a memory, a processor, and a program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps in the AI-based public opinion video tag aggregation method as described in Embodiment 1 of this disclosure.

[0061] The steps and methods involved in the apparatuses of Embodiments 2, 3, and 4 above correspond to those in Embodiment 1. For specific implementation details, please refer to the relevant description section of Embodiment 1. The term "computer-readable storage medium" should be understood as a single medium or multiple media including one or more instruction sets; it should also be understood as including any medium capable of storing, encoding, or carrying an instruction set for execution by a processor and enabling the processor to perform any of the methods in this invention.

[0062] Those skilled in the art will understand that the modules or steps of the present invention described above can be implemented using general-purpose computer devices. Optionally, they can be implemented using computer-executable program code, thereby allowing them to be stored in a storage device for execution by a computer device, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. The present invention is not limited to any particular combination of hardware and software.

[0063] While the specific embodiments of the present invention have been described above in conjunction with the accompanying drawings, this is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art without creative effort based on the technical solutions of the present invention are still within the scope of protection of the present invention.

Claims

1. A method for aggregating public opinion video tags based on artificial intelligence, characterized in that, Includes the following steps: Extract image and text information from short videos and perform semantic alignment; Different large language models were used to generate preliminary labels for semantically aligned images and text information; Cluster the semantically aligned images and text to obtain clusters; Calculate the labeling probability of each large language model for each initial label type in the current cluster, sum the labeling probabilities of all large language models for each initial label type, and select the initial label with the highest probability sum as the clustering label for the current cluster. Based on the clustering label of the current cluster, calculate the reliability weight of each large language model within the current cluster; Based on reliability weights, calculate the weighted support of all large language models for each data point in the current cluster for different initial label categories, and select the initial label corresponding to the maximum weighted support as the weighted label for the current data. When the weighted label of the current data is greater than the preset threshold and the weighted label is consistent with the clustering label of the current cluster, the weighted label of the current data is determined to be the final label.

2. The AI-based video tag aggregation method for public opinion as described in claim 1, characterized in that, Extracting image and text information from short videos and performing semantic alignment, specifically including: Construct a multimodal autoencoder with a dual-stream network architecture for image and text streams; Image and text information are input into a multimodal autoencoder, and independent encoders are used to map the image and text information to a high-dimensional space to obtain high-dimensional image feature vectors and high-dimensional semantic feature vectors. The high-dimensional image feature vector and the high-dimensional semantic feature vector are input into the multimodal cyclic unification module. First, cross-modal alignment is performed. The high-dimensional image feature vector and the high-dimensional semantic feature vector are projected into a common subspace through a multilayer perceptron to achieve alignment mapping. Then, cross-modal cyclic consistency is checked. The decoder obtains the reconstructed image and text.

3. The AI-based public opinion video tag aggregation method as described in claim 2, characterized in that, The cross-modal alignment specifically refers to: ； ； in, Represents the feature vector of a high-dimensional image; Represents a high-dimensional semantic feature vector; Alignment mapping representing image features; Alignment mapping representing text features; Represents image features after cross-modal alignment; Represents text features after cross-modal alignment; The cross-modal cycle consistency check, obtained through the decoder, yields reconstructed images and text, specifically as follows: ； ； in, express The inverse mapping; express The inverse mapping; The image represents the reconstructed image; The text indicating reconstruction; Indicates an image decoder; This indicates a text decoder.

4. The AI-based public opinion video tag aggregation method as described in claim 3, characterized in that, The loss function of the multimodal autoencoder is: ； ； ； ； ； in, Total loss; Indicates the losses incurred during reconstruction; Indicates the cross-modal cyclic consistency check loss; This represents the cluster alignment loss; Indicates the regularization loss; All are hyperparameters; For the original image data, It is a one-hot vector sequence of real text, with a length of L ; N This represents the total number of samples in the dataset. For the decoder in the t Predicted probability distribution on each token; This is a soft-assignment probability; For the target distribution; , This is the MLP weight matrix. , For bias; This represents the total number of clusters.

5. The AI-based public opinion video tag aggregation method as described in claim 1, characterized in that, The clustering label of the current cluster is calculated as follows: Model annotator i In cluster C Internal tag label is k probability for: ； in, C Represented as a cluster, C =1,2,..., M, M The total number of clusters; TC This represents the total number of data points within the cluster. lij Indicates the model annotator i Data j ( j= 1,2,..., TC Initial labels for model annotation; model annotators i Indicates the first i A large language model; For Kronek function; cluster C Preliminary labeling k The total proportion of occurrences is: ； in, Indicates cluster C The set of model annotators who annotate the data in the middle; ； in, Cluster C Clustering labels.

6. The AI-based public opinion video tag aggregation method as described in claim 5, characterized in that, The specific calculation method for the reliability weight is as follows: ； in, W ic For clusters C Chinese model annotators i Reliability weights; , All are smooth terms; The specific calculation method for the weighted support is as follows: ； in, Indicates in cluster C The Middle j The initial labels for the data are: k The weighted support; The specific calculation method for the weighted label is as follows: ； in, For each data j Weighted labels.

7. The AI-based public opinion video tag aggregation method as described in claim 1, characterized in that, When the weighted label of the current data is less than the preset threshold, or when the weighted label is inconsistent with the clustering label of the current cluster, the final label of the current data is determined manually.

8. An AI-based public opinion video tag aggregation system, characterized in that, include: The semantic alignment module is configured to: acquire image and text information from short videos and perform semantic alignment; The initial label generation module is configured to generate initial labels for semantically aligned images and text information using different large language models. The clustering module is configured to cluster semantically aligned images and text to obtain clusters. The cluster label determination module is configured to: calculate the labeling probability of each large language model for each preliminary label type in the current cluster, sum the labeling probabilities of all large language models for each preliminary label type, and select the preliminary label with the highest probability sum as the clustering label of the current cluster. The reliability weight calculation module is configured to calculate the reliability weight of each large language model within the current cluster based on the clustering label of the current cluster. The data weighted label calculation module is configured to: calculate the weighted support of all large language models for each data in the current cluster with different initial label types based on reliability weights, and select the initial label corresponding to the maximum weighted support as the weighted label of the current data; The data final label determination module is configured to determine the final label when the weighted label of the current data is greater than a preset threshold and the weighted label is consistent with the clustering label of the current cluster.

9. A computer-readable storage medium having a program stored thereon, characterized in that, When the program is executed by the processor, it implements the steps in the AI-based public opinion video tag aggregation method as described in any one of claims 1-7.

10. An electronic device comprising a memory, a processor, and a program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps in the AI-based public opinion video tag aggregation method as described in any one of claims 1-7.

Images