Cross-modal text sentiment polarity analysis method based on visual knowledge graph

Abstract

This invention relates to the field of text analysis technology, and in particular to a cross-modal text sentiment analysis method based on visual knowledge graphs. The method receives text and image data to be analyzed, obtains a text structure representation graph through multi-granular semantic parsing, and outputs a cross-modal joint representation vector by fusing image visual features. It then uses the text structure representation graph to match sentiment interpretation paths and generates preliminary judgment results through context-based dynamic disambiguation. Sentiment weights are corrected based on external domain and cultural context rules, and real-time public opinion data is integrated for collaborative optimization analysis to adjust the strategy. Under the constraint of the disambiguation verification window, adaptive iterative evaluation and optimization are performed on the preliminary results, effectively improving the accuracy and anti-interference capability of cross-modal sentiment analysis.

Description

Technical Field

[0001] This invention relates to the field of text analysis technology, and in particular to a cross-modal text sentiment analysis method based on visual knowledge graphs. Background Technology

[0002] With the rapid development of social media and digital content, user-generated content (UGC) widely exhibits cross-modal expressions combining text and images. Its emotional expression is often highly implicit, context-dependent, and ambiguous, posing a significant challenge to traditional unimodal sentiment analysis methods. Existing technologies are mostly based on pure text or pure image modalities for sentiment recognition, making it difficult to effectively handle complex scenarios such as inconsistencies between text and images, semantic conflicts, or cultural metaphors. This leads to biases in sentiment polarity judgments, particularly prominent in ambiguous expressions such as irony, sarcasm, and symbolic images. Although some studies have attempted to introduce multimodal fusion strategies, they generally suffer from coarse-grained semantic alignment between modalities, insufficient context-dependent modeling, and a lack of external knowledge guidance, resulting in limited sentiment disambiguation capabilities. Furthermore, existing methods typically ignore the influence of user context, cultural background, and real-time public opinion dynamics on sentiment expression, making it difficult to achieve personalized and context-aware sentiment bias analysis.

[0003] The above content is only used to help understand the technical solution of the present invention and does not represent an admission that the above content is prior art. Summary of the Invention

[0004] The main objective of this invention is to provide a cross-modal text sentiment analysis method based on visual knowledge graphs. This method aims to solve the technical problems of existing cross-modal sentiment analysis methods, which are unable to effectively integrate text and image semantics, identify and resolve sentiment ambiguities, and lack the ability to model dynamic factors such as contextual dependence, cultural context and real-time public opinion, resulting in low accuracy of sentiment judgment.

[0005] To achieve the above objectives, this invention provides a method for cross-modal text sentiment analysis based on visual knowledge graphs, the method comprising:

[0006] The system receives the text to be analyzed and associated image data, and performs multi-granular semantic parsing on the text to be analyzed to obtain a text structure representation map. The text structure representation map stores multiple candidate sentiment segments and multiple potentially ambiguous semantic units and their contextual dependencies.

[0007] The image visual features and text semantic features are fused and analyzed to output a cross-modal joint representation vector, wherein the image visual features are extracted in real time through a multi-level convolutional neural network;

[0008] Based on the cross-modal joint representation vector, candidate sentiment interpretation paths are retrieved and matched in the pre-constructed text structure representation graph. Then, combined with dynamic disambiguation in the current context, a preliminary sentiment tendency determination result is generated.

[0009] After receiving the domain sentiment dictionary and cultural context rules returned by the external knowledge base, the domain sentiment dictionary and cultural context rules are input into the context adaptation network for sentiment weight correction, and the sentiment intensity adjustment strategy is output.

[0010] Real-time public opinion background data is obtained through social media interaction interfaces, and collaborative impact analysis is performed based on the real-time public opinion background data and the current cross-modal joint representation to output a tendency optimization strategy.

[0011] The preliminary sentiment tendency determination result is updated using the aforementioned sentiment intensity adjustment strategy and tendency optimization strategy; a disambiguation verification window is preset, and the preliminary sentiment tendency determination result is subjected to adaptive iterative online consistency evaluation and optimization using the disambiguation verification window as a constraint.

[0012] Optionally, the fusion analysis of image visual features and text semantic features to output a cross-modal joint representation vector includes:

[0013] The system interactively obtains multiple sample visual representations, multiple sample text representations, and multiple sample sentiment tag records for the multiple candidate sentiment fragments. The sample visual representations include historical image sentiment annotations, visual object co-occurrence relationships, and scene sentiment mapping tables.

[0014] A graph-text semantic complementarity model is constructed based on the attention mechanism;

[0015] The multiple sample visual representations, multiple sample text representations, and multiple sample sentiment labels are recorded as training data to optimize the parameters of the image-text semantic complementarity model, thereby completing the localized deployment of the image-text semantic complementarity model.

[0016] The image visual features and text semantic features are loaded into the image-text semantic complementarity model, and multiple cross-modal fusion weight parameters of the multiple candidate emotion segments are output.

[0017] The multiple cross-modal fusion weight parameters are aggregated by graph neural network propagation to output the cross-modal joint representation vector.

[0018] Optionally, the step of receiving the text to be analyzed and associated image data, and performing multi-granular semantic parsing on the text to be analyzed to obtain a text structure representation map, includes:

[0019] A preset semantic unit division granularity is used, and the text to be analyzed is segmented with the semantic unit division granularity as a constraint to obtain M text semantic segments, wherein the text semantic segments include lexical-level sentiment polarity, phrase-level sentiment combination and sentence-level sentiment tendency;

[0020] After inputting the M text semantic fragments into a pre-trained language model for semantic encoding, M semantic dependency structures are obtained through dependency parsing.

[0021] Cluster analysis and ambiguous unit detection are performed on the M semantic dependency structures to obtain the multiple potential ambiguous semantic units and their contextual dependencies.

[0022] Based on the multiple potentially ambiguous semantic units, aggregate the M visual association regions and M historical sentiment annotation records corresponding to the M text semantic segments to obtain multiple sets of image-text sentiment association data;

[0023] The multiple sets of image-text sentiment association data are subjected to ambiguity quantification assessment and feature extraction to obtain the text structure representation map.

[0024] Optionally, the step of performing ambiguity quantification assessment and feature extraction on the multiple sets of image-text sentiment association data to obtain the text structure representation map includes:

[0025] An ambiguity evaluation index system is preset, and the multiple sets of image-text sentiment association data are quantitatively scored using the ambiguity evaluation index system as the standard to obtain multiple sets of semantic unit ambiguity scores.

[0026] Construct a semantic ambiguity evaluation matrix based on the multiple sets of semantic unit ambiguity scores;

[0027] Principal component analysis and feature selection are performed on the semantic ambiguity evaluation matrix to generate the multiple potential ambiguous semantic units and their corresponding context dependencies.

[0028] The multiple candidate sentiment fragments and their corresponding multiple potentially ambiguous semantic units, along with their contextual dependencies, are associated and stored to complete the construction of the text structure representation map.

[0029] Optionally, after retrieving and matching candidate sentiment interpretation paths in the pre-constructed text structure representation graph based on the cross-modal joint representation vector, and then dynamically disambiguating in the current context to generate a preliminary sentiment tendency determination result, the process includes:

[0030] The multimodal alignment module calculates the correlation between image visual features and text semantic features in real time to obtain a cross-modal consistency score.

[0031] User preference information and cultural context can be obtained by accessing domain knowledge bases and user history behavior records;

[0032] A multi-dimensional fusion analysis is performed on the cross-modal consistency scores and user preference information with cultural background context to obtain a comprehensive assessment of the current context.

[0033] Based on the current context comprehensive assessment and cross-modal joint representation vector, the emotion interpretation path adaptability determination is performed to obtain path adjustment parameters;

[0034] After the candidate sentiment interpretation path is personalized according to the path adjustment parameters, the candidate sentiment interpretation path and the adjustment basis are integrated to generate the preliminary sentiment tendency judgment result.

[0035] Optionally, the image visual features include object emotional attributes, scene emotional classification, and visual emotional intensity, and the user preference information includes historical emotional expression habits, cultural symbol cognitive preferences, and social context sensitivity.

[0036] Optionally, the step of performing graph neural network propagation and aggregation on the plurality of cross-modal fusion weight parameters to output the cross-modal joint representation vector includes:

[0037] Interactively retrieve historical image and text alignment records;

[0038] Based on the historical image-text alignment records, multiple sets of semantic-visual alignment strengths among the candidate emotional segments are calculated;

[0039] The cross-modal relationship propagation graph is constructed using the aforementioned multiple sets of semantic-visual alignment intensities;

[0040] The multiple cross-modal fusion weight parameters are used as initial node features and input into the cross-modal relationship propagation graph. Multiple enhanced fusion feature vectors are obtained through graph convolution operation.

[0041] The multiple cross-modal fusion weight parameters and multiple enhanced fusion feature vectors are weighted and fused, and the multiple candidate emotional segments are sorted according to the fusion results to obtain an emotional saliency ranking sequence;

[0042] The emotional saliency ranking sequence is subjected to high-level semantic aggregation to output the cross-modal joint representation vector.

[0043] Optionally, the step of determining the path adjustment parameters based on the current context comprehensive evaluation and cross-modal joint representation vector to adapt the sentiment interpretation path includes:

[0044] Interactively obtain multiple sentiment interpretation path adaptation threshold ranges for the multiple candidate sentiment fragments;

[0045] The target adaptation threshold range is invoked from the multiple sentiment interpretation path adaptation threshold ranges based on the cross-modal joint representation vector;

[0046] The current context comprehensive evaluation index is used to traverse the target adaptation threshold range to obtain the context deviation degree, wherein the context deviation degree includes cultural context deviation degree, personal style deviation degree and social scene deviation degree;

[0047] Based on the aforementioned context deviation, the selection weight, confidence level, and adaptability of the fusion method for the sentiment interpretation path are determined to obtain path adjustment parameters.

[0048] Furthermore, to achieve the above objectives, the present invention also provides a cross-modal text sentiment analysis device based on visual knowledge graphs. The device includes: a memory, a processor, and a cross-modal text sentiment analysis program based on visual knowledge graphs stored in the memory and executable on the processor. The cross-modal text sentiment analysis program based on visual knowledge graphs is configured to implement the steps of the cross-modal text sentiment analysis method based on visual knowledge graphs as described above.

[0049] Furthermore, to achieve the above objectives, the present invention also provides a medium storing a cross-modal text sentiment analysis program based on a visual knowledge graph, wherein when the cross-modal text sentiment analysis program based on a visual knowledge graph is executed by a processor, it implements the steps of the cross-modal text sentiment analysis method based on a visual knowledge graph as described above.

[0050] This invention provides a cross-modal text sentiment analysis method based on visual knowledge graphs. The method constructs a text structure representation graph to achieve refined modeling of candidate sentiment fragments and potentially ambiguous semantic units. It then combines multi-level convolutional neural networks to extract image visual features and utilizes a text-image semantic complementarity mechanism for cross-modal fusion, significantly improving the accuracy of inter-modal semantic alignment and the ability to model contextual dependencies. Furthermore, it retrieves candidate sentiment interpretation paths through a pre-constructed text structure representation graph and integrates external domain knowledge, cultural context rules, and real-time public opinion data to achieve sentiment weight correction and sentiment optimization based on contextual adaptation and dynamic synergistic influence, effectively enhancing the ability to disambiguate complex sentiment expressions such as irony and metaphor. Finally, an adaptive iterative optimization mechanism driven by a disambiguation verification window optimizes the consistency and stability of sentiment judgment results online. The overall method overcomes the limitations of traditional single-modal or shallow fusion analysis, achieving deep synergy, knowledge guidance, and dynamic adaptation in cross-modal sentiment understanding, significantly improving the accuracy, robustness, and interpretability of sentiment analysis in complex social scenarios. Attached Figure Description

[0051] Figure 1This is a flowchart illustrating an embodiment of the cross-modal text sentiment analysis method based on visual knowledge graphs according to the present invention.

[0052] The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0053] It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0054] Reference Figure 1 , Figure 1 This is a flowchart illustrating an embodiment of the cross-modal text sentiment analysis method based on visual knowledge graphs according to the present invention.

[0055] In one embodiment, the cross-modal text sentiment analysis method based on visual knowledge graphs includes:

[0056] Step S100: Receive the text to be analyzed and associated image data, and perform multi-granular semantic parsing on the text to be analyzed to obtain a text structure representation map. The text structure representation map stores multiple candidate sentiment segments and multiple potentially ambiguous semantic units and their contextual dependencies.

[0057] The text to be analyzed can be the natural language text portion of user-generated content, containing sentiment expression and potential ambiguity, serving as one of the main input sources for sentiment analysis. For example, the text to be analyzed can include one or more of the following: the body text of a social media post, comment text, title text, etc. Associated image data can be image data co-occurring with the text in the same UGC content, used to provide visual contextual information to assist or correct the sentiment judgment of the text. In an exemplary embodiment, associated image data can include emoji images, scene photos, infographics, etc. Multi-granularity semantic parsing can be a multi-level semantic structure analysis of the text, from words and phrases to sentences and even entire texts, used to identify candidate sentiment fragments and potentially ambiguous semantic units, supporting the construction of a text structure representation graph. Furthermore, multi-granularity semantic parsing can be achieved by using a pre-trained language model (such as BERT) for context-aware semantic segmentation, or by combining a rule engine and a statistical model for multi-level parsing, thereby achieving the technical effect of generating structured semantic units and supporting graph construction.

[0058] A text structure representation graph can be a semantic representation model organized in a graph structure to store candidate sentiment fragments, potentially ambiguous semantic units, and their contextual dependencies in the text to be analyzed. In one specific embodiment, the text structure representation graph performs multi-granular semantic parsing on the input text, identifies sentiment-carrying units and ambiguous expressions, and constructs a directed graph composed of nodes (semantic units) and edges (dependencies or modification relationships). Furthermore, the text structure representation graph can support fine-grained modeling of complex semantic structures, providing a structured semantic foundation for subsequent sentiment interpretation path retrieval and dynamic disambiguation. For example, the text structure representation graph may include dependency syntax subgraphs, sentiment polarity propagation subgraphs, metaphorical reference association subgraphs, etc. Candidate sentiment fragments can be linguistic units in the text that may carry sentiment polarity, used as basic candidate units for sentiment tendency determination. In an exemplary embodiment, candidate sentiment fragments may include explicit sentiment words, implicit evaluative phrases, ironic marker structures, etc. Potentially ambiguous semantic units can be linguistic expressions that may have multiple sentiment interpretations in a specific context, used to identify key semantic points that need to be disambiguated in conjunction with context or external knowledge. For example, potentially ambiguous semantic units may include irony, cultural metaphors, context-dependent pronouns, etc. Contextual dependencies can be modifications, references, or logical connections between semantic units within a text, used to characterize the structural paths of emotion transmission and ambiguity generation. In a specific embodiment, contextual dependencies may include syntactic dependencies, discourse coherence relations, and emotion polarity transmission chains, etc.

[0059] Step S200: The image visual features and text semantic features are fused and analyzed to output a cross-modal joint representation vector, wherein the image visual features are extracted in real time through a multi-level convolutional neural network.

[0060] Image visual features can be numerical features extracted from an image that reflect its content, style, or emotional implications, used as visual input for cross-modal fusion. For example, image visual features can include object-level features, scene-level features, style-level features, etc. A multi-level convolutional neural network (CNN) can be a deep neural network composed of multiple stacked convolutional layers, used to extract image features hierarchically. In a specific embodiment, the multi-level CNN inputs an image into the CNN, extracting visual features layer by layer from low to high levels. Furthermore, the multi-level CNN can be implemented by using a ResNet backbone network to extract deep semantic features, or by combining a multi-scale FPN structure to fuse multi-scale features, thereby achieving hierarchical visual semantic representation and supporting fine-grained image-text alignment. Text semantic features can be vector representations extracted from text that reflect its semantic content and emotional tendency, used as text-side input for cross-modal fusion. For example, text semantic features can include word embedding features, sentence vector features, context-aware semantic features, etc.

[0061] A cross-modal joint representation vector can be a unified semantic vector generated by fusing image visual features and text semantic features, used to represent the overall sentiment semantics of image and text content. In an exemplary embodiment, the cross-modal joint representation vector aligns and fuses image visual features extracted by a multi-level convolutional neural network with text semantic features through an image-text semantic complementarity mechanism. Furthermore, the cross-modal joint representation vector can improve the semantic alignment accuracy between modalities, enhance context dependency modeling capabilities, and provide a unified semantic space for sentiment interpretation path matching and synergistic influence analysis. For example, the cross-modal joint representation vector may include early fusion vectors, late fusion vectors, and intermediate interactive fusion vectors. The fusion analysis of image visual features and text semantic features can involve mapping image and text features to a shared semantic space and performing interactive computation. Furthermore, the fusion analysis of image visual features and text semantic features can achieve feature alignment using a cross-modal attention mechanism, or feature interaction through bilinear pooling or tensor fusion, thereby achieving the technical effect of generating a cross-modal joint representation vector and improving semantic alignment accuracy. Real-time extraction of image visual features through a multi-level convolutional neural network can involve inputting the image into a CNN and extracting visual features layer by layer from low to high levels. Furthermore, real-time extraction of image visual features through multi-level convolutional neural networks can be achieved by using ResNet backbone network to extract deep semantic features, or by combining FPN structure to fuse multi-scale features, thereby achieving the technical effect of obtaining hierarchical visual semantic representation and supporting fine-grained image-text alignment.

[0062] Step S300: After retrieving and matching candidate sentiment interpretation paths in the pre-constructed text structure representation map based on the cross-modal joint representation vector, and combining the current context with dynamic disambiguation, a preliminary sentiment tendency determination result is generated.

[0063] The candidate sentiment interpretation path can be a semantic reasoning path in the text structure representation graph that connects candidate sentiment fragments with disambiguation criteria, providing an interpretable reasoning basis for dynamic disambiguation. For example, candidate sentiment interpretation paths may include cultural metaphor interpretation paths, irony recognition paths, and contextual negation paths. The preliminary sentiment tendency determination result can be an initial sentiment polarity judgment generated based on cross-modal joint representation and graph path matching, used as a benchmark for subsequent weight correction and optimization. In a specific embodiment, the preliminary sentiment tendency determination result may include a positive tendency initial judgment, a negative tendency initial judgment, a neutral or uncertain initial judgment, etc. In this embodiment, the text structure representation graph serves as the index structure for path retrieval, working in conjunction with the cross-modal joint representation vector to provide text-side semantic constraints to guide the fusion process; the cross-modal joint representation vector serves as the query basis for path matching in the graph, working in conjunction with the text structure representation graph.

[0064] Retrieving and matching candidate sentiment interpretation paths in a pre-constructed text structure representation graph can be achieved by using cross-modal joint representation vectors as queries to search for semantically similar paths within the graph. Furthermore, this retrieval and matching can be accomplished through path embedding matching based on graph neural networks or heuristic path search using semantic similarity ranking, thus achieving the technical effect of locating the most likely sentiment interpretation logical chain. Combining dynamic disambiguation with the current context to generate a preliminary sentiment tendency judgment result can be achieved by integrating path matching results with the current context and selecting the optimal sentiment interpretation. Further, combining dynamic disambiguation with the current context to generate a preliminary sentiment tendency judgment result can be achieved through probabilistic disambiguation based on Bayesian inference or by using reinforcement learning to select the optimal interpretation path, thus achieving the technical effect of outputting an initial sentiment polarity judgment and possessing a certain disambiguation capability.

[0065] Step S400: After receiving the domain sentiment dictionary and cultural context rules returned by the external knowledge base, input the domain sentiment dictionary and cultural context rules into the context adaptation network to correct the sentiment weights and output the sentiment intensity adjustment strategy.

[0066] The external knowledge base can be a structured knowledge system storing domain sentiment dictionaries and cultural context rules, used to provide external semantic constraints to enhance the accuracy of sentiment understanding. For example, the external knowledge base may include a social media sentiment dictionary, a cross-cultural pragmatic rule library, a domain-specific sentiment ontology, etc. The domain sentiment dictionary can be a set of sentiment terms and their polarity strength annotations customized for a specific application domain, used to correct the bias of a general sentiment dictionary in a specific context. In a specific embodiment, the domain sentiment dictionary may include a political sentiment dictionary, an entertainment gossip dictionary, a technology product evaluation dictionary, etc. Cultural context rules can be a set of rules describing the relationship between language expression and sentiment mapping in different cultural backgrounds, used to support the correct interpretation of culturally specific expressions (such as Chinese irony and Western satire). For example, cultural context rules may include high-context cultural expression rules, low-context cultural expression rules, cross-cultural sentiment mapping rules, etc.

[0067] A context-adaptive network can be a neural network module used to receive external knowledge (such as a domain sentiment lexicon or cultural context rules) and adjust sentiment weights accordingly. In an exemplary embodiment, the context-adaptive network embeds external knowledge into the current sentiment representation based on attention or gating mechanisms, dynamically adjusting the contribution weights of each sentiment unit. Furthermore, the context-adaptive network can implement context-aware correction of sentiment weights, outputting sentiment intensity adjustment strategies to improve adaptability to culture-specific or domain-specific sentiment expressions. Exemplarily, the context-adaptive network may include a cultural rule gating network, a domain lexicon attention network, a hybrid knowledge adapter, etc. Sentiment weight correction can be achieved by dynamically adjusting the contribution weights of each sentiment unit in the final judgment based on external knowledge, thereby improving the adaptability of sentiment judgment to context and domain. In a specific embodiment, sentiment weight correction may include cultural sensitivity weight adjustment, domain polarity shift correction, and contextual negation weight suppression. The sentiment intensity adjustment strategy can be a specific operational instruction output by the context-adaptive network for adjusting the intensity of sentiment polarity, used to guide the quantitative update of subsequent sentiment judgment results. For example, emotion intensity regulation strategies may include polarity amplification strategies, polarity attenuation strategies, polarity reversal strategies, etc.

[0068] Receiving domain sentiment lexicons and cultural context rules from external knowledge bases can be achieved by loading structured external knowledge via API or local caching, thus introducing prior domain and cultural knowledge. Inputting these lexicons and cultural context rules into the context adaptation network for sentiment weight correction involves encoding external knowledge and fusing it with the current sentiment representation to adjust the weights of each unit. Furthermore, this sentiment weight correction can be achieved by using gating mechanisms to control the intensity of knowledge injection or by using cross-attention mechanisms to achieve knowledge alignment, thereby generating context-aware sentiment weight configurations. Outputting sentiment intensity adjustment strategies can transform the corrected weights into specific sentiment intensity adjustment instructions, providing executable strategies for subsequent updates.

[0069] Step S500: Obtain real-time public opinion background data through social media interaction interface, and conduct collaborative impact analysis based on real-time public opinion background data and current cross-modal joint representation, and output bias optimization strategy.

[0070] The social media interaction interface can be a program interface used to access social media platform APIs to obtain real-time public opinion data, providing dynamic public opinion background to support sentiment optimization. For example, the social media interaction interface may include the Weibo API interface, Twitter stream data interface, Reddit topic monitoring interface, etc. Real-time public opinion background data can be the sentiment distribution and public opinion trend on social media regarding a specific topic or event within the current time period, used as an external dynamic context for collaborative influence analysis. In a specific embodiment, real-time public opinion background data may include the sentiment trend of hot events, group sentiment fluctuation data, and the migration trajectory of topic sentiment polarity, etc.

[0071] Collaborative influence analysis can analyze the interaction between cross-modal joint representations and real-time public opinion background to generate bias optimization strategies that align individual sentiment judgments with the group context. For example, collaborative influence analysis may include individual-group sentiment alignment analysis, public opinion-driven polarity transfer analysis, and cross-modal-public opinion coupling strength analysis. Bias optimization strategies can be strategies generated based on the results of collaborative influence analysis to adjust the direction or confidence level of sentiment bias, thereby improving the rationality and robustness of sentiment judgments in dynamic social contexts. In an exemplary embodiment, bias optimization strategies may include public opinion consistency enhancement strategies, extreme sentiment smoothing strategies, and topic relevance weighting strategies. In this embodiment, cross-modal joint representation vectors and real-time public opinion background data collaborate to jointly participate in the generation of bias optimization strategies. Obtaining real-time public opinion background data through social media interaction interfaces can involve calling social media APIs to pull sentiment distribution data for the current topic.

[0072] Furthermore, obtaining real-time public opinion background data through social media interaction interfaces can be achieved by subscribing to streaming public opinion data or by periodically batch-grabbing sentiment statistics of trending topics, thereby achieving the technical effect of obtaining dynamic group sentiment context. Collaborative impact analysis based on real-time public opinion background data and current cross-modal joint representations can model the interaction between individual sentiment and group public opinion. Further, collaborative impact analysis based on real-time public opinion background data and current cross-modal joint representations can be achieved by constructing graph neural networks to simulate individual-group sentiment propagation or by using contrastive learning to align individual and group sentiment distributions, thereby achieving the technical effect of generating biased optimization strategies. Outputting biased optimization strategies can transform the results of collaborative impact analysis into sentiment bias adjustment instructions, thereby achieving the technical effect of dynamically optimizing sentiment judgment results.

[0073] Step S600: Update the preliminary sentiment tendency judgment results using sentiment intensity adjustment and sentiment optimization strategies. A disambiguation verification window is preset, and adaptive iterative online consistency evaluation and optimization are performed on the preliminary sentiment tendency judgment results using this window as a constraint.

[0074] The disambiguation verification window can be a temporal or logical constraint window used to limit the evaluation scope of adaptive iterative optimization. In one specific embodiment, the disambiguation verification window uses a preset sliding time window or semantic consistency threshold as the boundary condition for online evaluation. Furthermore, the disambiguation verification window can ensure the consistency and stability of sentiment judgment results within a local temporal or semantic range, preventing excessive fluctuations. For example, the disambiguation verification window may include a sliding time window, a semantic consistency window, a user interaction feedback window, etc. Adaptive iterative online consistency evaluation and optimization can be a process of multiple evaluations and parameter fine-tuning of the sentiment judgment results under the constraints of the disambiguation verification window, used to ensure the stability and consistency of the output results in the temporal or semantic dimensions. In an exemplary embodiment, adaptive iterative online consistency evaluation and optimization may include consistency verification within a sliding window, adaptive adjustment of the confidence threshold, and multiple rounds of disambiguation feedback loops.

[0075] In this embodiment, the disambiguation verification window collaborates with the adaptive iterative online consistency evaluation and tuning to provide the evaluation boundary; the disambiguation verification window also collaborates with the preliminary sentiment tendency judgment result as an input constraint for tuning. Updating the preliminary sentiment tendency judgment result using sentiment intensity adjustment and sentiment optimization strategies can be achieved by applying both strategies to the initial judgment result and performing weighted or reordered calculations. Furthermore, updating the preliminary sentiment tendency judgment result using sentiment intensity adjustment and sentiment optimization strategies can be achieved through a linear weighted fusion strategy output or a strategy combination mechanism based on a decision tree, thereby achieving the technical effect of generating optimized sentiment tendency judgments. Using the disambiguation verification window as a constraint, performing adaptive iterative online consistency evaluation and tuning on the preliminary sentiment tendency judgment result can be achieved by repeatedly evaluating the consistency of the result and fine-tuning the parameters within the window range. Furthermore, using the disambiguation verification window as a constraint, performing adaptive iterative online consistency evaluation and tuning on the preliminary sentiment tendency judgment result can be achieved through confidence smoothing based on a sliding window or feedback tuning using historical judgment results, thereby achieving the technical effect of improving the temporal stability and semantic consistency of the result.

[0076] Taking the identification of satirical content on social media as an example, the cross-modal text sentiment analysis method based on visual knowledge graphs in this embodiment can be as follows: A user posts a picture of "sunny weather" with the caption "Today is another beautiful day!", but the actual day is the day of a major negative event. The system receives this image-text pair, identifies "beautiful" as a candidate sentiment fragment through multi-granular semantic parsing; a multi-level convolutional neural network extracts sunny weather features from the image and fuses them with text semantic features to generate a cross-modal joint representation vector; the system retrieves the irony interpretation path in the text structure representation graph; simultaneously, a context adaptation network combines cultural context rules to correct sentiment weights; the social media interaction interface obtains the public opinion on the day showing that the group sentiment is negative, and collaborative influence analysis generates a sentiment optimization strategy; finally, iterative optimization is completed within the disambiguation verification window, and the sentiment is determined to be negative.

[0077] In one embodiment, the image visual features and text semantic features are fused and analyzed to output a cross-modal joint representation vector, including:

[0078] The interaction obtains multiple sample visual representations, multiple sample text representations, and multiple sample sentiment label records for multiple candidate sentiment fragments. Among them, the sample visual representations include historical image sentiment annotations, visual object co-occurrence relationships, and scene sentiment mapping tables.

[0079] The sample visual representation can be a multi-dimensional semantic representation of the image used for model training, providing rich visual emotional context and supporting fine-grained image-text alignment learning. In this embodiment, the sample visual representation can include, but is not limited to, one or more of the following: historical image sentiment annotation, visual object co-occurrence relationships, and scene sentiment mapping tables. Historical image sentiment annotation can be the sentiment category or intensity information of previously annotated images, which can be used to provide prior emotional semantic reference for new images. Historical image sentiment annotation can include annotations from a general sentiment image library, domain-specific image sentiment labels, and user feedback correction annotations. Visual object co-occurrence relationships can be spatial or semantic co-occurrence patterns among multiple visual objects in an image, which can be used to reveal potential emotional cues. In an exemplary embodiment, visual object co-occurrence relationships can include spatial proximity co-occurrence, semantic role co-occurrence, and cultural symbol combination co-occurrence. The scene sentiment mapping table can be a predefined mapping table of typical emotional tendencies, which can be used to help the model understand the modulating effect of the overall scene on local emotions. The sample text representation can be a text-side semantic vector paired with the sample visual representation, which can be used as text input for training the image-text semantic complementarity model. The sample sentiment label record can be the real sentiment polarity or intensity label corresponding to the multimodal sample labeled manually or semi-automatically, which can be used to supervise the learning process of the image-text semantic complementarity model.

[0080] Interactively obtaining multiple visual representations, text representations, and sentiment label records for multiple candidate sentiment segments can be achieved by extracting image-text label triples from a multimodal sentiment annotation dataset based on alignment with candidate sentiment segments. Furthermore, this operation can be implemented by aligning historical UGC data with sentiment segment anchors and constructing image-text pairs with fine-grained sentiment annotations through a crowdsourcing platform, thereby building a training dataset with semantically fine-grained alignment.

[0081] A graph-text semantic complementarity model is constructed based on the attention mechanism.

[0082] The attention mechanism can be a neural network component used to achieve fine-grained alignment and complementarity between image and text modalities. The image-text semantic complementarity model can be a neural network model built on the attention mechanism, used to dynamically model the fine-grained association between images and text at the sentiment expression level. In this embodiment, the image-text semantic complementarity model can include, but is not limited to, cross-modal cross-attention models, sentiment-oriented gating fusion models, and context-aware alignment models. Building an image-text semantic complementarity model based on the attention mechanism can involve designing a neural network architecture that includes cross-modal attention layers, allowing image and text features to mutually guide and focus. Furthermore, this operation can be achieved by using a Transformer encoder-decoder structure to implement bidirectional attention, and by using a two-stream network with a cross-attention module to achieve feature interaction, thereby establishing a learnable image-text sentiment complementarity mechanism.

[0083] Multiple sample visual representations, multiple sample text representations, and multiple sample sentiment labels are recorded as training data to optimize the parameter tuning of the image-text semantic complementarity model, thus completing the localized deployment of the image-text semantic complementarity model.

[0084] The training data can be a triplet dataset consisting of sample visual representations, sample text representations, and sample sentiment tag records, which can be used for parameter optimization of the image-text semantic complementarity model. For example, the training data can include manually labeled multimodal sentiment datasets, weakly supervised crawling of UGC sentiment pairs, and synthetic adversarial generation of sentiment samples. Using multiple sample visual representations, multiple sample text representations, and multiple sample sentiment tag records as training data for parameter tuning and optimization of the image-text semantic complementarity model can involve using sentiment tags as a supervision signal to minimize prediction error and update model parameters. Furthermore, this operation can be achieved by employing contrastive learning loss to enhance modality alignment and combining sentiment consistency regularization terms to prevent overfitting, thereby enabling the model to possess fine-grained image-text sentiment alignment capabilities.

[0085] Local deployment of a graph-text semantic complementarity model can be achieved by converting the trained model into a format suitable for the target platform and loading it into the inference engine. In an exemplary embodiment, this operation can be implemented by reducing the model size using model compression techniques and by using containerization to achieve microservice deployment, thereby supporting efficient online inference. Local deployment can involve deploying the trained graph-text semantic complementarity model to the target runtime environment (such as edge devices or servers), which can be used to support low-latency, high-efficiency online inference. Exemplary examples include model quantization deployment, ONNX format export deployment, and TensorRT accelerated deployment.

[0086] Image visual features and text semantic features are loaded into a graphic-text semantic complementarity model, which outputs multiple cross-modal fusion weight parameters for multiple candidate sentiment segments.

[0087] The cross-modal fusion weight parameters can be learnable parameters generated by the image-text semantic complementarity model for each candidate sentiment segment. These parameters reflect the relative credibility and contribution of the image and text modalities in the current context and can be used to achieve dynamic weighted fusion between modalities, enhance the ability to focus on key sentiment cues, and suppress misleading modal signals. In a specific embodiment, the cross-modal fusion weight parameters may include visual-dominant weights, text-dominant weights, conflict-balancing weights, etc. Loading image visual features and text semantic features into the image-text semantic complementarity model can be achieved by inputting the feature vectors extracted in the previous steps into the deployed model, thereby initiating the cross-modal fusion inference process. Outputting multiple cross-modal fusion weight parameters for multiple candidate sentiment segments can be achieved by the model calculating the fusion weights of the image and text for each candidate sentiment segment. Furthermore, this operation can be implemented by outputting soft attention weight vectors and generating gated fusion coefficients, thereby enabling dynamic allocation of modal contributions.

[0088] A graph neural network is used to propagate and aggregate multiple cross-modal fusion weight parameters to output a cross-modal joint representation vector.

[0089] Graph neural network propagation aggregation can be a graph-based message-passing mechanism used to structurally integrate multiple cross-modal fusion weight parameters. This aggregation can be achieved by using candidate sentiment fragments as nodes and their contextual dependencies as edges, performing graph message passing. Furthermore, this operation can be implemented using graph attention networks (GAT) for weighted aggregation or graph convolutional networks (GCN) for mean aggregation, thus integrating local fusion weights into a globally consistent representation. The output cross-modal joint representation vector can be generated by concatenating or pooling the final node representations of the graph neural network into a unified vector, thereby generating a joint semantic representation with embedded contextual dependencies and sentiment logic constraints.

[0090] For example, in the scenario of identifying irony on social media, the cross-modal text sentiment analysis method based on visual knowledge graphs in this embodiment could be as follows: A user posts a picture of a "sunny day" with the caption "Today is absolutely terrible!" The system extracts the visual features of the image (sunshine, blue sky) and the semantic features of the text ("terrible"), and inputs them into a locally deployed image-text semantic complementarity model. Based on the pattern learned during training that "sunny day + negative text" is often ironic, the model generates cross-modal fusion parameters with high text weights and low visual weights for the candidate sentiment segment. Subsequently, the graph neural network propagates and aggregates the data, considering the dependency relationship between the segment and other sentiment units in the context (such as emojis and hashtags), and outputs a cross-modal joint representation vector that strengthens the negative sentiment, supporting accurate subsequent determination of negative sentiment.

[0091] In one embodiment, the system receives the text to be analyzed and associated image data, and performs multi-granular semantic parsing on the text to be analyzed to obtain a text structure representation map, including:

[0092] The semantic unit division granularity is preset, and the text to be analyzed is segmented with the semantic unit division granularity as a constraint to obtain M text semantic segments, where the text semantic segments include lexical sentiment polarity, phrase sentiment combination and sentence sentiment tendency;

[0093] The semantic unit division granularity can be a preset text segmentation hierarchy standard, used to guide the granularity boundary of multi-granularity semantic parsing, and can constrain the text segmentation process to ensure that the generated fragments cover three levels of sentiment semantics: lexical, phrase, and sentence. The M text semantic fragments can be M sub-units with independent sentiment semantics segmented from the text to be analyzed according to the semantic unit division granularity, and can be used as the basic processing units for multi-granularity sentiment modeling. For example, the M text semantic fragments can include lexical-level sentiment polarity, phrase-level sentiment combinations, and sentence-level sentiment tendencies. Lexical-level sentiment polarity can be the basic sentiment tendency carried by a single word (e.g., "good" is positive, "bad" is negative), and can be used to constitute the atomic units of fine-grained sentiment analysis. In an exemplary embodiment, lexical-level sentiment polarity can include, but is not limited to, one or more of explicit sentiment words, implicit evaluation words, and sentiment intensity modifiers. Phrase-level sentiment combinations can be phrase structures composed of multiple words that express a specific sentiment tendency as a whole, and can be used to capture new sentiment meanings generated by local semantic combinations (e.g., "not so good" is negative overall). Furthermore, phrase-level emotional combinations can include, but are not limited to, negative emotional phrases, intensity phrases, and ironic fixed collocations.

[0094] Sentence-level sentiment can be the overall emotional direction expressed by a complete sentence in its context. It can be used to reflect high-level semantic intent and is often used to identify irony or sarcasm. In a specific embodiment, sentence-level sentiment can include, but is not limited to, sentences that appear positive but are actually negative, rhetorical questions, and sentences with multiple modifiers. A preset semantic unit granularity is used to segment the text to be analyzed, and this granularity is used as a constraint to obtain M text semantic segments. This segmentation can be based on predefined three-level granularity rules: vocabulary, phrase, and sentence. Furthermore, this operation can be achieved through multi-granularity segmentation using rule template matching or by combining sequence labeling models to predict granularity boundaries, thereby generating a set of text segments covering multiple levels of sentiment semantics.

[0095] After inputting M semantic text fragments into a pre-trained language model for semantic encoding, M semantic dependency structures are obtained through dependency parsing.

[0096] The pre-trained language model can be a deep neural network language model pre-trained on a large-scale corpus, which can be used for context-aware deep semantic encoding of text semantic fragments. For example, the pre-trained language model can include, but is not limited to, BERT-like models, RoBERTa-like models, and DeBERTa-like models. Semantic encoding can be the process of converting text semantic fragments into dense vector representations, which can be used to provide a numerical semantic foundation for subsequent dependency analysis and clustering. In an exemplary embodiment, semantic encoding can include, but is not limited to, context embedding, sentence vector representation, and hierarchical feature encoding. Dependency parsing can be a natural language processing technique that identifies grammatical dependency relationships (such as subject-verb, verb-object) between words in a sentence, which can be used to construct semantic dependency structures and reveal sentiment modification and transmission paths.

[0097] The M semantic dependency structures can be syntax-semantic graphs obtained after performing dependency parsing on each text semantic segment, which can be used to support ambiguous unit detection and context dependency modeling. Inputting the M text semantic segments into a pre-trained language model for semantic encoding can be achieved by feeding each segment separately into the pre-trained language model to obtain context embeddings. Furthermore, this operation can be achieved by using the last hidden state as the encoding or by fusing multi-layer representations to enhance semantic expression, thereby obtaining deep context-aware semantic vectors. Obtaining the M semantic dependency structures through dependency parsing can be achieved by performing dependency parsing on each encoded segment to construct a syntax graph. For example, this operation can be implemented using the Stanford CoreNLP toolkit or by employing a Transformer-based dependency parser, thereby revealing the structural paths of sentiment modification and transmission.

[0098] Cluster analysis and ambiguous unit detection are performed on M semantic dependency structures to obtain multiple potential ambiguous semantic units and their contextual dependencies.

[0099] Clustering analysis, an unsupervised learning method, groups semantic dependency structures based on similarity. It can be used to identify groups of structures with similar semantic patterns, aiding in the discovery of ambiguous units. Ambiguous unit detection is the process of identifying semantic units with polysemy or context sensitivity from the clustering results. It can be used to locate key points requiring cross-modal disambiguation. Furthermore, ambiguous unit detection can include, but is not limited to, irony marker detection, cultural metaphor identification, and context-dependent pronoun detection.

[0100] Clustering analysis and ambiguous unit detection are performed on M semantic dependency structures to obtain multiple potentially ambiguous semantic units and their contextual dependencies. This can be achieved by first clustering similar structures and then identifying highly ambiguous units within each cluster. For example, this operation can be implemented by combining graph edit distance-based clustering with manual rule detection, or by using end-to-end graph neural networks for joint clustering and detection, thereby accurately locating key semantic points that require cross-modal disambiguation.

[0101] Based on multiple potentially ambiguous semantic units, M visual association regions corresponding to M text semantic segments and M historical sentiment annotation records are aggregated to obtain multiple sets of image-text sentiment association data;

[0102] The M visually associated regions can be local areas in the image that are semantically or emotionally related to each text semantic segment, and can be used to establish fine-grained anchor points for cross-modal alignment of images and text. The M historical sentiment annotation records can be sentiment annotation data for the same or similar image-text combinations in the past, and can be used to provide prior sentiment evidence to support the current ambiguity judgment. Furthermore, the M historical sentiment annotation records can include, but are not limited to, user feedback annotations, expert annotation records, and crowdsourced sentiment tags. Multiple sets of image-text sentiment association data can be a set of triplets consisting of potentially ambiguous semantic units, corresponding visually associated regions, and historical sentiment annotation records, which can be used to form a cross-modal sentiment evidence chain to support quantitative assessment of ambiguity. For example, multiple sets of image-text sentiment association data can include, but are not limited to, ironic image-text evidence sets, cultural metaphor evidence sets, and sentiment conflict evidence sets.

[0103] By aggregating M visually related regions and M historical sentiment annotation records corresponding to M text semantic fragments based on multiple potentially ambiguous semantic units, multiple sets of image-text sentiment association data are obtained. This can be achieved by retrieving relevant regions and historical annotations in an image using ambiguous units as anchor points. In an exemplary embodiment, this operation can be implemented by matching visual regions through a cross-modal retrieval model or by linking historical annotations based on a knowledge graph, thereby constructing a sentiment evidence chain that integrates image and text.

[0104] Ambiguity quantification and feature extraction were performed on multiple sets of text-image sentiment correlation data to obtain a text structure representation map.

[0105] Ambiguity quantification assessment can be a numerical measure of semantic conflict and modal consistency in image-text sentiment association data, providing calculable ambiguity indicators for graph construction. Further, ambiguity quantification assessment can include, but is not limited to, modal conflict scores, context sensitivity indices, and historical annotation divergence. Feature extraction is the process of extracting high-order structured features from image-text sentiment association data, which can be used to generate node and edge attributes for text structure representation graphs. Performing ambiguity quantification assessment and feature extraction on multiple sets of image-text sentiment association data to obtain text structure representation graphs can involve calculating indicators such as conflict degree and consistency, and extracting structured features to construct the graph. For example, this operation can be achieved through multi-task learning joint assessment and extraction, or through a hybrid processing based on rule engines and statistical models, thereby generating structured graphs with embedded cross-modal cues and ambiguous contexts.

[0106] Taking cultural metaphor content analysis as an example, the cross-modal text sentiment analysis method based on visual knowledge graphs in this embodiment can be as follows: A user posts a picture of a "dragon" with the caption "He really is a dragon." The system segments "dragon" into a lexical-level fragment by semantic unit division granularity; after encoding by a pre-trained language model, dependency analysis identifies it as a noun subject; clustering reveals that this structure often appears in ambiguous contexts; ambiguous unit detection marks "dragon" as a potential cultural metaphor; the system aggregates the visual regions of "dragon" in the image (traditional auspicious image) and historical annotations; ambiguity quantification assessment shows high cultural sensitivity; the finally constructed text structure representation graph associates "dragon" nodes with cultural context rules and visual features, supporting accurate subsequent determination of positive sentiment.

[0107] In one embodiment, multiple sets of image-text sentiment correlation data are subjected to ambiguity quantification assessment and feature extraction to obtain a text structure representation map, including:

[0108] An ambiguity evaluation index system was pre-set, and multiple sets of image-text sentiment association data were quantitatively scored using the ambiguity evaluation index system as the standard to obtain multiple sets of semantic unit ambiguity scores.

[0109] Construct a semantic ambiguity evaluation matrix based on multiple sets of semantic unit ambiguity scores;

[0110] Principal component analysis and feature selection are performed on the semantic ambiguity evaluation matrix to generate multiple potential ambiguous semantic units and their corresponding context dependencies.

[0111] Multiple candidate sentiment fragments and their corresponding multiple potentially ambiguous semantic units, along with their contextual dependencies, are associated and stored to complete the construction of a text structure representation graph.

[0112] The ambiguity evaluation index system can be a set of multi-dimensional indicators used to quantitatively assess the degree of semantic ambiguity in image-text sentiment association data. It can be used to transform subjective ambiguity into calculable and comparable numerical standards, supporting systematic ambiguity scoring. In this embodiment, the ambiguity evaluation index system can be established by defining a set of quantifiable ambiguity assessment indicators covering multiple dimensions (such as modality, culture, and history). For example, the ambiguity evaluation index system can be implemented by constructing rule-based indicators based on expert knowledge or by mining indicators from labeled divergent data through weakly supervised learning, thereby establishing an objective and reproducible ambiguity assessment standard. The multiple sets of semantic unit ambiguity scores can be numerical ambiguity scores calculated for semantic units in each image-text sentiment association data based on the ambiguity evaluation index system. These scores can be used to provide quantitative input for subsequent matrix construction and feature selection. In an exemplary embodiment, the multiple sets of semantic unit ambiguity scores can be obtained by calculating and normalizing the scores of each indicator for each set of image-text sentiment association data. Furthermore, this process can be achieved by using weighted summation to generate a comprehensive ambiguity score or by retaining the original multidimensional score for subsequent matrix construction, thereby enabling a numerical representation of the degree of ambiguity.

[0113] A semantic ambiguity evaluation matrix can be a numerical matrix composed of semantic units as rows and ambiguity evaluation indicators as columns, representing the distribution of each unit in a multidimensional ambiguity space. It can be used to structurally express ambiguity features and supports dimensionality reduction and key feature extraction. In this embodiment, the semantic ambiguity evaluation matrix can be formed by organizing the multidimensional scores of each semantic unit into a single row of the matrix, thus creating a structured high-dimensional ambiguity representation. Principal component analysis (PCA) can be an unsupervised linear dimensionality reduction technique used to extract the orthogonal feature directions with the largest variance in the data. This can be used to compress the dimension of the semantic ambiguity evaluation matrix while retaining the most discriminative ambiguity features.

[0114] Feature selection can be the process of filtering the subset that contributes most to ambiguity recognition from a high-dimensional feature space, and can be used to improve the accuracy and interpretability of identifying potentially ambiguous semantic units. Furthermore, feature selection can be achieved through methods such as variance threshold-based filtering, mutual information-based feature ranking, or embedded L1 regularization selection. In an exemplary embodiment, principal component analysis and feature selection of the semantic ambiguity evaluation matrix can be performed by first reducing the dimensionality through principal component analysis, and then filtering the principal components or original features that contribute the most to ambiguity discrimination. For example, this operation can be achieved by retaining principal components with 95% cumulative variance and then performing Lasso feature selection, or by directly using recursive feature elimination (RFE) in the original index space, thereby extracting the most discriminative ambiguous feature dimensions.

[0115] Generating multiple potentially ambiguous semantic units and their corresponding contextual dependencies can be achieved by identifying highly ambiguous semantic units based on filtered features and tracing back their dependency syntax or discourse structure to extract dependencies. Furthermore, this operation can be implemented by automatically filtering units by setting an ambiguity scoring threshold or by locating key dependency edges using a graph attention mechanism, thereby outputting data-driven validated ambiguous units and structured contexts. Associating and storing multiple candidate sentiment fragments and their corresponding potentially ambiguous semantic units and their contextual dependencies can be achieved by establishing nodes (sentiment fragments, ambiguous units) and edges (dependencies, modifications, referencing) in a graph database or in-memory graph structure. In a specific embodiment, this operation can be achieved by using RDF triples to store semantic relationships or constructing an attribute graph model to support efficient path retrieval, thereby forming a reasonable structured knowledge representation. Completing the construction of the text structure representation graph can be achieved by integrating all nodes and edges to form a complete graph instance, thereby providing a high-fidelity knowledge foundation for subsequent sentiment interpretation path retrieval.

[0116] Taking cross-cultural satirical content analysis as an example, the cross-modal text sentiment analysis method based on visual knowledge graphs in this embodiment can be as follows: A user posts a "clapping" emoji with the caption "Appropriately expressed," but the actual event is a major mistake. The system has obtained the image-text sentiment association data; the ambiguity evaluation index system calculates that the modal consistency score of this combination is low (the image is positive, the text is positive on the surface but negative in the context), and the cultural context conflict is high (this combination is often ironic in Chinese online contexts); a high ambiguity score is generated; the semantic ambiguity evaluation matrix highlights the "cultural irony" dimension after principal component analysis; the feature selection retains this dimension; the system identifies "Appropriately expressed" as a potential ambiguous semantic unit and extracts its negative dependency with the event context; finally, it is associated and stored with the candidate sentiment fragment "clapping + appropriately expressed" in the text structure representation graph, supporting subsequent accurate determination of negative sentiment.

[0117] In one embodiment, after retrieving and matching candidate sentiment interpretation paths in a pre-constructed text structure representation graph based on cross-modal joint representation vectors, and then dynamically disambiguating in the current context to generate a preliminary sentiment tendency determination result, the process includes:

[0118] The multimodal alignment module calculates the correlation between image visual features and text semantic features in real time to obtain a cross-modal consistency score.

[0119] User preference information and cultural context can be obtained by accessing domain knowledge bases and user history behavior records;

[0120] A multi-dimensional fusion analysis of cross-modal consistency scores, user preference information, and cultural context is conducted to obtain a comprehensive assessment of the current context.

[0121] Based on the current contextual comprehensive assessment and cross-modal joint representation vector, the path adaptability of sentiment interpretation is determined, and path adjustment parameters are obtained;

[0122] After making personalized adjustments to the candidate sentiment interpretation paths based on the path adjustment parameters, the candidate sentiment interpretation paths and the adjustment basis are integrated to generate preliminary sentiment tendency judgment results.

[0123] The multimodal alignment module can be a dedicated computational unit for calculating the fine-grained correlation between image visual features and text semantic features. It can be used to generate a cross-modal consistency score, quantifying the degree of semantic consistency between the image and text, and supporting subsequent path adaptation determination. In this embodiment, the multimodal alignment module can align corresponding semantic regions or concepts in the image-text feature space based on a cross-modal attention mechanism or a contrastive learning objective. Furthermore, the multimodal alignment module can collaborate with image visual features and text semantic features as an input processing unit; and collaborate with the cross-modal consistency score as a score generator. The cross-modal consistency score can be a numerical indicator reflecting the degree of semantic consistency or conflict between the image and text, and can be used as one of the key dimensions for comprehensive evaluation of the current context, used to identify image-text inconsistencies (such as ironic illustrations). In an exemplary embodiment, the cross-modal consistency score can be output by the multimodal alignment module and calculated based on feature similarity, alignment confidence, or conflict detection signals.

[0124] User historical behavior records can be a collection of a user's historical interaction data on social media platforms, including their past emotional expression patterns and content preferences. This data can be used to provide personalized contextual information, supporting the extraction of user preference information and cultural background context. In one specific embodiment, user historical behavior records can retrieve historical posting, liking, commenting, and other behavioral logs from the platform database through a user-authorized interface. For example, user historical behavior records may include, but are not limited to, logs of emotional expression habits, topic attention trajectories, and the social circles of interacting objects, among others. User preference information can be an individualized emotional expression tendency or content preference pattern summarized from a user's historical behavior. This data can be used to adjust the emotional interpretation path to suit the expression style of a specific user. For example, user preference information may include, but is not limited to, one or more of the following: frequency of using sarcasm, preference for emotional intensity, and sensitivity to specific topics. Cultural background context can be a set of cultural pragmatic rules inferred based on a user's region, language, or community affiliation. This data can be used to assist in understanding the impact of culturally specific expressions (such as Chinese subtlety versus Western directness) on emotional polarity. For example, cultural context may include, but is not limited to, one or more of the following: high / low context cultural labels, dialect sentiment mapping rules, and subcultural symbol interpretation sets.

[0125] Real-time correlation calculation of image visual features and text semantic features is performed through a multimodal alignment module. This can be achieved by inputting image visual features and text semantic features into the multimodal alignment module and calculating their alignment degree in the semantic space. Furthermore, this operation can be implemented by using a cross-modal contrastive learning framework to calculate region-word matching scores, or by using a bidirectional attention mechanism to generate a fine-grained alignment matrix, thereby generating fine-grained image-text semantic correlation signals and supporting consistency quantification. Obtaining a cross-modal consistency score can be achieved by aggregating or weighted summing the correlation signals output by the multimodal alignment module. This operation can provide a quantitative indicator of image-text semantic consistency. User preference information and cultural context are obtained from a domain knowledge base and user historical behavior records. This can be achieved by querying the structured knowledge base and user behavior database in parallel to extract personalized contextual features. Furthermore, this operation can be achieved by indexing user IDs in their historical behavior logs and clustering preference patterns, or by combining IP addresses or language settings to infer cultural context, thereby introducing individual differences and cultural prior information.

[0126] Multidimensional fusion analysis can be a process of jointly modeling cross-modal consistency scores, user preference information, and cultural context. It can be used to generate a structured comprehensive assessment of the current context, providing a basis for path adaptability determination. In one specific embodiment, multidimensional fusion analysis can integrate multi-source heterogeneous contextual signals through weighted fusion, tensor splicing, or graph neural networks. The comprehensive assessment of the current context can be a multidimensional contextual representation formed by fusing cross-modal consistency, user preferences, and cultural context. It can be used to drive personalized adjustments to sentiment interpretation paths, improving the targeting of disambiguation. In an exemplary embodiment, the comprehensive assessment of the current context can be output by multidimensional fusion analysis, encoding the personalized and cultural contextual state of the current UGC content. Furthermore, the comprehensive assessment of the current context can be coordinated with sentiment interpretation path adaptability determination as a core input; and coordinated with path adjustment parameters to determine the direction of parameter generation.

[0127] Multi-dimensional fusion analysis of cross-modal consistency scores, user preference information, and cultural context can be achieved by mapping three types of heterogeneous signals to a unified context space and performing joint modeling. Furthermore, this operation can be implemented by dynamically adjusting the weights of each dimension using a gated fusion network or by constructing a heterogeneous graph neural network for relational reasoning, thereby generating a structured comprehensive assessment of the current context. Obtaining this comprehensive assessment can be achieved by outputting a vector or structure of the results of the multi-dimensional fusion analysis. This operation can provide contextual basis for path adaptability determination.

[0128] The sentiment interpretation path adaptability determination can be a process of evaluating the applicability of candidate sentiment interpretation paths under the comprehensive evaluation of the current context and generating adjustment instructions. It can be used to output path adjustment parameters to achieve dynamic path reconstruction. In a specific embodiment, the sentiment interpretation path adaptability determination can identify path branches that need to be strengthened, suppressed, or replaced by matching the compatibility between path semantics and the comprehensive evaluation of the current context.

[0129] The adaptability determination of sentiment interpretation paths based on the comprehensive evaluation of the current context and cross-modal joint representation vectors can assess the rationality of existing candidate sentiment interpretation paths in the current context. Further, this operation can identify the path parts that need optimization and generate adjustment instructions by calculating the cosine similarity between the path semantics and the comprehensive evaluation of the current context, or by using a classifier to determine whether the path needs adjustment. Obtaining path adjustment parameters can transform the adaptability determination results into specific structural adjustment or weight modification parameters. This operation can provide executable path modification instructions. Path adjustment parameters can be specific numerical or structural instructions used to guide the modification of candidate sentiment interpretation paths, and can be used to control the personalized adjustment direction and magnitude of the path. Personalized adjustment can be an operation to modify the structure or weights of the original candidate sentiment interpretation path based on the path adjustment parameters, and can be used to adapt the sentiment interpretation path to the current user's expression habits and cultural background.

[0130] Personalizing candidate sentiment interpretation paths based on path adjustment parameters can involve modifying node weights or substructures within the text structure representation graph. This can be achieved by dynamically activating / suppressing specific edges in the graph or inserting user-specific interpretation sub-paths, thereby generating personalized sentiment interpretation paths tailored to the current user. The adjustment basis can be interpretable metadata recording the contextual factors and decision logic upon which the path adjustment depends, enhancing the traceability and interpretability of the preliminary sentiment tendency determination results. For example, the adjustment basis can include, but is not limited to, one or more of user preference trigger logs, cultural rule reference records, and inconsistency detection evidence. Integrating candidate sentiment interpretation paths and adjustment basis to generate a preliminary sentiment tendency determination result can involve parsing the adjusted path semantics into sentiment polarity, along with the adjustment basis metadata. Further, this operation can determine the tendency through sentiment labels at the end nodes of the path or through weighted voting of all sentiment units on the path, thus outputting interpretable preliminary sentiment determination results.

[0131] Taking personalized irony recognition as an example, the cross-modal text sentiment analysis method based on visual knowledge graphs in this embodiment can be as follows: A user frequently uses irony to express dissatisfaction, and posts a picture of a "delicate cake" with the caption "The service is very thoughtful." The system discovers a conflict in the sentiment polarity between the image and text through a multimodal alignment module, obtaining a low consistency score; at the same time, it identifies a preference for frequently using irony from the user's historical behavior records, and combines this with the high-context cultural background in Chinese; multi-dimensional fusion analysis generates a comprehensive assessment of the current context emphasizing "high probability of irony"; the sentiment interpretation path adaptive determination generates path adjustment parameters accordingly, replacing the original "positive service evaluation" path with a "ironic negation" sub-path; finally, the adjusted path and the basis (such as "user's historical irony frequency is 85%" and "image-text polarity conflict") are integrated to generate a negative sentiment determination.

[0132] In one embodiment, the image visual features include object emotional attributes, scene emotional classification, and visual emotional intensity, while user preference information includes historical emotional expression habits, cultural symbol cognitive preferences, and social context sensitivity.

[0133] The object sentiment attribute can be the semantic features of emotional tendency carried by identifiable objects (such as people, animals, and objects) in an image. It can be used to provide fine-grained visual sentiment signals and support the detection of local semantic conflicts between images and text (such as a smiling face paired with negative text). In an exemplary embodiment, the object sentiment attribute can be obtained by semantic segmentation and sentiment annotation of target regions in the image. Furthermore, the object sentiment attribute can collaborate with other image visual features to participate in cross-modal sentiment alignment. Scene sentiment classification can be a sentiment category determination based on the overall atmosphere of the image, reflecting the macroscopic visual emotional tone. It can be used to characterize the global emotional context of the image and assist in judging the overall emotional tendency (such as a funeral scene vs. a celebration scene). In a specific embodiment, scene sentiment classification can be achieved by mapping the entire image to a predefined sentiment category space after high-level semantic encoding using a convolutional neural network. In this embodiment, scene sentiment classification and object sentiment attributes together constitute a multi-level emotional representation of the image. Furthermore, scene sentiment classification can include, but is not limited to, one or more of the following: joyful scenes, sad scenes, and tense / conflict scenes.

[0134] Visual emotional intensity can be a quantitative indicator of the strength of the emotion conveyed by an image. It can be used to adjust the weight of visual modalities in cross-modal fusion, avoiding excessive influence of weak emotional images on the judgment. For example, visual emotional intensity can be obtained by calculating the proportion of high-arousal regions in an image or the integral value of an emotional salience heatmap. In an exemplary embodiment, visual emotional intensity, along with scene emotional classification and object emotional attributes, is jointly input to the attention mechanism module to dynamically adjust the contribution of each sub-feature. Further, visual emotional intensity can include, but is not limited to, one or more of high-arousal visual signals, low-arousal visual signals, and neutral-intensity visual content. Historical emotional expression habits can be stable patterns or preferred styles of emotional expression in a user's past UGC content. They can be used as a basis for personalized disambiguation, identifying whether a user habitually uses irony, exaggeration, or subtlety. In this embodiment, historical emotional expression habits can be obtained by analyzing the user's historical published text and its corresponding emotional tag sequence modeling. Further, historical emotional expression habits can be temporally consistent with the current text and image content. For example, historical emotional expression habits can include, but are not limited to, one or more of irony usage frequency patterns, emotional polarity expression intensity preferences, and negation structure usage tendencies.

[0135] Cultural symbol cognitive preference can be a user's tendency to understand the emotional meaning associated with specific cultural symbols (such as colors, gestures, and totems), and can be used to support the accurate interpretation of cultural metaphorical content (such as the mourning meaning of white in the East vs. the West). In one specific embodiment, cultural symbol cognitive preference can be inferred from user geographic information, language settings, and historical interaction behavior. In this embodiment, cultural symbol cognitive preference forms a cross-cultural semantic mapping relationship with the emotional attributes of the object. Further, cultural symbol cognitive preference can include, but is not limited to, one or more of the following: color emotional mapping preference, religious / folk symbol emotional interpretation, and regional emoji understanding habits. Social context sensitivity can be a user's ability or tendency to adjust emotional expression rules in different social situations (such as public posts vs. private group chats), and can be used to model user adaptability to context norms, improving the recognition accuracy of situation-dependent emotional expressions. For example, social context sensitivity can be obtained by analyzing the statistical differences in emotional expression of users at different social levels (such as public homepages, friend circles, and private messages). In an exemplary embodiment, after matching social context sensitivity with the current content posting context, it is used to adjust the openness threshold of the emotional interpretation path. Furthermore, social context sensitivity can include, but is not limited to, one or more of the following: tendency to restrain emotions in public, tendency to express emotions in close-knit circles, and behavior of adjusting to topic sensitivity.

[0136] Taking cross-cultural satirical content recognition as an example, the cross-modal text sentiment analysis method based on visual knowledge graphs in this embodiment can be as follows: A user posts an image containing a "thumbs up" gesture with the caption "a statement expressing evaluation." The system extracts the image's visual features: the object's emotional attribute is "positive gesture," but the scene's emotional classification is "accident scene," indicating low visual emotional intensity. Simultaneously, it acquires user preference information: their cultural symbol cognitive preference shows they are from a certain region (the gesture has an insulting connotation), their historical emotional expression habits indicate frequent use of seemingly positive but actually derogatory irony, and high sensitivity to social context (currently a publicly discussed political topic post). Multi-dimensional fusion analysis determines that the image and text contain cultural metaphorical conflicts, triggering path adjustment by replacing the original positive interpretation path with a culturally specific negative interpretation path, ultimately outputting a negative sentiment tendency judgment.

[0137] In one embodiment, multiple cross-modal fusion weight parameters are aggregated using graph neural network propagation to output a cross-modal joint representation vector, including:

[0138] Interactively retrieve historical image and text alignment records.

[0139] Historical image-text alignment records can be statistical or instance records of sentiment semantic alignment relationships in previously labeled or automatically mined image-text pairs, which can be used to provide prior knowledge for calculating semantic-visual alignment strength. In an exemplary embodiment, historical image-text alignment records may include one or more of the following: manually labeled alignment logs, weakly supervised alignment sample libraries, cross-modal co-occurrence statistics tables, etc. Interactively obtaining historical image-text alignment records can be achieved by retrieving past image-text sentiment alignment instances from annotated databases or log systems. Furthermore, interactively obtaining historical image-text alignment records can be achieved by extracting alignment samples from manually labeled multimodal sentiment datasets or by automatically constructing alignment records using weakly supervised methods (such as co-occurrence statistics), thereby obtaining the prior knowledge required to construct semantic-visual alignment strength.

[0140] Based on historical image-text alignment records, multiple sets of semantic-visual alignment strengths are calculated among multiple candidate sentiment fragments.

[0141] The semantic-visual alignment strength can be a numerical indicator that quantifies the consistency of two candidate emotional segments at the semantic and visual levels. It can be used as edge weights in a cross-modal relationship propagation graph, reflecting the fine-grained correlation between text and image emotional expressions. In a specific embodiment, the semantic-visual alignment strength can include one or more of the following: ironic consistency strength, cultural symbol matching strength, and emotional polarity conflict strength. Calculating multiple sets of semantic-visual alignment strengths between multiple candidate emotional segments based on historical text-image alignment records can be achieved by statistically analyzing or learning the co-occurrence patterns and emotional consistency of different emotional segment pairs in historical records. For example, this operation can be achieved by using point mutual information (PMI) to calculate co-occurrence strength or by training a Siamese network to learn alignment similarity, thereby quantifying the fine-grained correlation between text-image emotional expressions and supporting graph structure construction.

[0142] A cross-modal relationship propagation graph is constructed using multiple sets of semantic-visual alignment strengths.

[0143] The cross-modal relationship propagation graph can be a directed or undirected graph structure built with candidate emotional segments as nodes and their semantic-visual alignment strength as edge weights. It can be used to model fine-grained associations between text and image sentiment expressions. In this embodiment, the cross-modal relationship propagation graph can calculate the semantic-visual alignment strength between each candidate emotional segment pair based on historical text-image alignment records, and use this as the edge weight to construct the graph. Furthermore, the cross-modal relationship propagation graph can work in conjunction with semantic-visual alignment strength as its structured carrier; it can also work in conjunction with enhanced fusion feature vectors as its topological basis. For example, the cross-modal relationship propagation graph can include one or more of the following: ironic alignment subgraph, cultural metaphor association subgraph, and emotional conflict suppression subgraph. Constructing the cross-modal relationship propagation graph using multiple sets of semantic-visual alignment strengths can be achieved by constructing a weighted graph with candidate emotional segments as nodes and alignment strength as edge weights. In a specific embodiment, this operation can be achieved by constructing a fully connected graph and pruning low-strength edges, or by connecting only strongly aligned segment pairs exceeding a threshold, thereby forming a structured propagation topology that encodes semantic associations of sentiment.

[0144] Multiple cross-modal fusion weight parameters are used as initial node feature inputs to the cross-modal relationship propagation graph, and multiple enhanced fusion feature vectors are obtained through graph convolution operations.

[0145] The initial node features can be the initial representations of each node input to the cross-modal relationship propagation graph, specifically referring to multiple cross-modal fusion weight parameters, which can be used as the starting features for graph convolutional propagation. In an exemplary embodiment, the initial node features can include one or more of the following: single-modal dominant features, conflict balancing features, and context-independent initial features. The enhanced fusion feature vector can be a context-aware node feature vector obtained by aggregating neighborhood information on the cross-modal relationship propagation graph through graph convolution operations. It can be used to enhance the semantic consistency and context sensitivity of the original fusion weights, improving the ability to discriminate ambiguous sentiment units. In this embodiment, the enhanced fusion feature vector can use the cross-modal fusion weight parameters as initial node features and perform multi-layer graph convolutional message passing on the cross-modal relationship propagation graph. Furthermore, the enhanced fusion feature vector can work in conjunction with the cross-modal fusion weight parameters as its context-enhanced version; it can also work in conjunction with the weighted fusion result as one of its inputs.

[0146] For example, the enhanced fusion feature vector can include one or more of the following: local neighborhood aggregation vector, global graph pooling enhancement vector, and multi-hop propagation enhancement vector. Inputting multiple cross-modal fusion weight parameters as initial node features into the cross-modal relationship propagation graph can be achieved by mapping the weight parameters output by the preceding model to the initial vectors of each node in the graph, thereby initiating the graph neural network propagation process. Obtaining multiple enhanced fusion feature vectors through graph convolution operations can be achieved by performing multiple rounds of neighbor information aggregation on the cross-modal relationship propagation graph. Furthermore, this operation can be implemented by using GCN for mean aggregation or by employing GAT to introduce an attention mechanism for weighted neighbors, thereby generating enhanced representations of fused contextual cues.

[0147] Multiple cross-modal fusion weight parameters and multiple enhanced fusion feature vectors are weighted and fused, and multiple candidate emotional segments are sorted according to the fusion results to obtain an emotional saliency ranking sequence.

[0148] The weighted fusion result can be a comprehensive feature obtained by linearly or nonlinearly combining the original cross-modal fusion weight parameters and the enhanced fusion feature vector. This feature can be used to balance local modal credibility and global contextual consistency for sentiment saliency ranking. In one specific embodiment, the weighted fusion result can include one or more of static weighted fusion, dynamic gated fusion, and residual enhanced fusion. The sentiment saliency ranking sequence can be an ordered list formed by ranking multiple candidate sentiment segments according to their dominance or credibility in the current textual context based on the weighted fusion result. This sequence can be used to identify the most likely sentiment interpretation path, suppress literal or misleading sentiment signals, and highlight implicit or dominant sentiments. In this embodiment, the sentiment saliency ranking sequence can be based on the comprehensive score after fusing the original weights and enhanced features, ranking candidate sentiment segments in descending order. Furthermore, the sentiment saliency ranking sequence can be used in conjunction with the weighted fusion result as its ranking output; or it can be used in conjunction with high-level semantic aggregation as its input sequence.

[0149] For example, the sentiment salience ranking sequence may include one or more of the following: a satirical dominant sequence, a literal sentiment suppression sequence, and a polysemous sentiment priority sequence. Weighted fusion of multiple cross-modal fusion weight parameters and multiple enhanced fusion feature vectors can be achieved by weighting the original weights and enhanced features. Furthermore, this operation can be implemented through dynamic fusion using a learnable gating mechanism or fixed-ratio linear interpolation, thereby balancing local modality judgments and global semantic consistency. Ranking multiple candidate sentiment segments based on the fusion result to obtain the sentiment salience ranking sequence can be achieved by arranging the candidate sentiment segments in descending order of fusion score. In an exemplary embodiment, this operation can be implemented by directly ranking based on scores or by introducing a ranking loss to optimize the ranking process, thereby identifying the dominant sentiment interpretation path and suppressing misleading signals.

[0150] Perform high-level semantic aggregation on the sentiment saliency ranking sequence and output a cross-modal joint representation vector.

[0151] High-level semantic aggregation can be an operation that structurally compresses or abstracts the sentiment saliency ranking sequence to generate a unified semantic vector, which can be used to output the final cross-modal joint representation vector while preserving the saliency hierarchy and logical dependencies. In a specific embodiment, high-level semantic aggregation can include one or more of attention pooling aggregation, ranking-aware Transformer encoding, and saliency-weighted summation. Performing high-level semantic aggregation on the sentiment saliency ranking sequence to output a cross-modal joint representation vector can be achieved by performing pooling or encoding operations on the ranking sequence to generate a unified vector. Furthermore, this operation can be implemented by using ranking position encoding + Transformer aggregation, or by weighted summation based on saliency weights, thereby outputting a joint semantic representation with embedded saliency hierarchy and logical dependencies.

[0152] In one embodiment, the path adjustment parameters are obtained by performing an adaptive determination of the sentiment interpretation path based on a comprehensive assessment of the current context and cross-modal joint representation vectors, including:

[0153] Interactively obtain multiple sentiment interpretation paths and their matching threshold ranges for multiple candidate sentiment fragments.

[0154] The emotional interpretation path adaptation threshold range can be a set of dynamic numerical intervals preset for each candidate emotional segment to measure whether its corresponding emotional interpretation path is applicable. This can provide a flexible judgment standard for path selection, avoiding misjudgments caused by fixed thresholds and supporting context-sensitive path activation. In this embodiment, the emotional interpretation path adaptation threshold range can be configured with multiple sets of threshold boundaries for different emotional expression modes (such as irony, exaggeration, and metaphor) through human-computer interaction or offline learning, forming a callable adaptation rule base. Furthermore, the emotional interpretation path adaptation threshold range can collaborate with candidate emotional segments: each segment is associated with a unique threshold range; and collaborate with the target adaptation threshold range: serving as its source pool. For example, the emotional interpretation path adaptation threshold range can include, but is not limited to, one or more of the following: irony activation threshold range, cultural metaphor triggering threshold range, and emotional intensity saturation threshold range.

[0155] The system interactively obtains multiple threshold ranges for adapting to various sentiment interpretation paths from multiple candidate sentiment fragments. This can be achieved through a human-computer interaction interface or configuration interface, binding multiple sets of threshold ranges to different sentiment fragments. Furthermore, this operation can be implemented by having domain experts annotate typical expression cases and derive threshold intervals, or by automatically optimizing threshold boundaries in the user feedback loop through reinforcement learning. This allows for the establishment of a dynamically invoked path adaptation rule base, enhancing system resilience.

[0156] The target adaptation threshold range is invoked based on the cross-modal joint representation vector to adapt threshold ranges from multiple sentiment interpretation paths.

[0157] The target adaptation threshold range can be the most relevant subset selected from multiple sentiment interpretation path adaptation threshold ranges based on the current cross-modal joint representation vector. This can be used to narrow the adaptation judgment space and improve the targeting and efficiency of path adjustment. In an exemplary embodiment, the target adaptation threshold range can be matched and recalled based on the semantic features of the cross-modal joint representation vector, finding the threshold range that is semantically closest to the current text and image content. Furthermore, the target adaptation threshold range can be a subset of the sentiment interpretation path adaptation threshold range, serving as a specific instance of its activation. Calling the target adaptation threshold range from multiple sentiment interpretation path adaptation threshold ranges based on the cross-modal joint representation vector can be achieved by using the cross-modal joint representation vector as the query key to retrieve the semantically most relevant subset from the threshold range library. Further, this operation can be implemented by matching threshold range meta-labels using vector similarity retrieval (such as FAISS) or by predicting the threshold category to be activated using a lightweight classifier, thereby accurately locating the adaptation boundary applicable to the current text and image content.

[0158] The current contextual comprehensive evaluation index is used to traverse the target adaptation threshold range to obtain the contextual deviation degree, which includes cultural context deviation degree, personal style deviation degree and social scene deviation degree.

[0159] The current context comprehensive evaluation index can be a multi-dimensional numerical vector that quantitatively represents the current context comprehensive evaluation. It is used to systematically traverse the threshold range and can serve as input for traversing the target adaptation threshold range, driving the calculation of context deviation. In this embodiment, the current context comprehensive evaluation index can transform the previously generated current context comprehensive evaluation (including user preferences, cultural background, consistency scores, etc.) into a structured index vector. For example, the current context comprehensive evaluation index may include, but is not limited to, cultural norm deviation indicators, individual expression baseline indicators, and social scene popularity indicators. Context deviation can be a quantitative result of the degree of deviation between the current context comprehensive evaluation index and the target adaptation threshold range. It can be used to reveal the difference between the current expression and the expected context, guiding the dynamic adjustment of path weights. In a specific embodiment, the context deviation can be calculated using distance metrics (such as Euclidean distance, KL divergence) or rule matching to determine the deviation between the index and the threshold boundary. Furthermore, the context deviation can include cultural context deviation, personal style deviation, and social scene deviation.

[0160] Cultural context deviation refers to the degree to which the current expression deviates from a preset cultural context threshold in terms of cultural norms. It can be used to identify whether culturally specific expressions (such as implicit negation in Chinese or direct irony in Western culture) exceed the boundaries of conventional interpretation. For example, cultural context deviation may include, but is not limited to, exceeding limits in high-context expressions, misuse of cross-cultural symbols, and deviation from regional emotional norms. Personal style deviation refers to the degree to which the current expression deviates from the style baseline modeled by the user's historical behavior in terms of individual habits. It can be used to determine whether the user is using unconventional rhetoric (such as suddenly using irony), requiring adjustment of the default interpretation strategy. For example, personal style deviation may include, but is not limited to, the emotional intensity mutation index, the degree of rhetorical device anomalousness, and the degree of expression consistency disruption.

[0161] Social context deviation refers to the degree to which current UGC content deviates from the norm of group expression in a specific social event or topic context. It can be used to capture unconventional emotional expressions in trending events (such as dark humor in disasters) and prevent misjudgment. For example, social context deviation may include, but is not limited to, topic sentiment polarity shift, group emotional synchronization deviation, and event sensitivity exceeding limits. The context deviation is obtained by iterating through the target adaptation threshold range using a comprehensive evaluation index of the current context. This can be achieved by comparing the comprehensive evaluation index of the current context with the upper and lower bounds of the target threshold range dimension by dimension, calculating the deviation amount. Furthermore, this operation can be achieved by calculating the standardized deviation of each dimension index from the median of the threshold, or by using fuzzy logic to determine whether the index falls into the "high deviation" region, thereby quantifying the degree of difference between the current expression and the expected norms of the context.

[0162] Based on the context deviation, the selection weight, confidence level, and adaptability of the fusion method of the sentiment interpretation path are determined to obtain path adjustment parameters.

[0163] The selection weights for sentiment interpretation paths can be activation priority values ​​assigned to each path among multiple candidate sentiment interpretation paths. These weights can be used to control the contribution ratio of each path in the final fusion, enabling dynamic routing. For example, the selection weights for sentiment interpretation paths may include, but are not limited to, the activation weight of the main path, the suppression coefficient of alternative paths, and the mutual exclusion factor for conflicting paths. Confidence level can be an assessment of the credibility of a sentiment interpretation path's correctness in the current context. It can be used to determine whether the path participates in the final sentiment determination and the strength of its participation. For example, confidence level may include, but is not limited to, semantic consistency confidence level, cultural compliance confidence level, and user style matching confidence level. The fusion method can be a combination strategy of multiple sentiment interpretation paths when generating the final determination. It can be used to determine whether the paths are weighted average, maximum voting, or conditional coverage.

[0164] The adaptive determination of the selection weights, confidence levels, and fusion methods for sentiment interpretation paths based on contextual deviation can be achieved by using contextual deviation as input and generating path adjustment parameters through a rule engine or a small neural network. This operation can be implemented by reducing the weight of general paths and increasing the confidence of culture-specific paths if the cultural context deviation is high, or by enabling a style-abnormal fusion mode (such as introducing paths from historically similar cases) if the personal style deviation is significant. This allows for fine-grained dynamic adaptation of path selection and fusion strategies. Obtaining path adjustment parameters can be achieved by outputting specific values ​​or instructions containing selection weights, confidence levels, and fusion methods, thus providing executable configurations for subsequent personalized path adjustments.

[0165] Furthermore, to achieve the above objectives, the present invention also provides a cross-modal text sentiment analysis device based on visual knowledge graphs. The device includes: a memory, a processor, and a cross-modal text sentiment analysis program based on visual knowledge graphs stored in the memory and executable on the processor. The cross-modal text sentiment analysis program based on visual knowledge graphs is configured to implement the steps of the cross-modal text sentiment analysis method based on visual knowledge graphs as described above.

[0166] Furthermore, to achieve the above objectives, the present invention also provides a medium storing a cross-modal text sentiment analysis program based on a visual knowledge graph, wherein when the cross-modal text sentiment analysis program based on a visual knowledge graph is executed by a processor, it implements the steps of the cross-modal text sentiment analysis method based on a visual knowledge graph as described above.

[0167] Other embodiments or specific implementations of the cross-modal text sentiment analysis device based on visual knowledge graph described in this invention can be referred to the above-described method embodiments, and will not be repeated here.

[0168] The above are merely preferred embodiments of the present invention and do not limit the scope of the patent. Any equivalent structural or procedural transformations made based on the description and drawings of the present invention, or direct or indirect applications in other related technical fields, are similarly included within the scope of patent protection of the present invention.

Claims

1. A method for cross-modal text sentiment orientation analysis based on a visual knowledge graph, characterized in that, The method includes: The system receives the text to be analyzed and associated image data, and performs multi-granular semantic parsing on the text to be analyzed to obtain a text structure representation map. The text structure representation map stores multiple candidate sentiment segments and multiple potentially ambiguous semantic units and their contextual dependencies. The image visual features and text semantic features are fused and analyzed to output a cross-modal joint representation vector, wherein the image visual features are extracted in real time through a multi-level convolutional neural network; After retrieving and matching candidate sentiment interpretation paths in the pre-constructed text structure representation graph based on the cross-modal joint representation vector, and combining the current context with dynamic disambiguation, a preliminary sentiment tendency determination result is generated. After receiving the domain sentiment dictionary and cultural context rules returned by the external knowledge base, the domain sentiment dictionary and cultural context rules are input into the context adaptation network for sentiment weight correction, and the sentiment intensity adjustment strategy is output. Real-time public opinion background data is obtained through social media interaction interfaces, and collaborative impact analysis is performed based on the real-time public opinion background data and the current cross-modal joint representation to output a tendency optimization strategy. The preliminary sentiment tendency determination result is updated using the aforementioned sentiment intensity adjustment strategy and tendency optimization strategy; a disambiguation verification window is preset, and the preliminary sentiment tendency determination result is subjected to adaptive iterative online consistency evaluation and optimization using the disambiguation verification window as a constraint.

2. The method for cross-modal text sentiment analysis based on visual knowledge graphs as described in claim 1, characterized in that, The fusion analysis of image visual features and text semantic features outputs a cross-modal joint representation vector, including: The system interactively obtains multiple sample visual representations, multiple sample text representations, and multiple sample sentiment tag records for the multiple candidate sentiment fragments. The sample visual representations include historical image sentiment annotations, visual object co-occurrence relationships, and scene sentiment mapping tables. A graph-text semantic complementarity model is constructed based on the attention mechanism; The multiple sample visual representations, multiple sample text representations, and multiple sample sentiment labels are recorded as training data to optimize the parameters of the image-text semantic complementarity model, thereby completing the localized deployment of the image-text semantic complementarity model. The image visual features and text semantic features are loaded into the image-text semantic complementarity model, and multiple cross-modal fusion weight parameters of the multiple candidate emotion segments are output. The multiple cross-modal fusion weight parameters are aggregated by graph neural network propagation to output the cross-modal joint representation vector.

3. The method for cross-modal text sentiment analysis based on visual knowledge graphs as described in claim 2, characterized in that, The process of receiving the text to be analyzed and associated image data, and performing multi-granular semantic parsing on the text to be analyzed to obtain a text structure representation map, includes: A preset semantic unit division granularity is used, and the text to be analyzed is segmented with the semantic unit division granularity as a constraint to obtain M text semantic segments, wherein the text semantic segments include lexical-level sentiment polarity, phrase-level sentiment combination and sentence-level sentiment tendency; After inputting the M text semantic fragments into a pre-trained language model for semantic encoding, M semantic dependency structures are obtained through dependency parsing. Cluster analysis and ambiguous unit detection are performed on the M semantic dependency structures to obtain the multiple potential ambiguous semantic units and their contextual dependencies. Based on the multiple potentially ambiguous semantic units, aggregate the M visual association regions and M historical sentiment annotation records corresponding to the M text semantic segments to obtain multiple sets of image-text sentiment association data; The multiple sets of image-text sentiment association data are subjected to ambiguity quantification assessment and feature extraction to obtain the text structure representation map.

4. The method for cross-modal text sentiment analysis based on visual knowledge graphs as described in claim 3, characterized in that, The process of performing ambiguity quantification and feature extraction on the multiple sets of image-text sentiment association data to obtain the text structure representation map includes: An ambiguity evaluation index system is preset, and the multiple sets of image-text sentiment association data are quantitatively scored using the ambiguity evaluation index system as the standard to obtain multiple sets of semantic unit ambiguity scores. Construct a semantic ambiguity evaluation matrix based on the multiple sets of semantic unit ambiguity scores; Principal component analysis and feature selection are performed on the semantic ambiguity evaluation matrix to generate the multiple potential ambiguous semantic units and their corresponding context dependencies. The multiple candidate sentiment fragments and their corresponding multiple potentially ambiguous semantic units, along with their contextual dependencies, are associated and stored to complete the construction of the text structure representation map.

5. The method for cross-modal text sentiment analysis based on visual knowledge graphs as described in claim 1, characterized in that, After retrieving and matching candidate sentiment interpretation paths in the pre-constructed text structure representation graph based on the cross-modal joint representation vector, and then dynamically disambiguating in the current context to generate a preliminary sentiment tendency determination result, the process includes: The multimodal alignment module calculates the correlation between image visual features and text semantic features in real time to obtain a cross-modal consistency score. User preference information and cultural context can be obtained by accessing domain knowledge bases and user history behavior records; A multi-dimensional fusion analysis is performed on the cross-modal consistency scores and user preference information with cultural background context to obtain a comprehensive assessment of the current context. Based on the current context comprehensive assessment and cross-modal joint representation vector, the emotion interpretation path adaptability determination is performed to obtain path adjustment parameters; After the candidate sentiment interpretation path is personalized according to the path adjustment parameters, the candidate sentiment interpretation path and the adjustment basis are integrated to generate the preliminary sentiment tendency judgment result.

6. The method for cross-modal text sentiment analysis based on visual knowledge graphs as described in claim 5, characterized in that, The image visual features include object emotional attributes, scene emotional classification, and visual emotional intensity; the user preference information includes historical emotional expression habits, cultural symbol cognitive preferences, and social context sensitivity.

7. The method for cross-modal text sentiment analysis based on visual knowledge graphs as described in claim 2, characterized in that, The step of performing graph neural network propagation and aggregation on the multiple cross-modal fusion weight parameters to output the cross-modal joint representation vector includes: Interactively retrieve historical image and text alignment records; Based on the historical image-text alignment records, multiple sets of semantic-visual alignment strengths among the candidate emotional segments are calculated; The cross-modal relationship propagation graph is constructed using the aforementioned multiple sets of semantic-visual alignment intensities; The multiple cross-modal fusion weight parameters are used as initial node features and input into the cross-modal relationship propagation graph. Multiple enhanced fusion feature vectors are obtained through graph convolution operation. The multiple cross-modal fusion weight parameters and multiple enhanced fusion feature vectors are weighted and fused, and the multiple candidate emotional segments are sorted according to the fusion results to obtain an emotional saliency ranking sequence; The emotional saliency ranking sequence is subjected to high-level semantic aggregation to output the cross-modal joint representation vector.

8. The method for cross-modal text sentiment analysis based on visual knowledge graphs as described in claim 5, characterized in that, The step of determining the path adaptation for sentiment interpretation based on the current context comprehensive assessment and cross-modal joint representation vector, and obtaining path adjustment parameters, includes: Interactively obtain multiple sentiment interpretation path adaptation threshold ranges for the multiple candidate sentiment fragments; The target adaptation threshold range is invoked from the multiple sentiment interpretation path adaptation threshold ranges based on the cross-modal joint representation vector; The current context comprehensive evaluation index is used to traverse the target adaptation threshold range to obtain the context deviation degree, wherein the context deviation degree includes cultural context deviation degree, personal style deviation degree and social scene deviation degree; Based on the aforementioned context deviation, the selection weight, confidence level, and adaptability of the fusion method for the sentiment interpretation path are determined to obtain path adjustment parameters.

9. A cross-modal text sentiment analysis device based on visual knowledge graph, characterized in that, The device includes: a memory, a processor, and a cross-modal text sentiment analysis program based on a visual knowledge graph, stored in the memory and executable on the processor, the cross-modal text sentiment analysis program based on a visual knowledge graph configured to implement the steps of the cross-modal text sentiment analysis method based on a visual knowledge graph as described in any one of claims 1 to 8.

10. A medium, characterized in that, The medium stores a cross-modal text sentiment analysis program based on a visual knowledge graph, which, when executed by a processor, implements the steps of the cross-modal text sentiment analysis method based on a visual knowledge graph as described in any one of claims 1 to 8.

Images