A method for mongolian sentiment analysis based on multi-modal feature balanced contrast learning

Abstract

This invention belongs to the field of natural language processing and discloses a Mongolian sentiment analysis method based on multimodal feature-balanced contrastive learning. The method collects and preprocesses Mongolian text and image data, then expands the samples through generative enhancement incorporating cultural constraints to construct a Mongolian multimodal sentiment analysis dataset. A sentiment analysis model is built, generating culturally bridging texts and constructing ternary contrastive sample groups. Cross-modal alignment is achieved through culturally constrained contrastive learning. The model parameters are optimized by fusing features with dual-dimensional confidence assessment and dynamic weights, combined with a joint loss function. The input data can then be used to output classification results. This invention effectively solves the problems of data sparsity, semantic gap, and fusion imbalance in low-resource scenarios, achieving high-precision and robust Mongolian multimodal sentiment recognition, and has good application prospects.

Description

Technical Field

[0001] This invention relates to the field of natural language processing technology, specifically to a Mongolian sentiment analysis method based on multimodal feature balance contrastive learning. Background Technology

[0002] Multimodal sentiment analysis is a core research task at the intersection of natural language processing and computer vision. By integrating information from multiple sources such as text and images to identify sentiment tendencies in data, it is widely applied in scenarios such as smart education and mental health assessment. Mongolian, as a typical low-resource minority language, offers valuable research opportunities for multimodal sentiment analysis tailored to its linguistic and cultural characteristics. This research has significant academic and applied value in promoting the upgrading of digital services in ethnic minority areas and facilitating the intelligent dissemination of ethnic culture.

[0003] Current multimodal sentiment analysis technologies for high-resource languages ​​are relatively well-developed. They typically rely on large-scale pre-trained models, achieving semantic alignment through cross-modal contrastive learning and completing sentiment classification through multi-scale feature fusion, demonstrating high recognition performance on standard datasets. Mongolian sentiment analysis research, however, remains primarily focused on unimodal text analysis, mainly mitigating the sparsity of labeled data through cross-linguistic transfer learning. The limited number of multimodal studies largely adopts general technical architectures without specifically optimizing for the cultural and semantic characteristics of Mongolian.

[0004] When existing solutions are applied to Mongolian language scenarios, they generally lack guidance and constraints at the cultural and semantic level, and cannot effectively bridge the cross-modal semantic gap between text and images under low resource conditions. This leads to problems such as missed detection of cultural metaphor features and large fusion deviations when modal quality is uneven. As a result, the overall accuracy and robustness of emotion recognition are difficult to meet the needs of practical applications. Summary of the Invention

[0005] To address the above technical problems, this invention provides a Mongolian sentiment analysis method based on multimodal feature balance contrastive learning, the method comprising the following steps: S1. Collect multimodal raw data of Mongolian text and images, preprocess them, expand the sample size using a generative data augmentation method that incorporates cultural constraints, divide them into training set, validation set and test set, and construct a Mongolian multimodal sentiment analysis dataset. S2. Based on the Mongolian multimodal sentiment analysis dataset, construct a Mongolian multimodal sentiment analysis model; S3. Input the training set into the Mongolian multimodal sentiment analysis model, extract the basic text features and image features, and generate bridging texts that integrate Mongolian cultural semantics based on the original Mongolian text and corresponding images in the training set. Construct a multimodal contrast sample group containing the original text, bridging text and images, complete cross-modal semantic alignment through contrastive learning with cultural semantic constraints, and output the aligned multimodal features. S4. Perform a two-dimensional confidence assessment on the multimodal features, dynamically allocate the fusion weights of each modality based on the single-sample assessment results, perform balanced fusion of the multimodal features based on the fusion weights to obtain fusion features, perform sentiment classification prediction based on the fusion features to obtain sentiment prediction results, construct a joint loss function based on the sentiment prediction results and the corresponding real sentiment labels, and iteratively train the Mongolian multimodal sentiment analysis model to complete the optimization of model parameters. S5. Input the Mongolian multimodal data to be detected into the trained Mongolian multimodal sentiment analysis model, output the sentiment classification results, and complete the sentiment statistics and analysis.

[0006] As a further improvement of the present invention, in step S1, the generative data augmentation method incorporating cultural constraints specifically includes the following steps: Based on the seed words of Mongolian culture, a prompt word sequence is constructed. A generative architecture combining a large language model, StableDiffusion, and ControlNet is adopted to perform bidirectional mutual refinement processing on three types of input: plain text, plain image, and image-text pairs. The quality of the generated samples was evaluated from three dimensions: text sentiment consistency, image cultural relevance, and modal semantic matching degree. Qualified samples were selected and included in the dataset.

[0007] As a further improvement of the present invention, in step S1, data preprocessing and dataset partitioning specifically include the following steps: Collect Mongolian text-image pairing data covering various emotional categories, cultural scenarios, and dialect expressions, and simultaneously collect single-modal text and single-modal image samples; The original text is processed by format encoding normalization, meaningless noise removal and semantic normalization, and the original image is processed by size scaling normalization, resolution unification and noise reduction enhancement to obtain standardized bimodal data. The selected samples are divided into training set, validation set and test set according to a preset ratio.

[0008] As a further improvement of the present invention, in step S3, the step of generating the bridging text that integrates Mongolian cultural semantics specifically includes: Extract emotional keywords and cultural feature words from the original Mongolian text, and extract visual elements and cultural scene description information from the image; Textual semantic information and visual image descriptions are input into a large language model that has been fine-tuned for Mongolian culture. Through prompting engineering, a bridging text that integrates surface semantics and deep cultural metaphors is generated. Cultural symbol weight constraints are introduced during the generation process to increase the semantic proportion of Mongolian-specific cultural symbols and ensure that the bridging text conforms to Mongolian expression habits and cultural context.

[0009] As a further improvement of the present invention, in step S3, comparative learning of cultural semantic constraints is introduced, specifically including the following steps: The semantic features of the original text and the bridging text are extracted by a Mongolian pre-trained BERT encoder, and the visual features of the image are extracted by a multilingual CLIP encoder. All features are then mapped to a unified shared embedding space through a projection layer. Construct a ternary comparison sample of original text, bridging text and image, define sample pairs that match the sentiment category as positive samples, and define sample pairs that do not match the sentiment category or are unrelated to the cultural scene as negative samples, and calculate the semantic distance between features based on cosine similarity; The contrastive learning loss is constructed using the InfoNCE contrastive loss function, and a Mongolian cultural semantic regularization term is introduced as a cultural semantic constraint. Through iterative training, the feature distance of positive samples is brought closer and the feature distance of negative samples is pushed further apart, thus completing cross-modal semantic alignment.

[0010] As a further improvement of the present invention, in step S4, the two-dimensional confidence assessment specifically includes the following steps: Both text and image modalities were evaluated using a dual-dimensional assessment of general confidence and cultural confidence. Under the general confidence dimension, the text modality is scored based on grammatical fluency and sentiment semantic consistency, while the image modality is scored based on image resolution and sentiment expression intensity. Under the cultural confidence dimension, the text modality is scored based on the coverage of Mongolian emotional vocabulary and cultural symbols, while the image modality is scored based on the detection rate of Mongolian cultural symbols and typical scenes. By integrating general confidence and cultural confidence, a comprehensive confidence score for each modality is obtained.

[0011] As a further improvement of the present invention, in step S4, the dynamic allocation of fusion weights and the equalization fusion of multimodal features specifically include the following steps: The overall confidence score of each modality is input into the Softmax function and normalized to obtain the dynamic fusion weight of the corresponding modality. The sum of all weights is 1. The aligned multimodal features are weighted and summed based on dynamic fusion weights to obtain preliminary fusion features; A cross-attention gating mechanism is introduced to perform deep interaction optimization on the initial fusion features, enhance the ability to capture complementary information between modalities, and output the final balanced fusion features.

[0012] As a further improvement of the present invention, in step S4, the construction of the joint loss function specifically includes the following steps: using InfoNCE contrastive loss to optimize cross-modal alignment accuracy, cross-entropy loss to optimize sentiment classification performance, and cultural semantic regularization term to constrain cultural feature distribution. The three types of loss terms are weighted and combined to form a joint loss function for iterative model training; The formula for calculating the joint loss function is as follows: In the formula, For cross-modal contrast loss, For sentiment classification, cross-entropy loss, For cultural semantic regularization, These are the weighting coefficients for the corresponding loss terms.

[0013] As a further improvement of the present invention, a dynamic adjustment step for loss weights is also included: The weight coefficients of each loss term are dynamically adjusted based on the detection results of the validation set. In the early stage of training, the weight of the contrastive loss is increased to strengthen cross-modal semantic alignment learning. In the middle and later stages of training, the weights of the classification loss and the cultural regularization term are increased to improve classification accuracy and cultural adaptation ability.

[0014] Based on this, the present invention also provides a Mongolian sentiment analysis system based on multimodal feature balance contrastive learning, the system being used to implement the above method, comprising: The data acquisition unit is used to collect raw multimodal data of Mongolian language, complete data preprocessing, and construct a generative data augmentation and sentiment analysis dataset incorporating cultural constraints. The model building unit is used to build a Mongolian multimodal sentiment analysis model based on the dataset; The cross-modal semantic alignment unit is used to generate bridging text that integrates cultural semantics based on the original Mongolian text and the corresponding image, construct a multimodal contrast sample set, and complete cross-modal feature alignment through contrastive learning with cultural semantic constraints. The multi-feature balanced fusion and training unit is used to complete the balanced fusion of multi-modal features through dual-dimensional confidence assessment and dynamic weight allocation, and to construct a joint loss function based on sentiment prediction results and real labels to complete model iterative training and parameter optimization. The result output unit is used to input the multimodal data to be detected and output the sentiment classification results, thus completing sentiment statistics and analysis.

[0015] Compared with the prior art, the beneficial effects of the present invention are as follows: This invention achieves high-precision and robust multimodal sentiment classification and recognition in low-resource Mongolian language scenarios by employing a generative data augmentation method incorporating cultural constraints, a triplet contrastive learning alignment method with cultural semantic regularization constraints, and a dynamic equilibrium fusion method driven by dual-dimensional confidence. Combined with a joint loss function that combines multiple loss terms with a phased dynamic weight optimization strategy, it effectively solves the core technical problems in existing technologies, such as sparse labeled data, significant cross-modal cultural semantic gaps, and imbalanced fusion features due to uneven modal quality. This invention has good practical value and application prospects. Attached Figure Description

[0016] To more clearly illustrate the technical solution of the present invention, the drawings used in the embodiments are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 This is a flowchart of a method according to an embodiment of the present invention; Figure 2 This is a diagram of the StableDiffusion+ControlNet model framework in an embodiment of the present invention; Figure 3 This is the original CLIP model framework diagram in the embodiments of the present invention; Figure 4 This is a diagram of the improved CLIP model framework in an embodiment of the present invention; Figure 5 This is a flowchart of the multi-feature equalization fusion module in an embodiment of the present invention; Figure 6 This is a detailed technical flowchart of an embodiment of the present invention; Figure 7 This is a system framework diagram in an embodiment of the present invention. Detailed Implementation

[0018] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0019] It should be noted that, unless otherwise defined, the technical or scientific terms used in the embodiments of this disclosure should have the ordinary meaning understood by one of ordinary skill in the art to which this disclosure pertains. The terms "first," "second," and similar words used in the embodiments of this disclosure do not indicate any order, quantity, or importance, but are merely used to distinguish different components. Terms such as "comprising" or "including" mean that the element or object preceding the word encompasses the elements or objects listed following the word and their equivalents, without excluding other elements or objects. Terms such as "connected" or "linked" are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. Terms such as "upper," "lower," "left," and "right" are used only to indicate relative positional relationships; when the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0020] First, some technical terms used in this invention will be explained: Multimodal sentiment analysis is a technology that integrates information from multiple modalities, such as text and images, to identify the sentiment tendencies of data. Compared with single-modal analysis, it has richer semantic information and stronger robustness.

[0021] Low-resource languages: Language types with limited labeled data and scarce corpus resources.

[0022] Generative data augmentation: This method, which uses large language models and generative image models to synthesize new samples to expand the dataset size, can effectively alleviate the data sparsity problem in low-resource scenarios.

[0023] Bridging text: Intermediate text that integrates textual semantics and image visual information to bridge the semantic distance between cross-modal features and reduce the semantic gap between text and images.

[0024] Contrastive learning: A self-supervised method for feature learning that brings positive sample features closer together and negative sample features further apart, which can effectively improve cross-modal alignment accuracy.

[0025] Two-dimensional confidence: An index system that evaluates the credibility of modal information from two dimensions: general quality and cultural semantics, and is used to drive dynamic weight fusion.

[0026] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0027] Example 1 As the background technology shows, Mongolian is a low-resource minority language with scarce labeled multimodal data. Furthermore, existing general multimodal sentiment analysis methods are not adapted to the cultural and semantic characteristics of Mongolian, resulting in significant cross-modal cultural semantic gaps, uneven modal quality that easily leads to fusion bias, and insufficient sentiment recognition accuracy and robustness, making it difficult to meet the needs of practical applications.

[0028] Please see Figures 1 to 6 This invention provides a Mongolian sentiment analysis method based on multimodal feature balance contrastive learning, which includes the following steps: S1. Collect multimodal raw data of Mongolian text and images, preprocess them, expand the sample size using a generative data augmentation method incorporating cultural constraints, divide them into training set, validation set and test set, and construct a Mongolian multimodal sentiment analysis dataset.

[0029] First, we collected Mongolian text-image pairs covering various emotional categories, cultural scenarios, and dialect expressions. Simultaneously, we collected unimodal text and unimodal image samples to enrich the scenario coverage and sample diversity of the dataset and reduce the risk of model overfitting.

[0030] Preferably, the emotion categories cover seven discrete emotions: happiness, anger, sadness, surprise, fear, disgust, and neutral. Preferably, the cultural scenarios include typical Mongolian language application scenarios such as Naadam festivals, herders' daily lives, folk rituals, and school education. Preferably, the text samples cover the expression forms of different dialect areas such as the Oirat dialect of Inner Mongolia and the Bargu-Buryat dialect, to improve the model's ability to generalize dialect variations.

[0031] Based on this, the original multimodal data is standardized and preprocessed to eliminate format differences and noise interference, thereby improving the quality of the input data.

[0032] Preferably, the text preprocessing performs format encoding normalization, meaningless noise removal and semantic standardization, unifying Uyghur Mongolian and Cyrillic Mongolian into a standard encoding format, removing garbled characters, special symbols and meaningless interjections, and unifying the synonymous emotional expressions of different dialects through a dialect synonym mapping table. More preferably, the image preprocessing performs size scaling and normalization, resolution unification and noise reduction enhancement processing, adjusts all image sizes to 224×224 pixels, uses Gaussian bilateral filtering algorithm to suppress image noise, and preserves key details such as cultural symbols and facial expressions.

[0033] To further expand the sample size and alleviate the sparsity problem of low-resource language data, a generative data augmentation method incorporating cultural constraints is adopted to augment the preprocessed data, ensuring the emotional consistency and cultural adaptability of the generated samples.

[0034] Preferably, generative data augmentation specifically includes the following steps: The first step involves constructing a prompt word sequence based on Mongolian cultural seed words. These seed words include cultural symbols with emotional connotations, such as Naadam, Khata, Suluding, yurt, and wrestling, ensuring that the generation process is integrated into the Mongolian cultural context. The prompt word generation process is implemented based on a conditional probability model, the expression of which is: in, The final generated sequence of Mongolian cultural prompts, such as "Naadam festival, herdsmen cheer, hada (ceremonial scarf)"; Input data: plain text Pure Image Or text and image pair<T,I> It is the semantic basis for the generation of prompt words; The i-th cue word in the cue word sequence, such as "Hada" in the sequence; The (i-1)th cue word in the cue word sequence, i.e., the preceding context, ensures the semantic coherence of the sequence; : Input and preceding prompts Under the joint constraints, the probability of generating the i-th prompt word.

[0035] The second step involves using a large language model combined with a StableDiffusion and ControlNet generative architecture to perform bidirectional refinement processing on three types of input: plain text, plain image, and image-text pairs. When inputting plain text, the emotional expression of the text is refined through a large language model and corresponding cultural scene images are generated through StableDiffusion. When inputting plain images, Mongolian text matching the image scene is generated through a large language model and the image pose and structure are refined through StableDiffusion combined with ControlNet. When inputting images and text, the text expression and image details are optimized in both directions synchronously.

[0036] ControlNet prevents generated content from deviating from the inherent form of Mongolian culture by locking in key pose and structural information of cultural scenes.

[0037] The third step involves evaluating the quality of the generated samples across three dimensions: text sentiment consistency, image cultural relevance, and modal semantic matching. A comprehensive quality assessment formula is used to calculate a score for each pair of generated data, and only qualified samples with scores exceeding a preset threshold are included in the dataset. The comprehensive quality assessment formula is as follows: in, Q The overall quality score for the generated data, ranging from 0 to 10 points; , , The initial value of each evaluation dimension is set to 1 / 3, and then optimized through 5-fold cross-validation to ensure that the contribution of each dimension is balanced. The sentiment consistency score is calculated by comparing the generated enhanced text with the input sentiment label. For example, if the input label is "joy", the smaller the deviation of the enhanced text, the higher the score. Image cultural relevance score; To score the modal semantic matching degree, the CLIP model is used to calculate the semantic similarity between text and image. For example, the text "Naadam cheers" matches the image containing "herdsmen cheers and colorful flags" better and gets a higher score.

[0038] After data augmentation and filtering, all samples were labeled with sentiment categories and divided into training, validation and test sets for model training, parameter tuning and performance verification, respectively.

[0039] Preferably, the dataset is divided into training, validation and test sets in a ratio of 7:2:1; preferably, the annotation process is performed by two people for cross-checking to remove invalid samples with annotation errors or ambiguous sentiment, so as to ensure the quality of the dataset annotation.

[0040] S2. Based on the Mongolian multimodal sentiment analysis dataset, construct a Mongolian multimodal sentiment analysis model.

[0041] After completing the dataset construction, a Mongolian multimodal sentiment analysis model is further built. Customized architectural improvements are made to address the shortcomings of general multimodal models in low-resource Mongolian language scenarios, such as poor cultural adaptability and imbalanced fusion. Preferably, the Mongolian multimodal sentiment analysis model sequentially includes a feature extraction layer, a cultural bridging and alignment layer, a two-dimensional balanced fusion layer, and a sentiment classification output layer. The feature extraction layer extracts basic semantic features from text and images respectively; the cultural bridging and alignment layer generates culturally adapted bridging text and completes cross-modal semantic alignment; the two-dimensional balanced fusion layer performs two-dimensional confidence assessment and dynamic weight balanced fusion; and the sentiment classification output layer outputs the final sentiment classification result.

[0042] S3. Input the training set into the Mongolian multimodal sentiment analysis model to extract the basic text features and image features. Based on the original Mongolian text and corresponding images in the training set, generate bridging text that integrates Mongolian cultural semantics. Construct a multimodal comparison sample group containing the original text, bridging text and images. Complete cross-modal semantic alignment by introducing contrastive learning with cultural semantic constraints, and output the aligned multimodal features.

[0043] After the model is built, the training set data is first input into the model, the basic features are extracted through the feature extraction layer, the bridging text is generated through the cultural bridging alignment layer and the contrastive learning alignment is performed, and the aligned multimodal features are output.

[0044] First, the basic features of the text and image are extracted through the feature extraction layer, providing a unified feature input for subsequent alignment and fusion.

[0045] Preferably, text feature extraction uses BERT-Mongolian, pre-trained on a large-scale Mongolian corpus, as the encoder to semantically encode the input Mongolian text, outputting a d-dimensional text feature vector that captures the contextual semantic information of emotional words and cultural symbols. Its core self-attention calculation expression is: in, , For input embedding, The key dimension is used. This formula ensures that long-distance dependencies in Mongolian are captured.

[0046] Preferably, the image feature extraction uses the multilingual CLIP as the visual encoder to perform forward propagation on the input image, extracting image basic feature vectors of dimension d, and mapping the visual features to a text-compatible semantic space. Its core VisionTransformer embedding expression is: Where Patch(image)∈RN×P divides the image into N fixed-size patches, and Emb(·)∈RN×d is the patch embedding projection. ∈R(N+1)×d represents the positional encoding, CLS∈R1×d represents the classification label, ViT(·) represents multi-layer Transformer encoding, and the output is the feature vector corresponding to the CLS label. ∈Rd.

[0047] After completing the basic feature extraction, a bridging text that integrates Mongolian cultural semantics is generated based on the original Mongolian text and corresponding images in the training set. This serves as an intermediary carrier for cross-modal semantics, bridging the semantic gap between text and images.

[0048] Preferably, the generation of bridging text specifically includes the following steps: extracting sentiment keywords and cultural feature words from the original Mongolian text, and simultaneously extracting visual elements and cultural scene description information from the image through visual recognition; inputting the text semantic information and image visual description together into a large language model fine-tuned for Mongolian culture, and guiding the model to generate bridging text that encompasses both surface semantics and deep cultural metaphors through preset prompt word templates. The generation of bridging text is based on a conditional probability model, expressed as: in, For the generated bridging text sequence, For the original text, For the first Each word. The formula quantifies the generation probability and incorporates cultural constraints. For example, given the input text "Naadam Festival" and the image "horse racing scene", the generated bridging text is: "At the Naadam Festival, which is full of grassland culture, riders drive their horses in fierce competition, showing their enthusiasm and boldness."

[0049] Cultural symbol weight constraints are introduced during the generation process to increase the semantic proportion of Mongolian-specific cultural symbols in the generated text, ensuring that the bridging text conforms to Mongolian expression habits and cultural context. The formula for adjusting cultural weights is as follows: Where C represents the set of cultural symbols. , For balance coefficient, The similarity between words and cultural symbols is used. This formula enhances the cultural adaptability of the generated text through cultural constraints. Bridging text not only integrates the surface semantics of text and images, but also supplements the deep semantics through a set of cultural symbols (such as "Suluding"), bridging the semantic gap in low-resource scenarios using traditional methods.

[0050] After obtaining the bridging text, a multimodal contrast sample set containing the original text, bridging text, and image is constructed. Cross-modal semantic alignment is achieved through contrastive learning that introduces cultural semantic constraints.

[0051] Preferably, the contrastive learning alignment specifically includes the following steps: The first step involves inputting the original text and bridging text into the same text encoder to obtain the original text features and bridging text features, respectively. Then, the image is input into the image encoder to obtain image features. All features are mapped to a unified shared embedding space through a projection layer. The projection layer expression is: in and For learnable parameters, This represents the feature representation after projection.

[0052] The second step is to construct a ternary comparison sample of original text, bridging text, and image. Text-image pairs and bridging text-image pairs with matching sentiment categories and consistent cultural scenes are defined as positive samples, while sample pairs with mismatched sentiment categories and unrelated cultural scenes are defined as negative samples. The semantic distance between features is calculated based on cosine similarity.

[0053] The cosine similarity expression is: in, and These represent feature representations from different modalities (text, augmented text, and image). By maximizing the similarity between positive sample pairs while minimizing the similarity between negative sample pairs, the model can learn semantically consistent and culturally sensitive cross-modal aligned representations.

[0054] The third step involves constructing a contrastive learning loss using the InfoNCE contrastive loss function and introducing a Mongolian cultural semantic regularization term as a cultural semantic constraint. Through iterative training, the distance between positive sample features is narrowed and the distance between negative sample features is widened, thus completing cross-modal semantic alignment.

[0055] The expression for InfoNCE contrastive loss is: in, For cosine similarity, For temperature coefficient, This represents the number of samples in the batch.

[0056] This objective function strengthens semantic aggregation within the same sentiment category by maximizing the similarity between the text and the enhanced text; at the same time, it minimizes the similarity between the text and image features to avoid visual noise interfering with the sentiment semantic space, thereby establishing a stable language-centered sentiment representation.

[0057] Considering that Mongolian emotional expression is heavily influenced by cultural symbols, a cultural semantic guidance regularization term will be introduced during the comparative learning process to enhance the model's sensitivity to cultural semantics and emotional intensity. Let the Mongolian cultural and emotional lexicon be a set. Each term The embedding vector is Define cultural and emotional constraints: This constraint ensures that samples containing Mongolian cultural symbols or emotional keywords are aggregated in the shared space, maintaining cultural consistency and the stability of emotional expression.

[0058] The comprehensive loss function is defined as: in, , which is a weighting coefficient used to balance the impact of cross-modal semantic consistency and cultural constraints.

[0059] Through multiple rounds of iterative training, the multimodal features ultimately achieve more accurate semantic alignment (especially the alignment of Mongolian-specific cultural symbols) in the "shared feature space," outputting the aligned feature vector. For use by the fusion module.

[0060] The cultural semantic regularization term is constructed by calculating the cosine similarity between the sample features and the embedding vectors of the Mongolian cultural sentiment lexicon, and its expression is: In the formula, The average embedding vector of the Mongolian cultural sentiment lexicon. This is the cosine similarity function.

[0061] S4. Perform a two-dimensional confidence assessment on the multimodal features, dynamically allocate the fusion weights of each modality based on the single-sample assessment results, perform balanced fusion of the multimodal features based on the fusion weights to obtain fused features, perform sentiment classification prediction based on the fused features to obtain sentiment prediction results, construct a joint loss function based on the sentiment prediction results and the corresponding real sentiment labels, and iteratively train the Mongolian multimodal sentiment analysis model to complete the optimization of model parameters.

[0062] After obtaining the aligned multimodal features, confidence evaluation and dynamic weight fusion are performed through a two-dimensional balanced fusion layer, and then the prediction results are obtained through the sentiment classification output layer. Finally, the model iterative training and parameter optimization are completed based on the joint loss function.

[0063] First, the confidence of the aligned multimodal features is evaluated in two dimensions: general and cultural. This quantifies the information quality and cultural semantic carrying capacity of each modality, providing a basis for subsequent weight allocation.

[0064] Preferably, the two-dimensional confidence assessment specifically includes the following steps: performing a two-dimensional assessment of general confidence and cultural confidence for both the text modality and the image modality.

[0065] The general confidence expression for text is: in, Represents the text's grammar score; Score for emotional consistency; , For weights.

[0066] The general confidence expression for an image is: in, This represents the image quality score; Score the intensity of emotional expression; , As weight.

[0067] The expression for textual cultural confidence is: in, The coverage rate of cultural vocabulary in the text; Scoring is given for cultural metaphorical expression. , As weight.

[0068] The expression for image culture confidence is: Wherein, in the formula The detection score for cultural symbols in the image. To score for cultural consistency in the setting, , As weight.

[0069] The overall confidence score for each modality is obtained by weighted summation and merging of general and cultural confidence scores. The overall confidence scores for text and images are as follows: in, For the general confidence level of the corresponding modality, The cultural confidence level for the corresponding modality. To adjust the weights.

[0070] After obtaining the overall confidence level of each modality, the fusion weights of each modality are dynamically allocated based on the single-sample evaluation results, and then the balanced fusion of multimodal features is performed.

[0071] Preferably, the dynamic weight allocation and balanced fusion specifically includes the following steps: inputting the comprehensive confidence score of the text and image modalities into the Softmax function, normalizing it to obtain the dynamic fusion weight of the corresponding modality, and the sum of all weights is 1 to ensure the rationality of the weight allocation.

[0072] The formula for calculating dynamic weights is: in, For the first Dynamic fusion weights for each modality This represents the overall confidence score for the corresponding modality.

[0073] The aligned multimodal features are weighted and summed based on dynamic fusion weights to obtain preliminary fused features. First, a linear mapping is performed on the features of each modality: in, Representing modes The original characteristics, and These are the weight matrix and bias term for this mode, respectively.

[0074] Then, a weighted average method is used to aggregate the features from different modalities to generate the final fused features: A cross-attention gating mechanism is introduced to perform deep interaction optimization on the initial fused features, capture complementary semantic information between modalities, enhance the discriminativeness of the fused features, and output the final balanced fused features.

[0075] After obtaining the balanced fusion features, they are input into the sentiment classification output layer for sentiment classification prediction to obtain the sentiment prediction result.

[0076] Preferably, the sentiment classification output layer uses a Transformer classification head structure, mapping the fused features to a probability distribution corresponding to the sentiment category, and selecting the category with the highest probability as the final sentiment prediction result. The sentiment classification probability expression is: in, It is a classification weight matrix. It is a bias term. This represents the probability distribution for sentiment classification. The softmax function transforms the fused feature vector into a probability distribution of multiple sentiment categories, and the model ultimately selects the category with the highest probability as the sentiment prediction.

[0077] Based on this, a joint loss function is constructed according to the sentiment prediction results and the corresponding real sentiment labels. The model is iteratively trained with the goal of minimizing the loss to complete the optimization of the model parameters.

[0078] Preferably, sentiment classification uses the cross-entropy loss function to optimize the classification boundary, as expressed in: in, For real labels, This represents the predicted sentiment category probability. The cross-entropy loss function is used to calculate the difference between the predicted result and the true label, optimizing the model parameters to improve classification accuracy.

[0079] Preferably, the method further includes a dynamic adjustment step for loss weights: dynamically adjusting the weight coefficients of each loss term based on the detection results of the validation set; increasing the contrastive loss weight in the early stage of training to enhance cross-modal semantic alignment learning; and increasing the classification loss weight and cultural regularization term weight in the middle and later stages of training to improve classification accuracy and cultural adaptation capabilities, thus adapting to the optimization needs of different training stages.

[0080] S5. Input the Mongolian multimodal data to be detected into the trained Mongolian multimodal sentiment analysis model, output the sentiment classification results, and complete the sentiment statistics and analysis.

[0081] After the model training and optimization are completed, the Mongolian multimodal data to be detected is input into the trained model to complete forward inference and result output, and sentiment statistics and scenario-based application output are also implemented.

[0082] Preferably, the model output results are first filtered by confidence level, retaining valid prediction results with confidence levels higher than a preset threshold, and outputting detection results that include emotion category, confidence level, and corresponding text and image information. Preferably, the distribution of emotion categories and the trend of emotion tendency changes in a specified scenario are statistically analyzed at a preset period to generate a standardized emotion analysis statistical report, providing a basis for decision-making in scenarios such as teaching evaluation. Preferably, when the proportion of negative emotions exceeds a preset threshold, an abnormal warning message can be generated and pushed to the management terminal to meet the warning needs of scenarios such as mental health monitoring.

[0083] This embodiment expands the training samples in low-resource scenarios by incorporating a generative data augmentation method with cultural constraints, and bridges the cross-modal cultural semantic gap by combining a triplet contrastive learning alignment method with cultural semantic regularization constraints. Furthermore, it alleviates the fusion bias caused by uneven modal quality through a dynamic equilibrium fusion method driven by dual-dimensional confidence. It also provides a complete set of standardized data preprocessing, joint loss function dynamic optimization and sentiment statistics early warning output process, which effectively solves the core problems of data sparsity, poor cultural adaptability and insufficient fusion accuracy in existing Mongolian multimodal sentiment analysis technology. It has good practical value and application prospects.

[0084] It should be noted that the method of this disclosure embodiment can be executed by a single device, such as a computer or server. The method of this embodiment can also be applied to a distributed scenario, where multiple devices cooperate to complete the task. In such a distributed scenario, one of these devices may execute only one or more steps of the method of this disclosure embodiment, and the multiple devices will interact with each other to complete the method described.

[0085] It should be noted that the above description describes some embodiments of this disclosure. Other embodiments are within the scope of the appended claims. In some cases, it should be understood that the sequence number of each step in the above embodiments does not imply the order of execution; the execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention. The actions or steps recorded in the claims can be performed in a different order than that in the above embodiments and can still achieve the desired result. In addition, the processes depicted in the drawings do not necessarily require a specific or sequential order to achieve the desired result. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0086] Example 2 Based on the same inventive concept, for methods corresponding to any of the above embodiments, please refer to [link / reference needed]. Figure 7 This disclosure also provides a Mongolian sentiment analysis system based on multimodal feature balance contrastive learning, the system comprising: The data acquisition unit is used to collect raw multimodal data of Mongolian language, complete data preprocessing, and construct a generative data augmentation and sentiment analysis dataset incorporating cultural constraints. The model building unit is used to build a Mongolian multimodal sentiment analysis model based on the dataset; The cross-modal semantic alignment unit is used to generate bridging text that integrates cultural semantics based on the original Mongolian text and the corresponding image, construct a multimodal contrast sample set, and complete cross-modal feature alignment through contrastive learning with cultural semantic constraints. The multi-feature balanced fusion and training unit is used to complete the balanced fusion of multi-modal features through dual-dimensional confidence assessment and dynamic weight allocation, and to construct a joint loss function based on sentiment prediction results and real labels to complete model iterative training and parameter optimization. The result output unit is used to input the multimodal data to be detected and output the sentiment classification results, thus completing sentiment statistics and analysis.

[0087] The system described in the above embodiments is used to implement the Mongolian sentiment analysis method based on multimodal feature balance contrastive learning in any of the foregoing embodiments, and has the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0088] It should be noted that the Mongolian sentiment analysis system based on multimodal feature balance contrastive learning described above is embodied in the form of functional units. The term "module" here can be implemented in software and / or hardware, without specific limitations.

[0089] For example, a "module" can be a software program, a hardware circuit, or a combination of both that implements the above functions. The hardware circuit may include an application-specific integrated circuit (ASIC), electronic circuitry, a processor (e.g., a shared processor, a proprietary processor, or a group processor) and memory for executing one or more software or firmware programs, integrated logic circuitry, and / or other suitable components that support the described functions.

[0090] This disclosure is intended to cover all such substitutions, modifications, and variations that fall within the broad scope of the appended claims. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A Mongolian sentiment analysis method based on multimodal feature balance contrastive learning, characterized in that, The method includes the following steps: S1. Collect multimodal raw data of Mongolian text and images, preprocess them, expand the sample size using a generative data augmentation method that incorporates cultural constraints, divide them into training set, validation set and test set, and construct a Mongolian multimodal sentiment analysis dataset. S2. Based on the Mongolian multimodal sentiment analysis dataset, construct a Mongolian multimodal sentiment analysis model; S3. Input the training set into the Mongolian multimodal sentiment analysis model, extract the basic text features and image features, and generate bridging texts that integrate Mongolian cultural semantics based on the original Mongolian text and corresponding images in the training set. Construct a multimodal contrast sample group containing the original text, bridging text and images, complete cross-modal semantic alignment through contrastive learning with cultural semantic constraints, and output the aligned multimodal features. S4. Perform a two-dimensional confidence assessment on the multimodal features, dynamically allocate the fusion weights of each modality based on the single-sample assessment results, perform balanced fusion of the multimodal features based on the fusion weights to obtain fusion features, perform sentiment classification prediction based on the fusion features to obtain sentiment prediction results, construct a joint loss function based on the sentiment prediction results and the corresponding real sentiment labels, and iteratively train the Mongolian multimodal sentiment analysis model to complete the optimization of model parameters. S5. Input the Mongolian multimodal data to be detected into the trained Mongolian multimodal sentiment analysis model, output the sentiment classification results, and complete the sentiment statistics and analysis.

2. The method according to claim 1, characterized in that, In step S1, the generative data augmentation method incorporating cultural constraints specifically includes the following steps: Based on the seed words of Mongolian culture, a prompt word sequence is constructed. A generative architecture combining a large language model, StableDiffusion, and ControlNet is adopted to perform bidirectional mutual refinement processing on three types of input: plain text, plain image, and image-text pairs. The quality of the generated samples was evaluated from three dimensions: text sentiment consistency, image cultural relevance, and modal semantic matching degree. Qualified samples were selected and included in the dataset.

3. The method according to claim 1, characterized in that, In step S1, data preprocessing and dataset partitioning specifically include the following steps: Collect Mongolian text-image pairing data covering various emotional categories, cultural scenarios, and dialect expressions, and simultaneously collect single-modal text and single-modal image samples; The original text is processed by format encoding normalization, meaningless noise removal and semantic normalization, and the original image is processed by size scaling normalization, resolution unification and noise reduction enhancement to obtain standardized bimodal data. The selected samples are divided into training set, validation set and test set according to a preset ratio.

4. The method according to claim 1, characterized in that, In step S3, the specific steps for generating the bridging text that integrates Mongolian cultural semantics are as follows: Extract emotional keywords and cultural feature words from the original Mongolian text, and extract visual elements and cultural scene description information from the image; Textual semantic information and visual image descriptions are input into a large language model that has been fine-tuned for Mongolian culture. Through prompting engineering, a bridging text that integrates surface semantics and deep cultural metaphors is generated. Cultural symbol weight constraints are introduced during the generation process to increase the semantic proportion of Mongolian-specific cultural symbols and ensure that the bridging text conforms to Mongolian expression habits and cultural context.

5. The method according to claim 1, characterized in that, In step S3, contrastive learning of cultural semantic constraints is introduced, specifically including the following steps: The semantic features of the original text and the bridging text are extracted by a Mongolian pre-trained BERT encoder, and the visual features of the image are extracted by a multilingual CLIP encoder. All features are then mapped to a unified shared embedding space through a projection layer. Construct a ternary comparison sample of original text, bridging text and image, define sample pairs that match the sentiment category as positive samples, and define sample pairs that do not match the sentiment category or are unrelated to the cultural scene as negative samples, and calculate the semantic distance between features based on cosine similarity; The contrastive learning loss is constructed using the InfoNCE contrastive loss function, and a Mongolian cultural semantic regularization term is introduced as a cultural semantic constraint. Through iterative training, the feature distance of positive samples is brought closer and the feature distance of negative samples is pushed further apart, thus completing cross-modal semantic alignment.

6. The method according to claim 1, characterized in that, In step S4, the two-dimensional confidence assessment specifically includes the following steps: Both text and image modalities were evaluated using a dual-dimensional assessment of general confidence and cultural confidence. Under the general confidence dimension, the text modality is scored based on grammatical fluency and sentiment semantic consistency, while the image modality is scored based on image resolution and sentiment expression intensity. Under the cultural confidence dimension, the text modality is scored based on the coverage of Mongolian emotional vocabulary and cultural symbols, while the image modality is scored based on the detection rate of Mongolian cultural symbols and typical scenes. By integrating general confidence and cultural confidence, a comprehensive confidence score for each modality is obtained.

7. The method according to claim 1, characterized in that, In step S4, the dynamic allocation of fusion weights and the equalization fusion of multimodal features specifically include the following steps: The overall confidence score of each modality is input into the Softmax function and normalized to obtain the dynamic fusion weight of the corresponding modality. The sum of all weights is 1. The aligned multimodal features are weighted and summed based on dynamic fusion weights to obtain preliminary fusion features; A cross-attention gating mechanism is introduced to perform deep interaction optimization on the initial fusion features, enhance the ability to capture complementary information between modalities, and output the final balanced fusion features.

8. The method according to claim 7, characterized in that, In step S4, the construction of the joint loss function specifically includes the following steps: InfoNCE contrastive loss is used to optimize cross-modal alignment accuracy, cross-entropy loss is used to optimize sentiment classification performance, and cultural semantic regularization term is used to constrain the distribution of cultural features. The three types of loss terms are weighted and combined to form a joint loss function for iterative model training; The formula for calculating the joint loss function is as follows: In the formula, For cross-modal contrast loss, For sentiment classification, cross-entropy loss, For cultural semantic regularization, These are the weighting coefficients for the corresponding loss terms.

9. The method according to claim 1, characterized in that, It also includes a step for dynamically adjusting the loss weights: The weight coefficients of each loss term are dynamically adjusted based on the detection results of the validation set. In the early stage of training, the weight of the contrastive loss is increased to strengthen cross-modal semantic alignment learning. In the middle and later stages of training, the weights of the classification loss and the cultural regularization term are increased to improve classification accuracy and cultural adaptation ability.

10. A Mongolian sentiment analysis system based on multimodal feature balance contrastive learning, the system being used to implement the method described in any one of claims 1-9, characterized in that, It includes: The data acquisition unit is used to collect raw multimodal data of Mongolian language, complete data preprocessing, and construct a generative data augmentation and sentiment analysis dataset incorporating cultural constraints. The model building unit is used to build a Mongolian multimodal sentiment analysis model based on the dataset; The cross-modal semantic alignment unit is used to generate bridging text that integrates cultural semantics based on the original Mongolian text and the corresponding image, construct a multimodal contrast sample set, and complete cross-modal feature alignment through contrastive learning with cultural semantic constraints. The multi-feature balanced fusion and training unit is used to complete the balanced fusion of multi-modal features through dual-dimensional confidence assessment and dynamic weight allocation, and to construct a joint loss function based on sentiment prediction results and real labels to complete model iterative training and parameter optimization. The result output unit is used to input the multimodal data to be detected and output the sentiment classification results, thus completing sentiment statistics and analysis.

Images