Method and system for generating minority image description based on memory mechanism and cognitive calculation

Abstract

The invention relates to the cross technical field of computer vision and natural language processing, and discloses a minority image description generation method based on a memory mechanism and cognitive calculation, which comprises the following steps of: extracting visual features through ConvNeXt, constructing national culture knowledge, executing a multi-level cognitive strategy on an input image, and carrying out national feature weighted fusion, so as to obtain the minority image description generation method based on the memory mechanism and the cognitive calculation. The method comprises the following steps of: obtaining a cognitive reasoning result, combining national culture knowledge with the cognitive reasoning result, constructing a logic cue word, splicing the logic cue word, visual features and the cognitive reasoning result, and inputting the spliced logic cue word, visual features and cognitive reasoning result into a multi-modal large-scale language model GLM-4V based on a Transform architecture to generate a description text. According to the method, high-precision understanding and description generation of cultural symbols, clothing utensils and scene semantics in minority images are achieved by fusing a dynamic generation strategy of a multi-modal large language model, and finally a natural language description text with accuracy, richness and culture depth is output.

Description

Technical Field

[0001] This invention relates to the field of computer vision and natural language processing, specifically to a method and system for generating image descriptions of ethnic minorities based on memory mechanisms and cognitive computing. Background Technology

[0002] While current image caption generation technology has made significant progress in general scenarios, its application in ethnic minority cultural scenarios still faces the following key challenges: Data scarcity and cultural annotation bottlenecks: Images of ethnic minorities possess high cultural specificity, requiring annotations that encompass fine-grained semantic information such as clothing patterns, religious artifacts, and traditional customs. However, existing public datasets (such as COCO and Flickr30k) lack such annotations, and manual annotation is costly, leading to severe data sparsity problems for supervised learning methods. Research shows that when the distribution of training data differs significantly from that of the target domain (such as images of ethnic minorities), model performance will decline exponentially.

[0003] Limitations of shallow semantic modeling: Traditional encoder-decoder architectures (such as CNN-LSTM) rely on local visual features (such as color and texture), making it difficult to model the abstract semantic relationships of cultural symbols (such as "Naxi people's clothing worn under the stars and moon - symbolizing - diligence"). Although attention mechanisms and graph neural networks attempt to improve relational reasoning capabilities, their feature space is still limited by low-order visual signals, making it impossible to achieve deep alignment of cultural connotations.

[0004] Modal gap and illusion risk of zero-shot methods: Zero-shot methods (such as CLIP-guided GPT-2) generate descriptions through cross-modal alignment, but due to the lack of cultural prior knowledge, they are prone to semantic deviation.

[0005] Insufficient domain generalization ability: Mainstream models (such as BLIP and GRIT) are pre-trained on general datasets, and their parameter spaces differ significantly from the high-dimensional semantic distribution of ethnic minority images. Experiments show that such models achieve a recognition accuracy of less than 17% on a test set of 55 ethnic minority images, and the descriptive text often confuses similar cultural elements (such as Tibetan and Qiang clothing).

[0006] Limitations of existing technologies: Recent studies have attempted to alleviate the above problems through memory enhancement or multimodal large models (GPT-4o), but the following shortcomings still exist: Static memory retrieval: MeaCap relies on a fixed text library and cannot dynamically adapt to the diversity of ethnic cultural semantics; Coarse-grained alignment: Although multimodal large models (such as GPT-4o) have strong generation capabilities, they lack fine-grained cultural symbol alignment mechanisms, resulting in low description consistency (CIDEr≤24.7); Broken reasoning chain: Existing methods do not simulate the hierarchical reasoning process in human cognition (target → attribute → relationship), making it difficult to generate descriptions that conform to cultural logic. Summary of the Invention

[0007] To address the shortcomings of existing technologies, this invention provides a method and system for generating descriptions of ethnic minority images based on memory mechanisms and cognitive computing. This method and system have the advantages of high-precision understanding and description of cultural symbols, clothing, objects, and scene semantics in ethnic minority images, thus solving the aforementioned technical problems.

[0008] To achieve the above objectives, the present invention provides the following technical solution: a method for generating image descriptions of ethnic minorities based on memory mechanisms and cognitive computing, comprising the following steps: S1: Input image and preprocess it: extract visual features using ConvNeXt; S2: Vectorize the visual features and retrieve the most relevant semantic triples from the constructed abstract semantic memory database of ethnic minority images as the corresponding ethnic cultural knowledge; S3: Execute a multi-level cognitive strategy on the input image and perform weighted fusion of ethnic features to obtain the cognitive reasoning result; S4: Combine the ethnic cultural knowledge in S2 with the cognitive reasoning results in S3 to construct logical prompts. Then, concatenate the logical prompts, visual features, and cognitive reasoning results and input them into the GLM-4V multimodal large-scale language model based on the Transformer architecture to generate descriptive text. S5: Perform a consistency check on the description text and logical prompts, and determine whether the consistency check passes. If the consistency check passes, S5 ends; otherwise, optimization is performed.

[0009] As a preferred technical solution of the present invention, the specific steps of constructing the ethnic minority image abstract semantic memory database in step S2 are as follows: S2.1: Using the Open Information Extraction (OpenIE) model, the subject, predicate, and object triplets are structurally extracted from the text describing images of ethnic minorities. The specific expression is as follows: in, They represent the subject, predicate, and object, respectively. This indicates the input text. This represents a semantic relation parsing tool used to extract structured semantic triples from text T; S2.2: The Word2Vec model is used to encode the elements of triples into low-dimensional dense vectors, which are then concatenated into a unified high-dimensional vector representation. The specific expression is as follows: in, This represents a triple vector, which is the concatenation of vectors for subject S, predicate P, and object O. It is used to represent the overall semantic features of cultural knowledge. This represents vector concatenation. These represent word vectors for the subject, predicate, and object, respectively.

[0010] As a preferred technical solution of the present invention, the cosine similarity retrieval mechanism is used in S2 for searching the abstract semantic memory database of ethnic minority images. By calculating the cosine similarity, search results below a preset similarity threshold are filtered out. The specific expression is as follows: in, Represents cosine similarity. The semantic vector representing the triples in the memory bank is provided by a pre-constructed abstract semantic memory bank of ethnic minority images. The query vector represents the semantic representation of the visual features of the input image after vectorization. Denotes the Euclidean norm; The abstract semantic memory database of ethnic minority images is dynamically expanded through a contrastive learning loss function, the specific expression of which is as follows: in, Represents an exponential function. Represents the logarithmic function. This represents the cosine similarity between the query vector and the positive sample vector. This represents the cosine similarity between the query vector and the k-th negative sample vector. This represents the candidate vector for the k-th positive sample. Indicates temperature parameter, This represents the k-th negative sample vector. This represents the summation operation on the set of negative samples.

[0011] As a preferred embodiment of the present invention, the multi-level cognitive strategy in S3 includes: Target recognition layer: Extracts visual feature vectors from the input image using Faster R-CNN. in, This refers to the Faster R-CNN detector. Indicates the input image. Represents the entity's bounding box; Attribute analysis layer: Extracts attribute features through the ConvNeXt network. in The fine-grained characteristics representing the detected entities are the core basis for the cultural salience of relational reasoning layer modeling. Representing the general characteristics of entities, Used to parse out specific attributes; Relational Reasoning Layer: By modeling the cultural semantic associations and spatial topological relationships between entities, a structured cognitive map representing ethnic minority cultural scenes is generated. This addresses the problem that shallow visual features cannot capture the logical combination of cultural symbols. Graph Neural Networks (GNNs) are used to update node features. , , Output results , Specifically as follows: in, Represents the features of the l-th layer nodes. Represents the set of neighboring nodes. , These represent the layer parameters, For Gaussian error linear units, This indicates a feature concatenation operation. For entities The spatial relative position vector, The output of the substantial dependency attribute analysis layer is used to construct semantic association features between nodes; The specific expression for the weighted fusion of ethnic characteristics in S3 is as follows: in, Weighting coefficients representing spatial relationships Weight coefficients representing semantic relations Indicates the result of cognitive reasoning. Indicates spatial relationships, Indicates semantic relationships.

[0012] As a preferred embodiment of the present invention, the expression for constructing the logical prompt word in S4 is as follows: in, Indicates logical prompt words, Indicates the result of cognitive reasoning. This represents a dynamic semantic fusion function. Represents a semantic vector of cultural knowledge; The specific expression for concatenating logical prompts, visual features, and cognitive reasoning results in S4 is as follows: in, express, This represents the visual features extracted by ConvNeXt. Indicates logical prompt words, Indicates the result of cognitive reasoning. These represent encoding the results of cognitive reasoning and logical prompts, respectively.

[0013] The present invention also provides a system for generating image descriptions of ethnic minorities based on memory mechanisms and cognitive computing, for performing the above-described method for generating image descriptions of ethnic minorities based on memory mechanisms and cognitive computing; Compared with existing technologies, this invention provides a method and system for generating image descriptions of ethnic minorities based on memory mechanisms and cognitive computing, which has the following beneficial effects: (1) This invention constructs an abstract semantic memory bank of ethnic minority images, designs a multi-level cognitive reasoning framework, and integrates a dynamic generation strategy of multimodal large language model to achieve high-precision understanding and description of cultural symbols, clothing and objects and scene semantics in ethnic minority images, and finally outputs natural language description text with accuracy, richness and cultural depth. (2) This invention designs a multi-level framework that simulates human visual cognition. Through dynamic attention and relational reasoning, it achieves progressive processing from low-level target recognition to high-level semantic completion, effectively capturing details of ethnic characteristics in images (such as clothing patterns and the uses of objects), and solving the problem of insufficient high-level semantic understanding in traditional methods.

[0014] (3) To verify the effectiveness of the multi-level cognitive framework of this invention in capturing ethnic feature details, this paper designed three sets of quantitative experiments on the self-built MN-28k dataset (55 ethnic groups, 28,000 clothing images) and compared them with the baseline model. Attached Figure Description

[0015] Figure 1 This invention provides a framework for generating image descriptions of ethnic minority costumes. Figure 2 A comparative diagram showing the description results of different models of ethnic minority costumes is provided. Figure 3 A comparative diagram of the various models across the BLEU series of indicators; Figure 4 This is a schematic diagram of a contrast experiment on hallucination inhibition. Figure 5 This is a schematic diagram of the process of the present invention; Figure 6 This is a schematic diagram of the comparative experiment of the present invention. Detailed Implementation

[0016] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0017] Please see Figures 1-6 A method for generating image descriptions of ethnic minorities based on memory mechanisms and cognitive computing includes the following steps: S1: Input image and preprocess it: extract visual features using ConvNeXt; S2: Vectorize the visual features and retrieve the most relevant semantic triples from the constructed abstract semantic memory database of ethnic minority images as the corresponding ethnic cultural knowledge; S3: A multi-level cognitive strategy is implemented on the input image, and weighted fusion of ethnic features is performed to obtain the cognitive reasoning result. This simulates the hierarchical reasoning process of human visual cognition (target → attribute → relationship → semantic fusion), solving the problems of shallow feature dependence, semantic fragmentation, and lack of cultural association in traditional methods for understanding images of ethnic minorities. Through multi-level collaborative modeling, a progressive understanding from low-level visual features to high-level cultural semantics is achieved. S4: Combining the ethnic cultural knowledge from S2 and the cognitive reasoning results from S3, logical prompts are constructed. These logical prompts, visual features, and cognitive reasoning results are then concatenated and input into the Transformer-based multimodal large-scale language model GLM-4V to generate descriptive text. This solution uses the Transformer-based multimodal large-scale language model GLM-4V as the core generation engine. GLM-4V achieves efficient alignment between visual and textual modalities through pre-training, possesses cross-modal reasoning and generation capabilities, and can jointly process image features and textual semantics. To address the specific needs of ethnic minority cultural scenarios, this invention deeply adapts and optimizes GLM-4V: Through domain-specific fine-tuning, it utilizes a small number of labeled ethnic minority image-text pairs (e.g., "Naxi people's clothing worn under the stars and moon - symbolizing - diligence") to adjust model parameters, enhancing its fine-grained semantic understanding of cultural symbols. A knowledge injection mechanism is introduced, using dynamically retrieved semantic triples from the memory bank (e.g., "silver ornaments - belonging to - festive attire") as external cultural priors. These are embedded into the intermediate layer of GLM4V through a lightweight Adapter module, achieving real-time fusion of cultural knowledge without altering the core model structure. This effectively suppresses illusions caused by missing domain knowledge (e.g., mistakenly describing "Qiang cloud shoulder" as "Tibetan wool"). This strategy enables GLM-4V to balance general generation capabilities with cultural specificity, accurately associating clothing patterns, religious artifacts, and their deep symbolic meanings in generated descriptions, ultimately outputting natural language descriptions that combine accuracy and cultural depth. S5: Perform a consistency check on the description text and logical prompts, and determine whether the consistency check passes. If the consistency check passes, S5 ends; otherwise, optimization is performed.

[0018] The specific steps for constructing the ethnic minority image abstract semantic memory database using S2 are as follows: S2.1: The Open Information Extraction (Open IE) model is used to structurally extract subject-verb-object triples from the text describing images of ethnic minorities. This method accurately captures the fine-grained semantic relationships of cultural symbols in the images (e.g., "clothing-symbol-diligence"). Compared to traditional methods that rely on unstructured text, triple extraction achieves explicit semantic decomposition and organization, solving the problem of abstract semantic modeling of ethnic cultural symbols. This step provides semantically clear and logically sound input for subsequent vectorization and retrieval, significantly improving the standardization and interpretability of cultural knowledge representation. The specific expression is as follows: in, They represent the subject, predicate, and object, respectively. Represents the input text, denotes a semantic relation parsing tool for extracting structured semantic triples from text T; S2.2: Use the Word2Vec model to encode the triple elements into low-dimensional dense vectors and concatenate them into a unified high-dimensional vector representation. Use the Word2Vec model to encode the triple elements (subject, predicate, object) into low-dimensional dense vectors and concatenate them into a unified high-dimensional vector representation, achieving numerical modeling of semantics. This step solves the problem that the discrete symbolic representation of cultural symbols (such as text labels) is difficult to directly use in machine learning. At the same time, through the semantic similarity of word vectors (such as the vector proximity between "traveling day and night" and "industrious"), the relevance between cultural symbols is implicitly modeled. The specific expression is as follows: Where, represents the triple vector, which is the result of concatenating the vectors of the subject S, predicate P, and object O, and is used to represent the overall semantic features of cultural knowledge, represents vector concatenation, respectively represent the word vectors of the subject, predicate, and object.

[0019] In S2, the cosine similarity retrieval mechanism is adopted for retrieving in the ethnic minority image abstract semantic memory bank. By calculating the cosine similarity, the retrieval results below the preset similarity threshold are filtered. Based on the cosine similarity retrieval mechanism, the triple most relevant to the semantics of the input image can be quickly matched from the memory bank, solving the problem that the traditional static knowledge base cannot dynamically adapt to cultural diversity. By setting the similarity threshold (such as 0.85), the low-confidence results are filtered, and only the triples with similarity greater than the threshold are retained to avoid noise interference. The specific expression is as follows: Where, represents the cosine similarity, represents the semantic vector of the triple in the memory bank, provided by the pre-constructed ethnic minority image abstract semantic memory bank, represents the query vector, which is the semantic representation after the visual features of the input image are vectorized and transformed, represents the Euclidean norm; When there is no matching result in the triple knowledge base retrieved by the input image (i.e., the highest similarity ), or when the consistency verification module (S5) detects that there are cultural fact errors in the generated description, the extension process is triggered. The specific expression is as follows: Where, represents the exponential function, represents the logarithmic function, This represents the cosine similarity between the query vector and the positive sample vector. This represents the cosine similarity between the query vector and the k-th negative sample vector. This represents the candidate vector for the k-th positive sample. Indicates temperature parameter, This represents the k-th negative sample vector. This represents the summation operation on the set of negative samples.

[0020] The multi-level cognitive strategies in S3 include: Target recognition layer: Extracts visual feature vectors from the input image using Faster R-CNN, and then uses Faster R-CNN to detect core ethnic cultural entities (such as clothing, religious artifacts, and human actions) in the image. in, express, express, Indicates the input image. Represents the entity's bounding box; Attribute Analysis Layer: Extracts attribute features through the ConvNeXt network, including color, texture, and shape, to capture subtle differences in cultural symbols (such as distinguishing between "fish patterns" and "bird patterns" in Miao embroidery). in, The fine-grained characteristics representing the detected entities are the core basis for the cultural salience of relational reasoning layer modeling. Representing the general characteristics of entities, Used to parse out specific attributes; Relational Reasoning Layer: By modeling the cultural semantic associations and spatial topological relationships between entities, a structured cognitive map representing minority cultural scenes is generated, solving the problem that shallow visual features cannot capture the logical combination of cultural symbols. Graph Neural Networks (GNNs) are used to update node features, with the input... , , Output results , The details are as follows: in, Represents the features of the l-th layer nodes. Represents the set of neighboring nodes. , These represent the layer parameters, Using Gaussian error linear units (GELUs), an adaptive gating mechanism is employed to enhance the modeling ability of implicit rules in ethnic cultures. This indicates a feature concatenation operation. For entities The spatial relative position vector, The output of the substantial dependency attribute analysis layer is used to construct semantic association features between nodes. The specific expression for the weighted fusion of ethnic characteristics in S3 is as follows: in, Weighting coefficients representing spatial relationships Weight coefficients representing semantic relations Indicates the result of cognitive reasoning. Indicates spatial relationships, Indicates semantic relationships.

[0021] The expression for constructing logical prompts in S4 is as follows: in, These are logical prompts, which contain ethnic information, entity descriptions, and abstract semantics (e.g., "Ethnic information: Naxi, Entity information: woman, riverside, Abstract semantics: working under the stars and moon symbolizes diligence"). This represents a dynamic semantic fusion function. Represents a semantic vector of cultural knowledge; The specific expression for concatenating logical prompts, visual features, and cognitive reasoning results in S4 is as follows: in, express, This represents the visual features extracted by ConvNeXt. Indicates logical prompt words, Indicates the result of cognitive reasoning. These respectively represent encoding the results of cognitive reasoning and logical prompts; Cross-modal alignment and feature fusion: Utilizing GLM4v's multi-head self-attention mechanism to align visual and textual modalities. Query (Q): Generated by prompt word encoding, focusing on cultural and semantic needs.

[0022] Key (K), Value (V): Derived from image features With semantic representation Joint encoding preserves visual details and cultural relevance. Attention weights are used to dynamically allocate the importance of visual and textual features. This mechanism enables the model to adaptively integrate details of clothing patterns (such as "Dongba script embroidery") with abstract cultural knowledge (such as "symbolizing the spirit of diligence"). Text generation: Based on attention fusion features, word-by-word descriptive text is generated through autoregression. .

[0023] Consistency check: Calculate the generated text With prompt words If the semantic similarity is below a threshold, the following optimizations are triggered; Search Enhancement: Retrieve more relevant triples from the search memory and update search suggestions. .

[0024] Weight adjustment: Reduce the attention weight of low-confidence features to suppress noise interference.

[0025] Cross-entropy loss: Optimizes the generation probability distribution to ensure language fluency. Contrastive learning constraints: By comparing positive samples (human-annotated descriptions) with negative samples (disruptive text with randomly replaced cultural symbols), the model is forced to align with cultural semantics, reducing illusions.

[0026] This invention also provides a system for generating image descriptions of ethnic minorities based on memory mechanisms and cognitive computing, and a method for generating image descriptions of ethnic minorities based on memory mechanisms and cognitive computing in execution. This invention proposes a large model guidance method based on logical prompt words, which transforms knowledge from memory into structured input. It enhances the understanding of ethnic semantics by multimodal large language models through modal alignment technology, significantly improves the semantic consistency of generated descriptions (ACC index reaches 81.91%), reduces illusion phenomenon, and achieves efficient synergy between domain knowledge and large model generation capabilities. The application prospects of this invention are as follows: Digital preservation of ethnic minority cultures: It can be applied to the automated description generation of images of ethnic minorities, providing technical support for the digitization of collections of museums and cultural institutions, realizing the efficient recording and dissemination of cultural elements such as clothing, artifacts, and customs, and helping to protect and inherit intangible cultural heritage.

[0027] Smart education and cultural tourism services: In the field of ethnic cultural education, it generates image descriptions containing cultural connotations to assist in textbook writing and teaching resource development; in cultural tourism scenarios, it provides accurate multilingual descriptions for the image promotion and guide systems of ethnic minority scenic spots, enhancing the depth and interest of cultural experiences.

[0028] Multimodal interaction and artificial intelligence applications: Embedded as a core technology module in multimodal systems such as smart photo albums and visual question answering, it enhances the ability to understand images of ethnic minorities, promotes the application of cross-modal interaction technology in vertical fields, and has broad commercial application value and social significance.

[0029] Technological scalability: The methodology can be transferred to other vertical fields (such as images of cultural relics and scenes with regional characteristics). By building a dedicated memory bank and cognitive strategies, it can solve the problem of knowledge gaps in image descriptions in specific fields, and has significant technical universality and expansion potential.

[0030] The experimental procedure of this invention is as follows: To address the issues of data scarcity and annotation quality in the field of ethnic minority costumes, this study constructed the MN-28k dataset. A joint annotation team comprised of five ethnic culture research experts and three computer vision researchers was formed. Experts were required to pass an ethnic costume knowledge assessment (pass rate ≥ 95%). A hybrid generation-correction paradigm was adopted. Initial descriptions were generated from 1100 random samples using GPT4o and GLM4V. The annotation team then performed a triple verification process based on authoritative literature: "ethnic attribute correction - detailed feature supplementation - cultural symbol filtering." For example, the descriptions of Miao silver ornaments were refined, and cross-ethnic symbol misuse was eliminated. Fleiss's Kappa coefficient was used to evaluate annotation consistency, achieving Kappa values ​​of 0.89 for entity-level annotations and 0.82 for statement-level annotations. The final dataset contains 28,000 images from 55 ethnic groups, each accompanied by five manually calibrated multi-granular descriptions. The training / validation / test sets were divided in a 6:2:2 ratio to ensure a balanced sample size across ethnic groups (minimum number of ethnic group samples ≥ 150).

[0031] In this study, the present invention compares two types of baselines. 1) Traditional encoder-decoder image caption generation methods: SmallCap uses a pre-trained CLIP visual encoder and a GPT-2 language decoder, adding only a trainable cross-attention layer between them, and retrieves image-related text from external data storage as generation cues. ViECap is a zero-shot image captioning model proposed by Fei et al., whose core solves the modal bias and object illusion problems of traditional methods through entity-aware hard cues and cross-modal alignment mechanisms. 2) Image captioning using multimodal large models: GPT4o captures cross-modal semantic associations through hundreds of billions of parameters, and can understand cultural scenes in images of ethnic minorities without explicit entity cues. DeepSeekVL adopts a hybrid expert architecture, including visual Transformer and language model components, and achieves joint image-text representation through a cross-modal contrastive learning framework. GLM4v adopts a visual encoder-text decoder architecture and achieves image-text feature space alignment through contrastive learning. CogvLM2 employs a deep fusion architecture of a 5 billion-parameter visual encoder and a 7 billion-parameter visual expert module, supporting 8K resolution image input and multi-turn conversational description generation. QwenVL uses an image encoder-language decoder architecture, supporting image, text, and bounding box inputs, achieving state-of-the-art (SOTA) performance for zero-shot image description on the Flickr30K and NoCaps datasets. YiVL uses a ViT visual encoder + projection module architecture, progressively optimizing visual feature extraction from low to high resolution, resulting in stronger detail capture capabilities in images.

[0032] The evaluation system of this study includes two dimensions: 1) Descriptive accuracy: The improved CHAIR index (Caption Hallucination Assessment with Image-Text alignment) is used to quantify the hallucination phenomenon, and the entity-level hallucination rate (CHAIR-i) and sentence-level hallucination rate (CHAIR-s) are defined as follows: 2) Cross-modal alignment: Constructing a visual-text similarity metric (VTS) based on CLIP-H / 14: in and These represent the visual encoder and text encoder of CLIP, respectively. Table 1 shows the comparative experimental results of each model in terms of illusion suppression.

[0033] Figure 4A comparative experiment on hallucination suppression. This experiment compared the performance of different models in terms of entity-level hallucination rate (CHAIR-i), statement-level hallucination rate (CHAIR-s), and cross-modal alignment (VTS). The framework in this paper (MnCap) reduced CHAIR-i / CHAIR-s by 56.7% / 63.4% and improved VTS by 20.9% compared with the best baseline, respectively, verifying the effectiveness of the memory-cognition synergy mechanism in hallucination suppression.

[0034] Experimental results show that the proposed framework (MnCap) significantly outperforms baseline models in hallucination suppression and cross-modal alignment, such as... Figure 4 As shown, compared to GPT4o, CHAIR-i and CHAIR-s are reduced by 56.7% and 63.4% respectively, while visual-text similarity (VTS) is improved by 20.9%. This breakthrough stems from the synergistic effect of multiple modules—the memory bank suppresses cross-cultural misjudgments through cultural semantic constraints (such as pre-set "Torch Festival-Chalwa-Wool Dyeing Process" triples), the cognitive computing module uses graph attention networks to analyze the spatial topological relationships between clothing components (such as the wearing position association between silver ornaments and collars), and the multimodal alignment mechanism enhances the fine-grained mapping between visual details and text descriptions through cross-domain feature projection. Taking Miao silver ornament images as an example, this framework not only accurately describes the "double dragons playing with a pearl" pattern of the silver ornaments (CHAIR-i=0), but also fully restores its cultural context of "festival attire" through the cross-modal correlation between the density of silver bubbles and the embroidery on the lapel (VTS=0.89). In contrast, GPT4o, lacking domain knowledge, misclassified the silver ornaments as ordinary metal decorations (VTS=0.62) and incorrectly associated them with the Dai Water Splashing Festival custom (CHAIR-s=0.41). Quantitative analysis shows that the memory mechanism contributed 38.6% to the VTS increase (VTS=0.72 after ablation), and the cognitive computing module contributed 29.4% (VTS=0.65 after ablation), verifying the synergistic gain effect of cultural semantic injection and visual relationship modeling on cross-modal alignment.

[0035] To comprehensively evaluate the performance of this framework in cross-modal generation tasks, this study constructed a systematic comparative experimental framework with multi-level evaluation metrics. The experiment employed stratified sampling to construct a balanced evaluation set, covering 28 ethnic groups with high sample sizes (≥100 samples) and 27 ethnic groups with low sample sizes (≤30 samples). The balance of ethnic distribution was verified by the Shannon diversity index, achieving H'=4.72. The comparative benchmarks covered three major technical approaches: traditional encoding / decoding architectures (VieCap, SmallCap), general vision-language models (QwenVL, DeepSeekVL), and cutting-edge multimodal large language models (GPT4o, GLM4V). The evaluation system has undergone domain-adaptive modifications, specifically including ethnic identification accuracy based on expert annotation (MN-ACC), symbol coverage calculated based on a predefined cultural symbol dictionary (containing 2,387 typical elements such as "panchang pattern" and "silver bubble ornament") (CSC-F1), and cross-modal alignment (CMA) based on the CLIP-ViT-L / 14 model. The definitions and calculation paradigms of each indicator are as follows: MN-ACC The accuracy of ethnic semantic recognition was evaluated using a prototype theory-based method (Rosch, 1975): Wherein Representing ethnicity The number of true positive samples, This is to account for false positives misclassified as belonging to other ethnic groups. The indicator is calculated using a confusion matrix weighting (weights are inversely proportional to the ethnic sample size), effectively eliminating the influence of the long-tailed distribution of data.

[0036] Cultural Symbol Coverage (CSC-F1) is based on the signifier-signified relationship theory in semiotics (Saussure, 1916), constructing a hierarchical evaluation system: Primary symbols: ethnic identity elements (e.g., Yi ethnic group's Chawa, Miao ethnic group's silver ornaments); Secondary symbols: culturally derived characteristics (e.g., semantic patterns, clothing regulations). A weighted F1 metric is defined as follows: CSC-F1 in Indicates symbol hierarchy, Hierarchical weights (empirically set to...) ), and These represent the number of symbols at the two levels, respectively.

[0037] Cross-Modal Semantic Coherence (CMA) is evaluated using a dual-channel model based on distributed semantic theory. CMA in For visual-text embedding similarity, The optimal weights are determined by calculating the semantic depth of the path retrieved from the memory bank (e.g., "Tibetan → Bangdian → Apron → Pulu material") and then performing a grid search. .

[0038] The cross-modal generation quality comparisons of each model are shown in Table 1. The framework presented in this paper significantly outperforms all baselines with an ethnic identification accuracy of 81.91% (MN-ACC), a cultural symbol coverage of 0.692 (CSC-F1), and a cross-modal alignment of 0.831 (CMA), validating the effectiveness of multi-module collaboration.

[0039] Table 1. Cross-modal generation quality comparison. The performance of each model is compared in terms of ethnic identification accuracy (MN-ACC), cultural symbol coverage (CSC-F1), and cross-modal alignment (CMA). The MnCap framework significantly outperforms all baselines with 81.91% MN-ACC, 0.692 CSC-F1, and 0.831 CMA, validating the effectiveness of multi-module collaboration in semantic understanding of ethnic minority clothing.

[0040] Table 1 Quantitative evaluation validated the comprehensive advantages of this framework in cross-modal cultural perception tasks: the ethnic identification accuracy reached 81.91%, a 5.49-fold improvement over the GLM4V baseline, demonstrating the effective capture of long-tail ethnic characteristics by the cultural semantic retrieval mechanism; the cultural symbol coverage (0.692) surpassed GPT4o by 34.6% (0.514→0.692), confirming that the cognitive computing module achieved fine-grained attribute parsing through clothing texture deconstruction and accessory combination relationship modeling; the cross-modal alignment (CMA 0.831) improved by 27.5% compared to QwenVL, thanks to the targeted injection strategy of "cultural symbol-function-ritual scene" triples in the memory bank, which significantly enhanced the density correlation between visual details and semantic descriptions. The experimental data systematically revealed the technical value of the multi-module collaborative mechanism—cultural semantic constraints suppress cross-ethnic misjudgments, relational reasoning enhances fine-grained feature interpretation, and knowledge injection improves the completeness of cultural descriptions, opening up a new path for culturally sensitive computing in the field of multimodal generation.

[0041] The quality of the framework in the task of describing ethnic minority costumes was evaluated through systematic experiments, focusing on its cross-modal semantic alignment ability and fine-grained cultural feature capture efficiency. The present invention constructs a hierarchical evaluation system: 1) Basic semantic indicators: the BLEU series (BLEU-1 to BLEU-4)

[37] and ROUGE-L

[38] are used to evaluate the n-gram matching degree between the generated text and the reference description; 2) Semantic deep indicators: the semantic coverage of cultural symbols is measured by CIDEr

[39] , the grammatical fluency is evaluated by METEOR

[40] , and the conceptual integrity is analyzed by SPICE

[41] ; 3) Domain-specific indicators: the CHAIR illusion detection index proposed in this invention and the ethnic identification accuracy of MN-ACC are combined to form a multi-dimensional evaluation matrix.

[0042] for Figure 2 The results of different models describing images of ethnic minority costumes show that the framework in this paper is significantly better than traditional multimodal models in terms of ethnic category recognition accuracy, costume detail capture, and cultural semantic integrity, effectively solving the problems of ethnic misjudgment and simplification of description.

[0043] In actual generated image descriptions of ethnic minorities, GPT4o performs well in general scenarios, but its ethnic identification accuracy is low, and the generated descriptions are relatively simple, failing to fully capture the deeper cultural meanings of ethnic minority artifacts. From specific generation examples, the framework presented in this paper has significant advantages in both ethnic identification accuracy and descriptive richness, such as... Figure 5 As shown. Compared to the simplified descriptions of GPT4o and the ethnic misidentifications of GLM4V, MnCap can accurately analyze clothing details (such as colors and patterns) and cultural connotations (such as totem symbols). This advantage stems from the fact that the model of this invention combines a cognitive computing framework and a memory bank during training, which stores information on 55 ethnic minority costumes, artifacts, and cultures, supplementing the knowledge of the multimodal large model and improving the accuracy and richness of the generated descriptions. The cognitive computing framework can perform in-depth analysis of entities and their relationships in the image, further guiding the logical relationships when the multimodal large model generates descriptions. Combining these advantages, the method of this invention, through implicit modality alignment, enables the generated ethnic minority image descriptions to perform excellently in both accuracy and richness.

[0044] Figure 3 The comparison of the models on the BLEU series of metrics shows that the BLEU-4 score (15.8) of the framework in this paper (MnCap) is 42.3% higher than the best baseline, and the generated text is closer to the expert annotation in terms of n-gram matching degree and sentence structure integrity.

[0045] In terms of basic semantic matching, the BLEU-4 score of our framework (15.8) is 42.3% higher than the best baseline GLM4V (11.1). Figure 6As shown in the figure. This result indicates that the text generated by MnCap is closer to expert annotation in terms of sentence structure and key information coverage. Notably, the deep semantic evaluation results further validate the advantages of the framework. MnCap's CIDEr score (44.3) is 79.4% higher than GPT4o (24.7), indicating its significant progress in cultural symbol coverage and semantic logic.

[0046] Comparative analysis reveals that general multimodal models have significant limitations in capturing fine-grained cultural features. GPT4o's CIDEr score is only 24.7 (44.2% lower than MnCap), and its generated descriptions mostly remain at the level of general clothing attributes (such as "silver decoration"), lacking the analysis of cultural semantics such as "silver bubble arrangement density reflects marital status." Traditional encoding and decoding models (such as VieCap) have a BLEU-4 score as low as 1.5, exposing their fundamental deficiencies in cross-modal alignment and modeling of long-tail cultural features.

[0047] Figure 4 MnCap outperforms traditional models and general multimodal large models in deep semantic metrics such as CIDEr and SPICE. The CIDEr score (44.3) is 79.4% higher than GPT4o, which confirms the strengthening effect of memory mechanism and cognitive computing on the coverage of cultural symbols.

[0048] The synergistic effect of the modules showed differentiated contributions among the indicators: the memory mechanism improved CIDEr by injecting professional triples such as "Yi ethnic group's Chaerwa → wool dyeing → Torch Festival" by 19.8 points; the cognitive computing module contributed 32% to the ROUGE-L score by analyzing the topological relationships of clothing components (such as the spatial dependency between the silver chain of the belt and the pattern of the skirt) using graph neural networks; and the multimodal alignment mechanism improved the visual-text similarity (VTS) to 0.831 through CLIP feature remapping, effectively suppressing cross-cultural illusions such as "misrepresenting Tibetan Bangdian as an apron". This multi-layered integration of technologies enabled the generated descriptions to accurately represent visible entities (MN-ACC 81.91%) and deeply relate to the cultural context (such as deriving the regional characteristics of Dali from the "wind, flowers, snow and moon" shape of Bai ethnic headdress), setting a new benchmark for the digital interpretation of cultural heritage.

[0049] To further verify the effectiveness of each module (memory mechanism, cognitive computing module, and multimodal alignment mechanism) in the framework proposed in this study, a series of ablation experiments were designed. By removing or replacing key modules one by one, the changes in the generated descriptions on various evaluation indicators were observed, thereby quantitatively analyzing the contribution of each module to the overall performance. The experimental results are shown in Table 2.

[0050] Table 2 shows the results of the ablation experiments, indicating that removing the memory mechanism, cognitive computing, or multimodal alignment modules all led to a significant decrease in model performance. The complete framework significantly outperformed the variants across all evaluation metrics, confirming the indispensability of the three modules in cultural semantic supplementation, relational reasoning, and cross-modal alignment.

[0051] Table 2 When removing the memory mechanism, the retrieval process no longer utilizes the vocabulary memory bank information of minority languages, i.e., the function that originally constructed logical words. Degenerates into relying solely on image semantic representation Simplified form .

[0052] Experimental results show that the model significantly decreased in the expression and richness of description of ethnic minority characteristics at this point, with the accuracy (MN-ACC) dropping from 81.91% of the complete model to 33.12%, and the BLEU-4 and CIDEr scores decreasing to 12.3 and 22.56, respectively. This indicates that the memory mechanism plays a crucial role in supplementing cultural semantic information.

[0053] After removing the cognitive computing module, the model no longer models complex relationships through multi-level attribute analysis and graph neural networks, i.e., the original deep semantic representation... The model degenerates into a direct concatenation of simple features of the target entity. Experimental results show that the model significantly declines in semantic consistency and accuracy, with the CIDEr score dropping from 44.3 to 24.52. The ROUGE and SPICE indices also decrease accordingly, indicating that the cognitive computing module plays a crucial role in capturing complex relationships between entities in images.

[0054] After removing the multimodal alignment mechanism, the model excessively combines different channels of visual features and textual semantics into their respective dimensional modules, resulting in a significant decrease in the coherence and accuracy of the generated descriptions; MN-ACC dropped from 81.91% to approximately 31.13%. This fully demonstrates that the close connection between vision and text is crucial for ensuring that the generated descriptions reflect the image content and possess cultural connotations.

[0055] To explore the interactions between the modules, this invention also conducted experiments with simultaneous bimodal ablation. After removing the memory mechanism and the multimodal alignment mechanism, the prompts relied solely on the original features of the image, further deteriorating the accuracy and semantic richness of the generated descriptions. The MN-ACC fell below 50%, and the BLEU-4 and CIDEr scores also decreased significantly. This verifies that there is a synergistic effect among the modules, and only when all three work together can the optimal description effect be achieved.

[0056] Ablation experiments showed that removing the memory mechanism, cognitive computation, or multimodal alignment module all led to a significant performance decline. The complete framework significantly outperformed the variants in metrics such as MN-ACC (81.91%) and BLEU-4 (38.89), confirming the crucial role of the three modules in semantic supplementation, relational reasoning, and cross-modal fusion. This indicates that the memory mechanism plays a vital role in supplementing semantic information about ethnic minorities and reducing spurious content in descriptions. Secondly, removing the cognitive computation module decreased the accuracy and logical consistency of the generated descriptions, with the CIDEr score dropping from 36.67 to 28.45. The cognitive computation module effectively captures complex relationships and semantic information in images, thus ensuring the accuracy and logical consistency of the descriptions. Finally, removing the multimodal alignment mechanism resulted in a comprehensive performance decline, with accuracy decreasing by 32.06%, description richness decreasing by 43%, and the proportion of hallucinations significantly increasing to 12.8%. This result demonstrates that the close integration of visual features and textual semantics is crucial for the coherence and accuracy of generated descriptions. Overall, the ablation experiments of each module show that the memory mechanism, cognitive computing module, and multimodal alignment mechanism all play key roles in generating high-quality image descriptions of ethnic minority costumes. Their synergistic effect enables the framework to effectively support the protection and dissemination of ethnic minority cultures.

[0057] Figure 1 The framework for generating image descriptions of ethnic minority costumes consists of five stages: a. Input image; b. Cognitive Computing: Visual analysis of the input image to extract target entities, attributes, and relationships; c. Memory Retrieval: Utilizing an abstract semantic memory, retrieving semantic information related to the input image (such as costume culture and ethnic customs) by calculating the similarity between the query vector and the triple vectors in the memory; d. Construct Prompt: Combining the results of image semantic understanding and memory retrieval, constructing logical prompts to guide the multimodal large language model in generating accurate and semantically rich descriptions; e. Generate Description: Inputting the constructed prompts and the input image into the multimodal large language model (MLLM) to generate the final image description through modal alignment.

[0058] like Figure 1 As shown, the framework consists of five stages: the input image first parses entities and relationships through the cognitive computing module, then dynamically acquires ethnic semantic knowledge through the memory retrieval module, and finally generates descriptive text through prompt words construction and MLLM.

[0059] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A method for generating image descriptions of ethnic minorities based on a memory mechanism and cognitive computing, characterized in that: The method comprises the following steps: S1: input an image and perform preprocessing: extract visual features by ConvNeXt; S2: vectorize the visual features and retrieve the most relevant semantic triples as corresponding ethnic cultural knowledge from the constructed abstract semantic memory library of ethnic images; S3: perform multi-level cognitive strategies on the input image and perform ethnic feature weighted fusion to obtain cognitive reasoning results; S4: combine the ethnic cultural knowledge in S2 and the cognitive reasoning results in S3 to construct a logical prompt word, and input the logical prompt word, visual features and cognitive reasoning results into a multi-modal large language model GLM-4V based on a Transformer architecture to generate a description text; S5: perform consistency checking on the description text and the logical prompt word, and determine whether the consistency checking is passed, if the consistency checking is passed, end S5, if the consistency checking is not passed, perform optimization.

2. The method of claim 1, wherein the method is based on a memory mechanism and cognitive computing for generating image descriptions of ethnic minorities. The specific steps of constructing the abstract semantic memory library of ethnic images in S2 are as follows: S2.1: extract triples of subject, predicate and object from ethnic image description texts by an open information extraction model Open IE, and the specific expression is as follows: wherein, respectively represent subject, predicate, object, represents an input text, represents a semantic relation parsing tool for extracting structured semantic triples from the text T; S2.2: encode the triple elements into low-dimensional dense vectors using a Word2Vec model, and concatenate them into a unified high-dimensional vector representation, and the specific expression is as follows: wherein, represents a triple vector, which is a vector splicing result of a subject S, a predicate P, and an object O, represents vector splicing, respectively represent word vectors of the subject, the predicate, and the object.

3. The method of claim 2, wherein the method is based on a memory mechanism and cognitive computing for generating image descriptions of ethnic minorities. The cosine similarity retrieval mechanism is used in the abstract semantic memory library of ethnic images in S2, and the cosine similarity is calculated to filter the retrieval results below the preset similarity threshold, and the specific expression is as follows: wherein, denotes the cosine similarity, denotes the triple vector, provided by a pre-constructed minority image abstract semantic memory, denotes the query vector, which is the semantic representation of the visual features of the input image after vectorization, denotes the Euclidean norm; The abstract semantic memory library of ethnic images is dynamically expanded by a contrastive learning loss function, and the specific expression is as follows: wherein, denotes an exponential function, denotes a logarithmic function, denotes a cosine similarity between a query vector and a positive sample vector, denotes a cosine similarity between a query vector and a k-th negative sample vector, denotes a k-th positive sample candidate vector, denotes a temperature parameter, denotes a k-th negative sample vector, denotes a summation operation over a set of negative samples.

4. The method of claim 1, wherein the method is based on a memory mechanism and cognitive computing for generating image descriptions of ethnic minorities. The multi-level cognitive strategy in S3 comprises: Target recognition layer: extract input image visual feature vectors by Faster R-CNN: wherein, represents a general feature of an entity, represents a Faster R-CNN detector, represents an input image, represents an entity bounding box; Attribute analysis layer: extract attribute features by ConvNeXt network: wherein, represents detecting fine-grained properties of the entity, represents an attribute extraction function; Relationship reasoning layer: update node features using graph neural network GNN: wherein, represents the l-th layer node feature, represents the neighbor node set, , respectively represent layer parameters, is a Gaussian error linear unit, represents a feature concatenation operation, is a spatial relative position vector of the entity . The specific expression of ethnic feature weighted fusion in S3 is as follows: wherein, a weight coefficient representing a spatial relationship, a weight coefficient representing a semantic relationship, a cognitive inference result, a spatial relationship, a semantic relationship, output by a graph neural network GNN in the relationship inference layer.

5. The method of claim 1, wherein the method is based on a memory mechanism and cognitive computing for generating image descriptions of ethnic minorities. The construction expression of the logical prompt word in S4 is as follows: wherein, represents a logical cue word, represents a cognitive inference result, represents a dynamic semantic fusion function, represents a cultural knowledge semantic vector; The specific expression of the logical prompt word, visual features and cognitive reasoning results in S4 is as follows: wherein, represents, represents visual features extracted by ConvNeXt, represents logical cues, represents cognitive inference results, respectively represent encoding of cognitive inference results and logical cues.

6. A minority image description generation system based on a memory mechanism and cognitive computing, characterized in that: A method for generating ethnic image descriptions based on a memory mechanism and cognitive computing is used to perform any one of claims 1-5.

Images