Adaptive visual layer fusion generation method and system based on national language and title

Abstract

The application provides a method and system for generating adaptive visual layer fusion based on national language and title, belonging to the field of artificial intelligence and multi-modal information processing technology, comprising: S1: collecting and cleaning samples, constructing training samples through description construction, auditing and anti-leakage division; S2: in the training stage of the language title adaptive visual layer fusion model, generating fusion weights through multi-layer visual token feature extraction, title semantic representation and language embedding, finally generating fusion visual representation, and then obtaining the target national language description; S3: in the reasoning stage, inputting an image and a title, and outputting a national language description. The method solves the problems of inaccurate visual evidence selection, weak image-text alignment and unstable target language generation of the existing fixed visual layer fusion strategy in the national language title guided generation scene.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence and multimodal information processing technology, and specifically relates to an adaptive visual layer fusion generation method and system based on ethnic languages ​​and titles. Background Technology

[0002] In recent years, multimodal large language models have made rapid progress in tasks such as image description, visual question answering, image-text understanding, and visual conditional text generation. Existing models typically extract image features through a visual encoder, then project these visual features into a language model space, where the language model generates text based on user instructions. However, current technologies are mainly built around high-resource languages ​​such as English and Chinese, and their ability to generate real-world image-text for ethnic minority languages ​​such as Tibetan, Uyghur, Korean, and Mongolian remains insufficient.

[0003] In real-world news reporting, cultural promotion, government communication, cultural tourism consulting, and ethnic language media production scenarios, the input is often not a single image, but rather an image plus an ethnic language title. The model needs to generate a complete description in the same ethnic language based on the event semantics provided in the title and the visible content in the image. This type of task requires the model to understand the image, the event, location, cultural objects, and language expression habits in the title, and to maintain consistency and fluency in the target ethnic language in the output.

[0004] Existing technical solutions include:

[0005] (1) Image conditional generation method based on general multimodal large model

[0006] These methods typically use a visual encoder to acquire image features and then input those features into a language model via a multimodal projection layer. The model can generate answers or descriptions based on image and text prompts, but it often lacks specialized modeling of the semantics and language identity of ethnic language titles.

[0007] (2) Method based on fixed multi-layer visual feature fusion

[0008] This type of method extracts visual features from multiple levels of the visual encoder and concatenates, sums, or applies fixed weights according to preset methods to enhance the expressive power of visual information. Its drawback is that the fusion strategy remains unchanged across different languages, titles, and image samples, making it difficult to adapt to differences in ethnic language scripts, title semantics, and scene complexity.

[0009] (3) Data construction methods based on machine translation or general web crawling

[0010] These methods typically translate English or Chinese image-text data into the target low-resource language, or directly crawl image-text pairs from the internet. Their advantage is low cost, but they are prone to problems such as semantic mismatch between image and text, loss of culturally specific expressions, leakage of duplicate samples, over-reliance on textual information, and difficulty in controlling sample quality.

[0011] (4) Automatic evaluation method based on general text similarity index

[0012] Existing generative tasks often use metrics such as character overlap, sequence overlap, or semantic similarity for evaluation. These metrics can measure the similarity between the model's output and the reference text, but they cannot prove on their own whether the output truly originates from image evidence. Therefore, in ethnic language title-guided generation tasks, it is also necessary to combine data quality control and leakage prevention measures to ensure the credibility of the evaluation results.

[0013] However, existing technical solutions have the following problems:

[0014] (1) The originality and controllability of multimodal training samples of ethnic languages ​​are insufficient.

[0015] Existing translation-based data construction methods tend to generalize objects with ethnic cultural characteristics into common vocabulary, weakening the correspondence between visual objects and target language expressions. Furthermore, extensive web scraping easily introduces invalid images, noisy text, non-target language fragments, weakly aligned image-text samples, and samples with copyright risks.

[0016] (2) Fixed visual layer fusion cannot adapt to different languages ​​and title semantics.

[0017] Visual encoders typically contain information at different granularities in their shallow, mid, and deep layers. Shallow layers may contain local texture and location cues, mid layers may contain objects and spatial relationships, and deep layers may contain semantic abstraction information. Existing fixed fusion methods use the same level of combination for all samples, making it impossible to dynamically adjust the source of visual information based on language features such as Tibetan, Uyghur, Korean, and Mongolian, as well as the semantics of the titles.

[0018] Furthermore, existing fixed-layer visual fusion methods typically predetermine the combination or fusion weights of visual layers before model training, such as fixed splicing, fixed summation, or fixed weighting of shallow, mid-layer, and deep visual features. The fusion weights in these methods do not change with the input ethnic language, title semantics, or specific image samples. Even with the introduction of text prompts, existing methods mostly use text prompts during the language model decoding stage, rather than conditionally selecting different visual layers based on target language identity and title semantics before the visual features enter the multimodal projector. Therefore, existing methods struggle to address the visual evidence mismatch problem in scenarios where ethnic language titles are generated.

[0019] (3) The model tends to rely on the title and ignore visual evidence.

[0020] In headline-driven generation tasks, headlines often contain strong event semantics. If the training samples contain a large amount of article text details that cannot be inferred from images and headlines, the model may learn a shortcut of "writing news-style text based on headlines" instead of learning a robust correspondence between images, headlines, and target language descriptions.

[0021] (4) Randomly dividing the training and test sets can easily lead to data leakage.

[0022] In real-world web page data, the same image, the same news event, the same article, or nearly duplicated content may appear multiple times. If we don't group these elements by image, article, and nearly duplicate event, the training and test sets may contain highly similar samples, leading to inflated evaluation results.

[0023] Therefore, a technical solution is needed that can simultaneously cover "construction of high-quality ethnic language image-text title samples" and "adaptive visual fusion under language title conditions." The data construction stage is used to obtain authentic, traceable, and quality-controlled training samples of images, titles, and descriptions; the model and method stage is used to dynamically fuse multi-layered visual features based on language identifiers and title semantics, thereby improving the image-text alignment capability, title semantic utilization capability, and target language generation quality in ethnic language multimodal generation. Summary of the Invention

[0024] To address the aforementioned technical problems, this invention provides an adaptive visual layer fusion generation method based on ethnic language and title, comprising the following steps:

[0025] Step S1: Collect the original samples and retain the metadata of the original samples; obtain the target samples by cleaning and filtering the metadata; then construct the image-title-description training samples by describing the target samples, having them reviewed by experts, labeled with difficulty and divided for leakage prevention;

[0026] Step S2: During the training phase of the language title adaptive visual layer fusion model, input training samples, extract visual token features from multiple specified levels through the visual encoder, and generate fusion weights for visual features by constructing title semantic representation and language embedding, and finally generate fused visual representation, thereby obtaining the target ethnic language description;

[0027] Step S3: In the inference phase, the user inputs an image and a title in the ethnic language, and the trained language title adaptive visual layer fusion model outputs the ethnic language description.

[0028] Beneficial effects:

[0029] 1. This invention reduces text-image mismatch and cultural semantic distortion caused by translated data and crude web page data by unifying data construction and model training into the same title-guided generation task.

[0030] 2. This invention reduces the risk of model learning invisible text details and memorizing duplicate samples by taking measures such as expert review, using the main text only for the construction stage, and preventing leakage grouping.

[0031] 3. This invention generates visual layer fusion weights by embedding the target ethnic language and representing the semantics of the title. This allows the visual layer fusion weights to change dynamically with different languages, titles, and image samples, avoiding the visual evidence mismatch problem caused by using the same visual layer combination for all samples in fixed visual layer fusion.

[0032] 4. This invention improves image-title alignment and the stability of target language description generation by conditionally weighting and fusing shallow, medium and deep visual token features before the multimodal projector. This allows the model to automatically adjust the usage ratio of local texture, object relationships and high-level semantic features according to the object, scene, event or cultural semantics pointed to by the title.

[0033] 5. This invention reduces training costs and improves the portability of the method on different multimodal large models by freezing the visual encoder and projection layer and training only the LoRA and adaptive fusion modules.

[0034] 6. This invention improves target language consistency, descriptive fluency, image-title alignment capability, and low-resource language generation stability by training and inferring against the target of ethnic language title generation. Attached Figure Description

[0035] Figure 1 This is a schematic diagram of the adaptive visual layer fusion generation method based on ethnic language and title according to the present invention;

[0036] Figure 2 A schematic diagram illustrating the process of constructing and controlling multimodal samples for ethnic language titles;

[0037] Figure 3 A schematic diagram of the architecture of the language title adaptive visual layer fusion model;

[0038] Figure 4 This is a structural block diagram of an adaptive visual layer fusion generation system based on ethnic language and title according to the present invention. Detailed Implementation

[0039] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.

[0040] Example 1

[0041] like Figure 1 As shown in the figure, an adaptive visual layer fusion generation method based on ethnic language and title provided by an embodiment of the present invention includes the following steps:

[0042] Step S1: Collect the original samples and retain the metadata of the original samples; obtain the target samples by cleaning and filtering the metadata; then construct the image-title-description training samples by describing the target samples, having them reviewed by experts, labeled with difficulty and divided for leakage prevention;

[0043] Step S2: During the training phase of the language title adaptive visual layer fusion model, input training samples, extract visual token features from multiple specified levels through the visual encoder, and generate fusion weights for visual features by constructing title semantic representation and language embedding, and finally generate fused visual representation, thereby obtaining the target ethnic language description;

[0044] Step S3: In the inference phase, the user inputs an image and a title in the ethnic language, and the trained language title adaptive visual layer fusion model outputs the ethnic language description.

[0045] In one embodiment, step S1 above involves: collecting original samples and retaining the metadata of the original samples; obtaining target samples by cleaning and filtering the metadata; and then constructing training samples for image-title-description by performing description construction, expert review, difficulty labeling, and anti-leakage classification on the target samples. Specifically, this includes:

[0046] Step S11: Collect raw samples from ethnic language web pages. Each raw sample includes at least: image I, ethnic language title T and its text description, forming an image-title-description triple; retain metadata for each raw sample, including: URL, domain name, source type, language tag, topic tag, title, publication time, text, image address, and image identifier;

[0047] Samples were collected from publicly accessible ethnic language web pages, including government portals, news websites, sports channels, cultural platforms, health websites, and websites related to education or local events. Each original sample included at least an image, an ethnic language title, and corresponding text. The metadata of the original samples was not used as input to the model but was used for sample tracing, expert review, data statistics, ethical review, and leakage prevention.

[0048] Step S12: Perform image validity filtering, text noise cleaning, image-title alignment, and security filtering on the original sample in sequence to obtain the filtered sample;

[0049] Filter samples from the original sample that have empty titles, empty body text, or body text containing a large amount of navigation bar, advertisements, copyright notices, repeated paragraphs, garbled characters, non-target language fragments, or content unrelated to the title.

[0050] The system uses rule checks, manual sampling, or multimodal alignment models to determine whether the image and title have a reasonable semantic relationship. Samples with weak correlation between title and image, images that cannot support the title's theme, descriptions that clearly depend on invisible text, those involving privacy sensitivities, sensitive scenarios involving minors, medical legal risks, violent or hate speech, unclear copyright, or risks of republication are deleted, restricted, or marked for URL-only publication.

[0051] Step S13: The annotators construct a target ethnic language description for the filtered samples;

[0052] For samples that pass the filtering process, annotators write descriptions in the target ethnic language based on images, titles, visible text, and identifiable cultural cues. The original text is used only as a reference during the model building phase and is not input into the model. If a large language model is used for initial draft generation or language polishing, the results must be manually reviewed and modified and cannot be directly used as the final reference description.

[0053] Step S14: Experts conduct quality and safety audits to obtain the target sample;

[0054] Samples for each ethnic language were reviewed by native speakers or trained experts to check for factual accuracy, visual evidence, title-image consistency, linguistic fluency, cultural appropriateness, and potential for illusion. Samples found to lack supporting details, express themselves unnaturally, exhibit cultural misinterpretations, or pose security risks were revised or removed.

[0055] Step S15: Difficulty labeling and grouping of the target samples in different dimensions to finally form training samples of image-title-description;

[0056] The samples were labeled with difficulty based on dimensions such as visual complexity, visible text and layout complexity, title-image alignment difficulty, generation complexity, and cultural or event specificity. Simultaneously, samples were grouped according to identical images, identical article URLs, identical source documents, and nearly duplicated news articles to ensure that samples in the same group only appear in one set of the training, validation, or test sets.

[0057] Figure 2 A schematic diagram of the process for constructing and controlling quality of multimodal samples guided by ethnic language titles is presented.

[0058] In one embodiment, step S2 above: During the training phase of the language title adaptive visual layer fusion model, training samples are input, visual token features are extracted from multiple specified levels through a visual encoder, and fusion weights for visual features are generated by constructing a title semantic representation and language embedding, ultimately generating a fused visual representation, thereby obtaining a description of the target ethnic language, specifically including:

[0059] Step S21: Obtain the image I, the ethnic language title T, and the ethnic language identifier of the training sample. Input the visual encoder to extract visual token features from image I from multiple specified visual layers:

[0060] make The input image can be represented as: ,in, Indicates the image height. Indicates the image width. Indicates the number of image channels; , , All are positive integers. In the embodiments of this invention... This represents an RGB color image. Input image It can be input into the visual encoder after undergoing size normalization, pixel value normalization, or other preprocessing required by the visual encoder.

[0061] Ethnic Language Titles For image The corresponding title text is in the same language as the target ethnic language description to be generated, and the language identifier is... Used to indicate titles in ethnic languages The target output describes the ethnic language to which it belongs. Language identifier. The value range is the preset language set: ; Represents a collection of ethnic languages. In this embodiment of the invention: ,in, Represents Tibetan language. Represents Uyghur language, It represents the Korean language. It represents Mongolian. The collection of ethnic languages ​​can also be expanded to include other ethnic languages ​​or low-resource languages.

[0062] The visual layer number specified by the visual encoder The set is: ,in, Indicates a set of specified visual layer numbers; Indicates the number of specified visual layers, and ;

[0063] Extract visual token features of the corresponding level from the visual encoder:

[0064] ;

[0065] in, Indicates the first Layered visual features, with dimensions of M represents the number of visual tokens. For visual feature dimensions, and All are positive integers; For visual encoders;

[0066] To ensure that subsequent weighted fusion can be executed, the output features of the selected multiple visual layers have the same number of tokens and feature dimensions; if the number of tokens or feature dimensions of the outputs of different visual layers are inconsistent, they are converted into a unified dimension through linear transformation, interpolation, pooling or projection.

[0067] In this embodiment of the invention, shallow, middle, and deep visual features are taken respectively, corresponding to the visual features of the 3rd, 18th, and 23rd layers of the visual encoder. At this time: Output the corresponding visual token features. .

[0068] Step S22: Segment the ethnic language title T into several tokens. The length is obtained as Title token sequence: ;in, Indicates the length of the title token sequence, and satisfies ; Indicates the preset maximum title token length; It is a positive integer;

[0069] Obtaining through the embedding layer of the base language model (e.g., Qwen3-VL-4B-Instruct) The word embedding layer obtains the title token embedding: ,in, The input embedding layer represents the base language model; Indicates the first Embedded vector of each title token , This represents the word embedding dimension or hidden state dimension of the basic language model. It is a positive integer;

[0070] To prevent padding tokens, special control characters, or other invalid tokens from interfering with the semantic representation of the header, a mask flag is set for each header token. ;when When the title token is valid, ,when When the header token is invalid, ;

[0071] Calculate the number of mask tokens for valid header tokens. ;

[0072] Perform masked average pooling on the embedding vectors of all valid title tokens to obtain the title semantic representation. :

[0073] ;

[0074] Through the above processing, this invention converts ethnic language titles of varying lengths into fixed-dimensional semantic vectors. This semantic representation of the title... It is used to characterize the event, object, scene, location, cultural semantics or image pointing information in the title, and serves as one of the input conditions for the subsequent adaptive gating network to generate visual layer fusion weights.

[0075] Step S23: Let the number of supported languages ​​be: ;in, Represents the set of preset languages The number of languages ​​in;

[0076] Set up a trainable language embedding table ,in, Represents a language embedding table, Represents the language embedding dimension;

[0077] According to ethnic language symbols Query the trainable language embedding table to obtain the ethnic language embeddings. :

[0078] ;

[0079] in, Represents the language embedding query functions; It indicates the embedding of ethnic languages, used to express conditional information about different ethnic languages ​​in terms of writing systems, writing direction, word order, expression habits and data distribution;

[0080] like and If the dimensions are different, then in the subsequent gating network, the two are adapted to the dimensions through linear mapping or multilayer perceptron before being spliced ​​together. and Together, they serve as input conditions for the adaptive gating network, used to generate the visual layer fusion weights corresponding to the current image-title sample.

[0081] Step S24: and By concatenating the vectors, we obtain the joint condition vector. ;

[0082] in, express and splicing along the feature dimension;

[0083] It includes both target language identity information and current title semantic information. The target language identity information is used to express the differences between different ethnic languages ​​in script, writing direction, word order, expression habits, and data distribution; the title semantic information is used to express the event, object, scene, location, or cultural semantics in the current title.

[0084] Will Input Adaptive Gating Network This yields unnormalized weight vectors corresponding to multiple visual layers. ;in, This represents an adaptive gating network. Indicates the parameters of the gating network; Indicates the first The unnormalized weight vectors corresponding to each visual layer;

[0085] The adaptive gating network of this invention is composed of a multilayer perceptron (MLP), and may also include linear layers, nonlinear activation functions, normalization layers, or dropout layers.

[0086] The unnormalized weight vector is processed using the Softmax function. Normalization is performed to obtain the fusion weight vector of multiple visual layers. ,in, For the first The fusion weights of visual layer features satisfy the following conditions: and ;

[0087] The Softmax normalization process can be represented as:

[0088] ;

[0089] The resulting fusion weight vector Instead of parameters that are fixed for all samples, they are determined by the language embedding of the current sample. and title semantic representation The weights are dynamically determined at the sample level. Different visual layer fusion weights can be generated for different ethnic languages, different title content, or different image-title samples.

[0090] In this embodiment of the invention, the adaptive gating network outputs the normalized weights of the visual features from layers 3, 18, and 23. , is represented as: ,and This weight remains fixed during the generation of the current sample and is used in subsequent steps to weightedly fuse features from multiple visual layers.

[0091] Step S25: Perform a weighted summation of the features from multiple visual layers to obtain the fused visual representation. :

[0092] ;

[0093] In the aforementioned integration process, The visual hierarchy weight corresponding to the current sample is used as a scalar weight in the first... All visual token features of the layer; that is, for the first layer... Each visual token in the layer is assigned a weight according to the same layer level. This allows for the fusion of visual information across multiple levels into a unified visual representation without altering the number of visual tokens or the dimensions of their features.

[0094] In this embodiment of the invention, adaptive visual layer fusion is represented as follows: ;

[0095] The fusion weights in this step are not fixed constants, but rather determined based on the current input image. Ethnic Language Titles and language identifiers Dynamically generated sample-level weights. In the same model, different ethnic languages, different title semantics, or different image-title samples can correspond to different visual layer fusion weights.

[0096] Compared to traditional fusion methods with a fixed visual layer, which typically employ preset weights that remain unchanged across all languages, titles, and image samples, the fusion weights used in this invention are embedded from the target language. and the semantic representation of the current title The decision is made jointly. Therefore, this invention enables conditional selection and dynamic fusion of shallow, mid-level, and deep visual features based on the target ethnic language and title semantics before visual features enter the multimodal projector. For example, when the ethnic language title mainly involves the overall scene, event semantics, or spatial relationships, the system can increase the fusion weight of mid-level or high-level visual features; when the title involves fine-grained objects, local textures, visible text, or cultural symbols, the system can increase the fusion weight of shallow or mid-level visual features. Through the above methods, this invention can enhance the matching degree between the fused visual representation and the ethnic language title semantics, improve image-title alignment capabilities, and enhance the stability of target ethnic language description generation.

[0097] Step S26: ... The projector of the input multimodal language model is mapped from the visual feature space to the latent space of the language model, resulting in a visual token representation consistent with the word embedding dimension of the language model. :

[0098] ;

[0099] in, For projectors of multimodal language models, Indicates projector parameters;

[0100] Title based on ethnic language and ethnic language symbols Construct the title hint text and obtain the title hint token representation through the input embedding layer of the language model. :

[0101] ;

[0102] in, Indicates by and Constructed text prompts; The input embedding layer represents the language model;

[0103] Representing visual tokens The token indicated in the title By concatenating the sequences, we obtain the conditional input sequence. ;

[0104] Will and Input Language Model Decoder The target ethnic language description Y is obtained:

[0105] ;

[0106] in, Description of the target ethnic language The Middle One valid target token; Represents the language model decoder and its trainable adaptation parameters; This indicates the number of tokens describing the target ethnic language.

[0107] The language model decoder generates a description of the target ethnic language in an autoregressive manner: in the... Each generation location is represented by the decoder based on the projected visual token. The title hints that the token indicates Language identifiers and the already generated predecessor target token Predict the current token Conditional probability: , Indicates the first The sequence of target tokens preceding each token.

[0108] Step S27: Construct a loss function that only calculates the negative log-likelihood on the target ethnic language description token:

[0109] ;

[0110] In the formula The model represents the correct target token given visual information, title cues, language identifiers, and preceding target tokens. The logarithm of the assigned probability; the negative sign is used to transform the problem of maximizing the target token generation probability into the problem of minimizing the loss; Used to normalize the number of valid target tokens in the current training sample.

[0111] When calculating the loss, this invention only describes the target ethnic language. The loss is calculated using valid target tokens generated by the model; image tokens, visual tokens, title tokens, voice prompt tokens, special control tokens, and padding tokens are not included. It is also not included in loss calculation. In one implementation, if the terminator token is set as part of the target native language description, then the terminator token can be included. .

[0112] During training, an autoregressive language modeling objective is used to train the model. The parameters of the visual encoder and multimodal projector are frozen, and only the low-rank adaptation parameters, adaptive gating network parameters, and language embedding parameters of the language model are updated. Let the set of parameters that need to be updated during training be: ;in, This represents the low-rank adaptation parameters of the language model; Indicates the parameters of the adaptive gating network; The language embedding parameters are represented. The visual encoder parameters and multimodal projector parameters remain unchanged during training. This training method reduces the scale of parameter updates and training costs, and improves the transfer adaptability of the method on different multimodal language models.

[0113] In one embodiment, step S3 above: During the inference phase, the user inputs an image and an ethnic language title, and the trained language title adaptive visual layer fusion model outputs an ethnic language description, specifically including:

[0114] During the model inference phase, the system receives user-uploaded images and titles in ethnic languages. First, it constructs language identifiers and extracts multi-layered visual features from the images. Second, it calculates the semantic representation of the titles and language embeddings. Then, an adaptive gating network generates visual layer fusion weights. Finally, it completes visual layer weighted fusion, projection, and language model decoding to output a description in the target ethnic language.

[0115] The inference output can be used for applications such as generating news image descriptions in ethnic languages, generating cultural and tourism promotional copy, intelligent explanation of ethnic culture, assisting in the release of government information in graphic and textual format, multimodal evaluation of low-resource ethnic languages, and organizing educational and cultural resources.

[0116] Figure 3 The diagram illustrates the architecture of the language title adaptive visual layer fusion model.

[0117] Example 2

[0118] like Figure 4 As shown, this embodiment of the invention provides an adaptive visual layer fusion generation system based on ethnic languages ​​and titles, comprising the following modules:

[0119] The training sample construction module 41 is used to collect original samples and retain the metadata of the original samples; obtain target samples by cleaning and filtering the metadata; and then construct training samples of image-title-description by describing the target samples, having them undergo expert review, difficulty labeling, and anti-leakage classification.

[0120] The language title adaptive visual layer fusion model training module 42 is used for the training phase of the language title adaptive visual layer fusion model. It takes training samples as input, extracts visual token features from multiple specified levels through a visual encoder, and generates fusion weights for visual features by constructing title semantic representation and language embedding, and finally generates fused visual representation, thereby obtaining the target ethnic language description.

[0121] The language title adaptive visual layer fusion model inference module 43 is used in the inference stage, where the user inputs an image and a national language title, and the trained language title adaptive visual layer fusion model outputs a national language description.

[0122] An adaptive visual layer fusion generation device based on ethnic language and title includes one or more electronic devices, wherein the one or more electronic devices are used to implement the adaptive visual layer fusion generation method based on ethnic language and title.

[0123] An electronic device includes: one or more processors; and a memory for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement an adaptive visual layer fusion generation method based on ethnic languages ​​and titles.

[0124] A computer-readable storage medium having executable instructions stored thereon, which, when executed by a processor, cause the processor to implement an adaptive visual layer fusion generation method based on ethnic languages ​​and titles.

[0125] A non-transitory computer-readable storage medium having a computer program stored thereon that, when executed by a processor, implements an adaptive visual layer fusion generation method based on ethnic languages ​​and titles.

[0126] The above description is merely a specific embodiment of the present invention, enabling those skilled in the art to understand or implement this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, the present invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features claimed herein.

Claims

1. An adaptive visual layer fusion generation method based on ethnic language and title, characterized in that, include: Step S1: Collect the original sample and retain the metadata of the original sample; The target sample is obtained by cleaning and filtering the metadata. Then, the target samples are subjected to description construction, expert review, difficulty labeling, and leakage prevention classification to construct training samples of image-title-description; Step S2: During the training phase of the language title adaptive visual layer fusion model, input training samples, extract visual token features from multiple specified levels through the visual encoder, and generate fusion weights for visual features by constructing title semantic representation and language embedding, and finally generate fused visual representation, thereby obtaining the target ethnic language description; Step S3: In the inference phase, the user inputs an image and a title in the ethnic language, and the trained language title adaptive visual layer fusion model outputs the ethnic language description.

2. The adaptive visual layer fusion generation method based on ethnic language and title according to claim 1, characterized in that, Step S1 specifically includes: Step S11: Collect raw samples from ethnic language web pages. Each raw sample includes at least: image I, ethnic language title T and its text description, forming an image-title-description triple; retain metadata for each raw sample, including: URL, domain name, source type, language tag, topic tag, title, publication time, text, image address, and image identifier; Step S12: Perform image validity filtering, text noise cleaning, image-title alignment, and security filtering on the original sample in sequence to obtain the filtered sample; Step S13: The annotators construct a target ethnic language description for the filtered samples; Step S14: Experts conduct quality and safety audits to obtain the target sample; Step S15: Difficulty labeling and grouping of the target samples are performed in different dimensions to finally form training samples of image-title-description.

3. The adaptive visual layer fusion generation method based on ethnic language and title according to claim 2, characterized in that, Step S2 specifically includes: Step S21: Obtain the image I, the ethnic language title T, and the ethnic language identifier of the training sample. Input the visual encoder to extract visual token features from image I from multiple specified visual layers: The visual layer number specified by the visual encoder The set is: ,in, Indicates a set of specified visual layer numbers; Indicates the total number of specified visual layers, and ; Extract visual token features of the corresponding level from the visual encoder: ； in, Indicates the first Layered visual features, with dimensions of , representing the number of visual tokens. For visual feature dimensions; For visual encoders; Step S22: Segment the ethnic language title T into several tokens. The length is obtained as Title token sequence: ;in, Indicates the length of the title token sequence, and satisfies ; Indicates the preset maximum title token length; It is a positive integer; By using the embedding layer of the basic language model, obtain The word embedding layer obtains the title embedding: ,in, The input embedding layer represents the base language model; Indicates the first Embedded vectors of each title token; for Set mask flag ;when When the title token is valid, ,when When the header token is invalid, ; Calculate the number of mask tokens for valid header tokens: ; Perform masked average pooling on the embedding vectors of all valid title tokens to obtain the title semantic representation. : ； Step S23: Let the number of supported languages ​​be: ;in, Represents the set of preset languages The number of languages ​​in; Set up a trainable language embedding table ,in, Represents a language embedding table, Represents the language embedding dimension; According to ethnic language symbols Query the trainable language embedding table to obtain the ethnic language embeddings. : ； in, Represents the language embedding query functions; It indicates the embedding of ethnic languages, used to express conditional information about different ethnic languages ​​in terms of writing systems, writing direction, word order, expression habits and data distribution; Step S24: and By concatenating the vectors, we obtain the joint condition vector. ; in, express and splicing along the feature dimension; Will Input Adaptive Gating Network This yields unnormalized weight vectors corresponding to multiple visual layers. ;in, This represents an adaptive gating network. Indicates the parameters of the gating network; Indicates the first The unnormalized weight vectors corresponding to each visual layer; The unnormalized weight vector is processed using the Softmax function. Normalization is performed to obtain the fusion weight vector of multiple visual layers. ,in, For the first The fusion weights of visual layer features satisfy the following conditions: and ; Step S25: Perform a weighted summation of the features from multiple visual layers to obtain the fused visual representation. : ； Step S26: ... The projector of the input multimodal language model is mapped from the visual feature space to the latent space of the language model, resulting in a visual token representation consistent with the word embedding dimension of the language model. ; ； in, For projectors of multimodal language models, Indicates projector parameters; Title based on ethnic language and ethnic language symbols Construct the title hint text and obtain the title hint token representation through the input embedding layer of the language model. : ； in, Indicates by and Constructed text prompts; The input embedding layer represents the language model; Representing visual tokens The token indicated in the title By concatenating the sequences, we obtain the conditional input sequence. ; Will Input Language Model Decoder The target ethnic language description Y is obtained: ； in, Description of the target ethnic language The Middle One valid target token; Represents the language model decoder and its trainable adaptation parameters; This indicates the number of tokens describing the target ethnic language. Step S27: Construct a loss function that only calculates the negative log-likelihood on the target ethnic language description token: 。 4. The adaptive visual layer fusion generation method based on ethnic language and title according to claim 3, characterized in that, Step S2 further includes: The model is trained using an autoregressive language modeling objective, and the parameters of the visual encoder and projection layer are frozen. Only the low-rank adaptation parameters, adaptive gating network parameters, and language embedding parameters of the model are updated.

5. An adaptive visual layer fusion generation system based on ethnic language and title, characterized in that, Includes the following modules: The training sample construction module is used to collect original samples and retain the metadata of the original samples; to obtain target samples by cleaning and filtering the metadata; and to construct training samples of image-title-description by describing the target samples, having them undergo expert review, difficulty labeling, and anti-leakage classification. The language title adaptive visual layer fusion model training module is used for the training phase of the language title adaptive visual layer fusion model. It takes training samples as input, extracts visual token features from multiple specified layers through a visual encoder, and generates fusion weights for visual features by constructing title semantic representation and language embedding, and finally generates fused visual representation, thereby obtaining the target ethnic language description. The Language Title Adaptive Visual Layer Fusion Model Inference Module is used during the inference phase. The user inputs an image and a national language title, and the trained Language Title Adaptive Visual Layer Fusion Model outputs a national language description.

6. An adaptive visual layer fusion generation device based on ethnic language and title, characterized in that, It includes one or more electronic devices, wherein the one or more electronic devices are used to implement the method of any one of claims 1 to 4.

7. An electronic device, characterized in that, include: One or more processors; A memory for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors cause the one or more processors to implement the method of any one of claims 1 to 4.

8. A computer-readable storage medium, characterized in that, It stores executable instructions that, when executed by a processor, cause the processor to implement the method described in any one of claims 1 to 4.

9. A non-transitory computer-readable storage medium, characterized in that, It stores a computer program that, when executed by a processor, implements the method as described in any one of claims 1 to 4.

Images