Methods, apparatus, devices, and media for automatic generation of zero-sample image text descriptions

Abstract

This application relates to a method, apparatus, device, and medium for automatically generating zero-shot image text descriptions. It includes: processing acquired text data to obtain local text signals and text embeddings; processing the text embeddings using a hierarchical mapper to obtain a global text signal; concatenating the local and global text signals and inputting them into a language model for autoregressive language modeling training; fixing the trained hierarchical mapper and the language model; acquiring the category information of the image to be tested, correcting the logical values ​​of the category information based on an entropy-based retrieval strategy, and acquiring the image local signal; extracting image features from the image to be tested, inputting the image features into the fixed hierarchical mapper for processing to obtain the image global signal; and concatenating the local and global image signals and inputting them into the fixed language model to obtain a text description of the image to be tested. This invention can accurately describe novel, unseen objects and scenes.

Description

Technical Field

[0001] This application relates to the field of information processing technology, and in particular to a method, apparatus, device and medium for automatically generating zero-sample image text descriptions. Background Technology

[0002] Image text description generation aims to automatically generate fluent descriptive text sentences for images, a task crucial for visual language understanding. Since 2014, thanks to the revolutionary development of deep neural networks, image text description generation has made breakthrough progress on complex and challenging benchmarks. While fully supervised image text description generation models perform exceptionally well on standard evaluation metrics, these models rely on large amounts of image text annotation data (such as MSCOCO) for training. However, constructing a large-scale image text description dataset covering different domains and styles requires significant human, material, and financial resources. Furthermore, fully supervised image text description generation models, once trained in the source domain, often struggle to perform equally well in the target domain with different data distributions, thus limiting their practical applicability. Therefore, it is essential to implement image text description generation in real-world scenarios where image text data is unavailable. Typically, humans can quickly extract useful information from unfamiliar things based on their experience; this ability is known as zero-shot learning. One of its key features is that it can identify new things based solely on feature descriptions without providing any new data. Zero-shot image text description generation is currently receiving increasing attention because it not only reduces the cost of dataset annotation but also facilitates more effective human-computer interaction in the real world.

[0003] Both vision and language can be used to explain the world, and they are closely related. This common sense has driven research into large-scale pre-trained vision-language models. Some existing work has explored combining generative language models with pre-trained large-scale vision-language models to achieve zero-shot image text description generation tasks. For example, ViECap uses a CLIP-based classifier to directly search for entities as hard cues to guide the language model to decode in the direction of visual elements in the image. However, ViECap only focuses on salient entities, ignoring small objects, resulting in generated text descriptions that can only be matched intuitively with the image. Furthermore, as a complement to hard cues, soft cues should contain richer information related to the attributes and relationships of entities. However, ViECap only uses a simple Transformer to learn soft cues, which is insufficient to capture the detailed information embedded in the different levels of the CLIP joint space. Summary of the Invention

[0004] Therefore, it is necessary to provide a method, apparatus, device, and medium for automatically generating zero-sample image text descriptions that can focus on small targets, capture detailed information at different levels, and accurately describe novel, unseen objects and scenes, in order to address the above-mentioned technical problems.

[0005] A method for automatically generating zero-sample image text descriptions, the method comprising:

[0006] Get text data;

[0007] The text data is processed to obtain local text signals and text embeddings; the text embeddings are then processed by a hierarchical mapper to obtain global text signals.

[0008] The local text signal and the global text signal are concatenated and then input into the language model to perform autoregressive language modeling training on the language model.

[0009] The trained hierarchical mapper is fixed to the language model;

[0010] Obtain the image to be tested;

[0011] The image to be tested is classified to obtain category information. The logical value of the category information is corrected based on an entropy-based retrieval strategy to obtain the local signal of the image.

[0012] Feature extraction is performed on the image to be tested to obtain image features. The image features are then input into a fixed hierarchical mapper for processing to obtain the global image signal.

[0013] The local signal and the global signal of the image are concatenated and then input into a fixed language model to obtain a text description of the image to be tested.

[0014] An apparatus for automatically generating zero-sample image text descriptions, the apparatus comprising:

[0015] The text acquisition module is used to acquire text data;

[0016] The text processing module is used to process the text data to obtain local text signals and text embeddings; and to process the text embeddings through a hierarchical mapper to obtain global text signals.

[0017] The model training module is used to concatenate the local signals of the text with the global signals of the text and input them into the language model to perform autoregressive language modeling training on the language model.

[0018] The model fixing module is used to fix the trained hierarchical mapper to the language model;

[0019] The image acquisition module is used to acquire the image to be tested.

[0020] The image local signal acquisition module is used to classify the image to be tested to obtain category information, and to correct the logical value of the category information based on an entropy retrieval strategy to acquire the image local signal.

[0021] The image global signal acquisition module is used to extract features from the image to be tested, obtain image features, and input the image features into a fixed hierarchical mapper for processing to obtain the image global signal.

[0022] The image text description automatic generation module is used to concatenate the local signal and the global signal of the image and input them into a fixed language model to obtain the text description of the image to be tested.

[0023] A computer device includes a memory and a processor, the memory storing a computer program, the processor executing the computer program to implement the steps of the method.

[0024] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method.

[0025] The aforementioned method, apparatus, device, and medium for automatically generating zero-sample image text descriptions involve: acquiring text data; processing the text data to obtain local text signals and text embeddings; processing the text embeddings through a hierarchical mapper to obtain global text signals; concatenating the local text signals and the global text signals and inputting them into a language model; training the language model using autoregressive language modeling; fixing the trained hierarchical mapper and the language model; acquiring a test image; classifying the test image to obtain category information; correcting the logical values ​​of the category information using an entropy-based retrieval strategy to obtain local image signals; extracting features from the test image to obtain image features; inputting the image features into the fixed hierarchical mapper for processing to obtain global image signals; and concatenating the local image signals and the global image signals and inputting them into the fixed language model to obtain a text description of the test image.

[0026] This invention captures and fuses features from different levels through a hierarchical mapper to capture detailed information at different levels. It also uses an entropy-based retrieval strategy to correct the logical values ​​of category information, enabling the model to focus on low-frequency targets. This improves the model's zero-shot image text description generation capability in different scenarios and demonstrates strong transfer adaptability across different data domains. The proposed zero-shot image text description automatic generation method can accurately describe novel, unseen objects and scenes when a computer "sees" an image, thereby enhancing the practicality of image text description generation tasks in real-world scenarios. Attached Figure Description

[0027] Figure 1 This is a schematic diagram of a method for automatically generating text descriptions for zero-sample images in one embodiment;

[0028] Figure 2 This is a schematic diagram of a framework for automatically generating zero-sample image text descriptions in one embodiment.

[0029] Figure 3 This is a structural block diagram of a zero-sample image text description automatic generation device in one embodiment;

[0030] Figure 4 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0031] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0032] In implementing this solution, the inventors discovered that existing models focus on salient entities while neglecting small targets. However, inconspicuous small targets in images often significantly impact the generation of accurate and comprehensive text descriptions. Furthermore, existing soft cue learning methods fail to capture detailed information at different levels. Based on this, the inventors proposed a zero-shot image text description automatic generation model—EntroCap. By combining entropy and a pre-trained vision-language model, EntroCap improves the accuracy of describing low-frequency small targets and enhances the model's transferability. A hierarchical mapper composed of varying numbers of Transformer layers is introduced to capture detailed information at different levels. When using EntroCap for zero-shot inference, it can accurately describe novel and unseen entities and scenes. By inputting an image, it outputs a natural language description, achieving the goal of automatic zero-shot image text description generation.

[0033] The zero-shot image text description automatic generation method provided by this invention is an efficient and reliable cross-domain image text description generation method. That is, when EntroCap sees a new object or a new image distribution that it did not see during training, it can generate a reasonable natural language description that is relevant to the image content based on its stored knowledge.

[0034] The embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0035] In one embodiment, such as Figure 1 As shown, a method for automatically generating zero-sample image text descriptions is provided, including the following steps:

[0036] Step 202: Obtain text data.

[0037] It is understandable that the acquired text data is a plain text corpus D, and EntroCap is trained on the plain text corpus D.

[0038] Step 204: Process the text data to obtain local text signals and text embeddings; process the text embeddings through a hierarchical mapper to obtain global text signals.

[0039] It is understandable that the training process involves two branches; see reference. Figure 2 This paper presents a schematic diagram of the framework for automatic generation of text descriptions from zero-sample images. As shown in the diagram, in the first branch, noun objects parsed from the text data or retrieved from the test image are treated as local concepts and converted into local text signals P using a predefined cue template. Tlocal In the second branch, a hierarchical projector consisting of varying numbers of Transformer layers is introduced. By embedding the text extracted by CLIP into different layers of the hierarchical projector, features from different layers are fused to generate a global text signal P. Tglobal .

[0040] It is worth noting that, due to the strong cross-modal characteristics of CLIP in EntroCap, the hierarchical mapper, after being trained on plain text data, can directly receive the visual embedding of CLIP as input and obtain the global image signal P during inference. Vglobal .

[0041] Step 206: After concatenating the local and global signals of the text, input the concatenation into the language model to train the language model for autoregressive language modeling.

[0042] It is understandable that the language model here is a generative language model, such as GPT-2. The language model is trained through autoregressive language modeling by using local and global signals from the text until the language model performs well on the training data and reaches the expected results. Once this is achieved, training stops, resulting in a trained hierarchical mapper and language model.

[0043] Step 208: Fix the trained hierarchical mapper with the language model.

[0044] It can be understood that steps 202 to 206 are the process of training the hierarchical mapper and language model in the EntroCap model. After fixing the trained hierarchical mapper and language model, the text description reasoning ability of the EntroCap model is tested using the test image. That is, after training the hierarchical mapper and language model on a pure text dataset, they are fixed and directly used for zero-sample image text description generation during the reasoning process. The reasoning process receives the image as input, and similarly, the reasoning process also consists of two branches, used to obtain the local signal and the global signal of the image, respectively.

[0045] Step 210: Obtain the image to be tested.

[0046] Step 212: Classify the image to be tested to obtain category information. Correct the logical value of the category information based on the entropy retrieval strategy to obtain the local signal of the image.

[0047] Specifically, the test image is input into a zero-shot classifier for classification to retrieve objects appearing in the test image, obtaining category information, which is used as a local concept of the image. Then, an entropy-based retrieval strategy is used to correct the logical value of the category information, retrieving low-frequency small targets, which are then used as a new local concept, i.e., the local image signal P. Vlocal .

[0048] Entropy can be used to measure the sparsity of a probability distribution; the smaller the entropy, the more concentrated the probabilities are in a few categories. In other words, the probabilities of other categories are closer to zero. Therefore, in the inference process, to retrieve low-frequency small targets from the test image, an entropy-based retrieval strategy is designed to correct the logical values ​​of the category information output by the CLIP-based zero-shot classifier. The low-frequency small targets are retrieved using the entropy-based retrieval strategy and used as the local image signal P. Vlocal .

[0049] Step 214: Extract features from the image to be tested to obtain image features. Input the image features into the fixed hierarchical mapper for processing to obtain the global image signal.

[0050] It is understandable that the CLIP image encoder extracts the image embedding, and then the extracted image embedding is input into a pre-trained hierarchical mapper to learn the overall context of the image and use it as global information of the image.

[0051] Step 216: After concatenating the local signal and the global signal of the image, input the concatenated signal into the fixed language model to obtain the text description of the image to be tested.

[0052] The above-described method for automatically generating text descriptions of zero-sample images involves: acquiring text data; processing the text data to obtain local text signals and text embeddings; processing the text embeddings using a hierarchical mapper to obtain global text signals; concatenating the local text signals and the global text signals and inputting them into a language model for autoregressive language modeling training; fixing the trained hierarchical mapper and the language model; acquiring a test image; classifying the test image to obtain category information; correcting the logical values ​​of the category information using an entropy-based retrieval strategy to obtain local image signals; extracting features from the test image to obtain image features; inputting the image features into the fixed hierarchical mapper for processing to obtain global image signals; and concatenating the local image signals and the global image signals and inputting them into the fixed language model to obtain the text description of the test image.

[0053] This invention captures and fuses features from different levels through a hierarchical mapper to capture detailed information at different levels. It also uses an entropy-based retrieval strategy to correct the logical values ​​of category information, enabling the model to focus on low-frequency targets. This improves the model's zero-shot image text description generation capability in different scenarios and demonstrates strong transfer adaptability across different data domains. The proposed zero-shot image text description automatic generation method can accurately describe novel, unseen objects and scenes when a computer "sees" an image, thereby enhancing the practicality of image text description generation tasks in real-world scenarios.

[0054] In one embodiment, text data is processed to obtain local text signals and text embeddings, including:

[0055] The text data is processed by a parser to obtain local text concepts. After random masking of the local text concepts, the random masked local concepts are converted into local text signals using a pre-defined prompt template. The text data is then processed by a CLIP text encoder to obtain text embeddings.

[0056] Specifically, a grammar parser is used to capture nouns as local concepts from the text in the plain text corpus D, and a corpus of all local concepts v is constructed. i The conceptual vocabulary V is composed of these elements.

[0057] It is worth noting that, in order to enhance the robustness of the model while maintaining its generalization ability and achieving excellent zero-shot image text description generation, this invention performs a random masking operation on local concepts with probability r. Then, it uses a pre-set cue template to obtain the local text signal P. Tlocal , is represented as:

[0058] [There][are][v1],...,[v n [in][the][image][.];

[0059] See Figure 2 Taking the training text "A man with a red helmet on a small moped on a dirtroad" as an example, firstly, a parser is used to obtain a set of local concepts C = {man, helmet, moped, road} composed of nouns in the text. Secondly, the local concepts [helmet] and [moped] are randomly masked, meaning they will not appear in the local signal later. Finally, during training, the text local signal P... Tlocal Represented as:

[0060] P Tlocal =[There][are][v1],...,[v n [in][the][image][.].

[0061] In one embodiment, the text embedding is processed by a hierarchical mapper to obtain the global text signal, the process of which is represented as follows:

[0062] T CLIP =CLIP text (D), D = {T1,T2,...,T} m};

[0063] HP1 = (TransLayer(T) CLIP ),M1,θ1);

[0064] HP2 = (TransLayer(T) CLIP ),M2,θ2);

[0065] HP3 = (TransLayer(T) CLIP ),M3,θ3);

[0066] P Tglobal =HP1 + HP2 + HP3;

[0067] In the formula, CLIP text (·) indicates CLIP text encoder; T CLIP Represents text embedding; TransLayer(·) represents a Transformer layer consisting of self-attention, layer normalization, and feedforward layers; M i HP represents the i-th level of the hierarchy mapper; i Represents the mapper of the i-th layer; θ i P represents the learnable parameters in the mapper of the i-th layer; Tglobal This indicates a global signal for the text.

[0068] It is worth noting that in the EntroCap proposed in this invention, considering that the global signal should carry information complementary to the local signal, a novel hierarchical mapper is designed in the second branch. By setting different numbers of Transformer layers at different levels, the text embedding TCLIP of CLIP is converted into a global signal. The CLIP text embedding is simultaneously input into different levels of the hierarchical mapper, and then the shallow text features are combined with the deep text features to obtain the text global signal P. Tglobal .

[0069] The number of levels in the hierarchical mapper is set according to the situation. In this embodiment, the hierarchical mapper includes three levels, each level consisting of M. i It consists of stacked Transformer layers.

[0070] In one embodiment, an entropy-based retrieval strategy corrects the logical values ​​of category information to obtain local image signals, including:

[0071] The logical values ​​of the category information are normalized to obtain normalized logical values; the normalized logical values ​​are then exponentialized to obtain the probability values ​​of the category information; the entropy of each category is calculated based on the normalized logical values ​​and the probability values ​​of the category information; the normalized logical values ​​are then corrected using the entropy of each category to obtain corrected logical values; the probabilities and indices corresponding to the top few maximum values ​​in the corrected logical values ​​are selected as the local signals of the image.

[0072] EntroCap employs an entropy-based retrieval strategy to correct the logistic values ​​of categories retrieved by its CLIP-based zero-shot classifier. The rationale behind this design lies in the fact that entropy is the average amount of information contained in a series of events, which can be used to measure the sparsity of a probability distribution. By modifying the logistic values ​​of the output category information using entropy, less prominent small objects in the image can be captured as new visual concepts, thus contributing to the generation of more accurate and reasonable image-text descriptions.

[0073] In one embodiment, the image features are input into a fixed hierarchical mapper for processing to obtain the global image signal. The process is represented as follows:

[0074] I CLIP =CLIP vision (I);

[0075] HP1 = (TransLayer(I) CLIP ),M1,θ1);

[0076] HP2 = (TransLayer(I) CLIP ),M2,θ2);

[0077] HP3 = (TransLayer(I) CLIP ),M3,θ3);

[0078] P Vglobal =HP1 + HP2 + HP3;

[0079] In the formula, CLIP vision (·) indicates the CLIP image encoder; I CLIP Represents image embedding; TransLayer(·) represents a Transformer layer; M i HP represents the i-th level of the hierarchy mapper; i Represents the mapper of the i-th layer; θ i P represents the learnable parameters in the mapper of the i-th layer; Vglobal This represents the global signal of the image.

[0080] It is understandable that CLIP, due to its cross-modal nature, allows for the matching of text embeddings and image embeddings. After training the hierarchical mapper using a text corpus, it can be fixed during inference and directly used to obtain the global signal of the test image I, i.e., the image global signal P. Vglobal .

[0081] In one embodiment, the image local signal and the image global signal are concatenated and then input into a fixed language model, including:

[0082] In the local signal P of the imageVlocal With the global signal P of the image Vglobal During stitching, weights are assigned using a balancing gate; then the weighted local image signal P is... Vlocal With the global signal P of the image Vglobal The language model is then input after being concatenated.

[0083] It's worth noting that the hierarchical mapper is trained on the visible domain, which may lead to a performance degradation of global signals from the unknown domain during inference. Local signals, on the other hand, provide more explicit visual cues and are crucial for the transferability of the zero-shot image-text description generation model across different scenarios. Therefore, a balancing gate is designed to comprehensively consider both local and global signals during inference, weighing their contributions to the language model to achieve efficient zero-shot image-text description generation. In this embodiment, the balancing gate assigns more weight to local signals than to global signals.

[0084] In one embodiment, when stitching the local signal of the image with the global signal of the image, a weight allocation is performed using a balancing gate, and the process is represented as follows:

[0085]

[0086] In the formula, P V This represents the visual signal obtained by weighting using a balance gate; α and β represent the balance coefficients; [;] represents the connection operation.

[0087] It's understandable, P V It includes local details of the retrieved discrete categories and overall contextual information from CLIP image embeddings. The design of the balanced gate facilitates the generation of more explicit textual descriptions that are relevant to the visual content of the images, and promotes language modeling.

[0088] It should be understood that, although Figure 1 The steps in the flowchart are shown sequentially as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order in which these steps are executed, and they can be performed in other orders. Figure 1 At least some of the steps in the process may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least some of the sub-steps or stages of other steps.

[0089] In one embodiment, such as Figure 3As shown, a zero-shot image text description automatic generation device is provided, including: a text acquisition module 402, a text processing module 404, a model training module 406, a model fixing module 408, an image acquisition module 410, an image local signal acquisition module 412, an image global signal acquisition module 414, and an image text description automatic generation module 416, wherein:

[0090] The text acquisition module 402 is used to acquire text data.

[0091] The text processing module 404 is used to process text data to obtain local text signals and text embeddings; and to process the text embeddings through a hierarchical mapper to obtain global text signals.

[0092] The model training module 406 is used to concatenate the local and global signals of the text and input them into the language model to perform autoregressive language modeling training on the language model.

[0093] The model fixing module 408 is used to fix the trained hierarchical mapper to the language model;

[0094] Image acquisition module 410 is used to acquire the image to be tested;

[0095] The image local signal acquisition module 412 is used to classify the image to be tested, obtain category information, and correct the logical value of the category information based on an entropy-based retrieval strategy to acquire the image local signal.

[0096] The global image signal acquisition module 414 is used to extract features from the image to be tested, obtain image features, and input the image features into a fixed hierarchical mapper for processing to obtain the global image signal.

[0097] The image text description automatic generation module 416 is used to concatenate the local signal and the global signal of the image and input them into a fixed language model to obtain the text description of the image to be tested.

[0098] Specific limitations regarding the zero-shot image text description automatic generation device can be found in the limitations of the zero-shot image text description automatic generation method described above, and will not be repeated here. Each module in the aforementioned zero-shot image text description automatic generation device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0099] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 4As shown, the computer device includes a processor, memory, network interface, and database connected via a system bus. The processor provides computational and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores the operating system, computer programs, and the database. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage medium. The database stores data for a zero-shot image-text description automatic generation method. The network interface communicates with external terminals via a network connection. When executed by the processor, the computer program implements a zero-shot image-text description automatic generation method.

[0100] Those skilled in the art will understand that Figure 4 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0101] In one embodiment, a computer device is provided, including a memory and a processor, the memory storing a computer program, the processor executing the computer program to perform the following steps:

[0102] Step 202: Obtain text data.

[0103] Step 204: Process the text data to obtain local text signals and text embeddings; process the text embeddings through a hierarchical mapper to obtain global text signals.

[0104] Step 206: After concatenating the local and global signals of the text, input the concatenation into the language model to train the language model for autoregressive language modeling.

[0105] Step 208: Fix the trained hierarchical mapper with the language model.

[0106] Step 210: Obtain the image to be tested.

[0107] Step 212: Classify the image to be tested to obtain category information. Correct the logical value of the category information based on the entropy retrieval strategy to obtain the local signal of the image.

[0108] Step 214: Extract features from the image to be tested to obtain image features. Input the image features into the fixed hierarchical mapper for processing to obtain the global image signal.

[0109] Step 216: After concatenating the local signal and the global signal of the image, input the concatenated signal into the fixed language model to obtain the text description of the image to be tested.

[0110] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, the computer program performing the following steps when executed by a processor:

[0111] Step 202: Obtain text data.

[0112] Step 204: Process the text data to obtain local text signals and text embeddings; process the text embeddings through a hierarchical mapper to obtain global text signals.

[0113] Step 206: After concatenating the local and global signals of the text, input the concatenation into the language model to train the language model for autoregressive language modeling.

[0114] Step 208: Fix the trained hierarchical mapper with the language model.

[0115] Step 210: Obtain the image to be tested.

[0116] Step 212: Classify the image to be tested to obtain category information. Correct the logical value of the category information based on the entropy retrieval strategy to obtain the local signal of the image.

[0117] Step 214: Extract features from the image to be tested to obtain image features. Input the image features into the fixed hierarchical mapper for processing to obtain the global image signal.

[0118] Step 216: After concatenating the local signal and the global signal of the image, input the concatenated signal into the fixed language model to obtain the text description of the image to be tested.

[0119] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0120] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0121] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A method for automatically generating zero-sample image text descriptions, characterized in that, The method includes: Get text data; The text data is processed to obtain local text signals and text embeddings; the text embeddings are then processed by a hierarchical mapper to obtain global text signals. The local text signal and the global text signal are concatenated and then input into the language model to perform autoregressive language modeling training on the language model. The trained hierarchical mapper is fixed to the language model; Obtain the image to be tested; The image to be tested is classified to obtain category information. The logical value of the category information is corrected based on an entropy-based retrieval strategy to obtain the local signal of the image. Feature extraction is performed on the image to be tested to obtain image features. The image features are then input into a fixed hierarchical mapper for processing to obtain the global image signal. The local signal and the global signal of the image are concatenated and then input into a fixed language model to obtain a text description of the image to be tested. An entropy-based retrieval strategy corrects the logical values ​​of the category information to obtain local image signals, including: The logical values ​​of the category information are normalized to obtain normalized logical values; The normalized logical value is subjected to an exponentialization operation to obtain the probability value of the category information. The entropy of each category is calculated based on the normalized logical value and the probability value of the category information. The normalized logical value is corrected by adjusting the entropy of each category to obtain the corrected logical value. The probabilities and indices corresponding to the first few maximum values ​​in the corrected logical values ​​are selected as the local signals of the image.

2. The method for automatically generating zero-sample image text descriptions according to claim 1, characterized in that, The text data is processed to obtain local text signals and text embeddings, including: The text data is processed by a parser to obtain local text concepts. After random masking of the local text concepts, the random masked local concepts are converted into local text signals through a pre-set prompt template. The text data is processed by the CLIP text encoder to obtain the text embedding.

3. The method for automatically generating zero-sample image text descriptions according to claim 2, characterized in that, The text embedding is processed by a hierarchical mapper to obtain the global text signal, the process of which is represented as follows: ； ； ； ； ； In the formula, Indicates CLIP text encoder; Indicates text embedding; Represents the Transformer layer; The first level of the hierarchical mapper layer; Indicates the first Layer mapper; Indicates the first Learnable parameters in the layer mapper; This indicates a global signal for the text.

4. The method for automatically generating zero-sample image text descriptions according to claim 1, characterized in that, The image features are input into a fixed hierarchical mapper for processing to obtain the global image signal. The process is as follows: ； ； ； ； ； In the formula, Indicates CLIP image encoder; Indicates image embedding; Represents the Transformer layer; The first level of the hierarchical mapper layer; Indicates the first Layer mapper; Indicates the first Learnable parameters in the layer mapper; This represents the global signal of the image.

5. The method for automatically generating zero-sample image text descriptions according to claim 4, characterized in that, The image local signal and the image global signal are concatenated and then input into a fixed language model, including: When concatenating the local image signal and the global image signal, a weight allocation is performed using a balancing gate; then, the weighted local image signal and the global image signal are concatenated and input into the fixed language model.

6. The method for automatically generating zero-sample image text descriptions according to claim 5, characterized in that, When concatenating the local and global signals of the image, a weight allocation is performed using a balancing gate. The process is as follows: ； In the formula, This represents the visual signal obtained by weighting using a balanced gate; and Indicates the balance coefficient; This indicates a join operation.

7. An automatic image text description generation device with zero samples, characterized in that, The device includes: The text acquisition module is used to acquire text data; The text processing module is used to process the text data to obtain local text signals and text embeddings; and to process the text embeddings through a hierarchical mapper to obtain global text signals. The model training module is used to concatenate the local signals of the text with the global signals of the text and input them into the language model to perform autoregressive language modeling training on the language model. The model fixing module is used to fix the trained hierarchical mapper to the language model; The image acquisition module is used to acquire the image to be tested. The image local signal acquisition module is used to classify the image to be tested to obtain category information, and to correct the logical value of the category information based on an entropy retrieval strategy to acquire the image local signal. The image global signal acquisition module is used to extract features from the image to be tested, obtain image features, and input the image features into a fixed hierarchical mapper for processing to obtain the image global signal. An image text description automatic generation module is used to concatenate the local signal and the global signal of the image and input them into a fixed language model to obtain a text description of the image to be tested. In the image local signal acquisition module, the logical values ​​of the category information are corrected based on an entropy-based retrieval strategy to acquire the image local signal, including: The logical values ​​of the category information are normalized to obtain normalized logical values; The normalized logical value is subjected to an exponentialization operation to obtain the probability value of the category information. The entropy of each category is calculated based on the normalized logical value and the probability value of the category information. The normalized logical value is corrected by adjusting the entropy of each category to obtain the corrected logical value. The probabilities and indices corresponding to the first few maximum values ​​in the corrected logical values ​​are selected as the local signals of the image.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

Images