Image-text understanding method and device, equipment, medium and computer program product

Abstract

The invention provides an image-text understanding method, device and equipment, a medium and a computer program product, and the method comprises the steps: inputting a target image and an understanding instruction corresponding to an image-text understanding task into a visual language model under the condition that the image-text understanding task is received, and obtaining an answer text; under the condition that an image generation task is received, inputting a text instruction corresponding to the image generation task into a visual language model to obtain an output image; when a visual positioning task is received, inputting a target image and a positioning instruction corresponding to the visual positioning task into a visual language model to obtain image positioning information; the visual language model is obtained by training on the basis of image-text question and answer sample data, image description sample data and visual positioning sample data. Through the visual language model of the unified framework, the tasks of image-text understanding, image generation and image positioning can be completed at the same time, and the accuracy of image-text understanding is improved.

Description

Technical Field

[0001] This invention relates to the field of graphic and text understanding technology, and in particular to a graphic and text understanding method, apparatus, device, medium, and computer program product. Background Technology

[0002] The Visual Language Model (VLM) is a large model that can process both images and text simultaneously. VLM combines Computer Vision (CV) and Natural Language Processing (NLP) to understand and generate images and text. Existing VLMs have the following problems: (1) When understanding images, they can only decode the IDs of the corresponding words through mapping and can only generate text within the vocabulary range; (2) When generating images, they can only receive control text input and generate the corresponding images, lacking the ability to understand images; (3) They cannot understand visual localization tasks and cannot obtain accurate visual localization results. Summary of the Invention

[0003] This invention provides a method, apparatus, device, medium, and computer program product for understanding images and text, in order to solve the defects of inaccurate understanding in existing image and text understanding solutions.

[0004] This invention provides a method for understanding images and text, comprising the following steps: Upon receiving a text-image comprehension task, the target image and comprehension instructions corresponding to the task are input into the visual language model to obtain the response text. Upon receiving an image generation task, the text instruction corresponding to the image generation task is input into the visual language model to obtain the output image; Upon receiving a visual positioning task, the target image and positioning instructions corresponding to the visual positioning task are input into the visual language model to obtain image positioning information; The visual language model is trained based on text-to-image question-and-answer sample data, image description sample data, and visual positioning sample data.

[0005] According to a text-image understanding method provided by the present invention, the visual language model includes an image segmentation component; the text-image understanding method further includes: When the image to be processed is input into the image segmentation part, the resolution of the image to be processed is converted to obtain the converted image; The converted image is segmented using a sliding window based on the minimum segmentation window to obtain image units; the minimum segmentation window is determined based on the resolution conversion process. The converted image is converted into a thumbnail; the resolution of the thumbnail is determined based on the resolution conversion process.

[0006] According to a text-image understanding method provided by the present invention, the visual language model further includes a visual processing part and a visual transformation part; the text-image understanding method further includes: When the image unit is input into the visual processing part, the visual embedding to be processed is obtained by the visual processing part. The visual transformation section performs dimensional transformation on the visual embedding to be processed to obtain the target visual embedding.

[0007] According to a text-image understanding method provided by the present invention, the visual language model further includes a language understanding component; the text-image understanding method further includes: When the text instruction to be processed and the target visual embedding are input into the language understanding part, the embedding vector of the language tag output by the language understanding part is obtained.

[0008] According to a text-image understanding method provided by the present invention, the visual language model further includes a generation part; the text-image understanding method further includes: When the embedding vector of the language tag is input into the generation part, the text prompt information, visual positioning box and generated image output by the generation part are obtained.

[0009] According to a text-image understanding method provided by the present invention, the step of obtaining text prompts, visual positioning boxes, and generated images output by the generation part when the embedding vector of the language tag is input into the generation part includes: If the text instruction to be processed is a text-image comprehension instruction, the text prompt information output by the generating part is determined to be the text-image comprehension result of the image to be processed; If the text instruction to be processed is an image generation instruction, the generated image output by the generation part is determined to be the image generation result of the text instruction to be processed. If the text instruction to be processed is a visual positioning instruction, the visual positioning box output by the generating part is determined to be the image positioning result of the image to be processed.

[0010] The present invention also provides a graphic and text understanding device, comprising the following modules: The image and text understanding module is used to input the target image and understanding instructions corresponding to the image and text understanding task into the visual language model when receiving an image and text understanding task, so as to obtain the answer text; The image generation module is used to input the text instructions corresponding to the image generation task into the visual language model when an image generation task is received, and to obtain the output image. The image localization module is used to input the target image and localization command corresponding to the visual localization task into the visual language model when a visual localization task is received, so as to obtain image localization information. The module is used to train the visual language model based on image-text question-and-answer sample data, image description sample data, and visual positioning sample data.

[0011] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and running on the processor, wherein the processor executes the computer program to implement any of the above-described image and text understanding methods.

[0012] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the graphic understanding method as described above.

[0013] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements any of the above-described image and text understanding methods.

[0014] The image-text understanding method provided by this invention utilizes a visual language model for image-text understanding and generation, trained using image-text question-and-answer sample data, image description sample data, and visual positioning sample data. When processing image-text understanding tasks, it analyzes the target image and understanding instructions input to the model to obtain the answer text; when processing image generation tasks, it analyzes the image generation text instructions input to the model to obtain the output image; and when processing visual positioning tasks, it analyzes the target image and positioning instructions input to the model to obtain the image's positioning information. This invention, through a unified framework visual language model, can simultaneously complete image-text understanding, image generation, and image positioning tasks, thus improving the accuracy of image-text understanding. Attached Figure Description

[0015] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0016] Figure 1 This is one of the flowcharts illustrating the text-image comprehension method provided by the present invention.

[0017] Figure 2This is a schematic diagram of the overall framework and operation process of the visual language model provided by the present invention.

[0018] Figure 3 This is the second flowchart of the text and image comprehension method provided by the present invention.

[0019] Figure 4 This is a schematic diagram of the graphic understanding device provided by the present invention.

[0020] Figure 5 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0021] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0022] The following is combined Figures 1-5 This invention describes the graphic understanding method, apparatus, device, medium, and computer program product.

[0023] Figure 1 This is one of the flowcharts illustrating the text-image comprehension method provided by the present invention, such as... Figure 1 As shown, the method includes the following: Step 100: Upon receiving a text-image comprehension task, input the target image and comprehension instructions corresponding to the text-image comprehension task into the visual language model to obtain the response text; The algorithmic framework of the visual language model provided by this invention is as follows: Figure 2 As shown, when performing a text-image comprehension task using a visual language model, the target image and comprehension instructions (e.g., "describe this image," i.e., describe the target image) are input into the visual language model provided by this invention, and the response text is output through reasoning and analysis by the visual language model provided by this invention.

[0024] Step 200: Upon receiving an image generation task, input the text instruction corresponding to the image generation task into the visual language model to obtain the output image; When performing an image generation task using a visual language model, the text describing the image to be generated (i.e., the text instruction in this embodiment, such as "output a spring park image") is input into the visual language model provided by this invention. The visual language model provided by this invention analyzes the text instruction and outputs the generated image, i.e., the output image in this embodiment.

[0025] Step 300: Upon receiving a visual positioning task, input the target image and positioning command corresponding to the visual positioning task into the visual language model to obtain image positioning information; When performing a visual localization task using a visual language model, the image to be localized (i.e., the target image in this embodiment) and the localization command (e.g., "localize an element in the image") are input into the visual language model provided by this invention. The visual language model provided by this invention is used to understand and analyze the target image and the localization command to obtain the localization information of the target image, i.e., the image localization information in this embodiment.

[0026] The visual language model is trained based on text-to-image question-and-answer sample data, image description sample data, and visual positioning sample data.

[0027] The training sample data for the visual language model provided by this invention mainly includes image-text question-and-answer sample data, image description sample data, and visual positioning sample data.

[0028] Image-text question-and-answer sample data is used to train the image-text understanding ability of the visual language model. For the image-text understanding task, the goal of training the visual language model is to generate the question-and-answer text corresponding to the task, and its loss function is the language model loss (LML) of the large language model.

[0029] Image description sample data is used to train the image generation capability of the visual language model. The text description of the input image is aligned using the MSE (Mean Squared Error) loss function with the embedding output of the Stable Diffusion transformation module (a deep learning model based on diffusion techniques for text-to-image generation) and the embedding encoded by the Stable Diffusion Prompt Encoder (the prompt word encoder in the Stable Diffusion model). This ensures that the output of the Stable Diffusion transformation module directly contains the semantics encoded by the original model, thus enabling image generation using the Stable Diffusion model.

[0030] Visual localization sample data is used to train the visual localization capability of the visual language model. The input is a target image and the corresponding localization command. The embedding output by the GLIP transformation module (GLIP model, a multimodal model that combines language and vision, and achieves fine-grained visual understanding of open vocabulary by unifying target detection and phrase localization tasks) and the embedding of the localized element encoded by the GLIP Prompt Encoder are aligned through the MSE loss function. The localization box is generated by GLIP to complete the visual localization.

[0031] This embodiment utilizes a visual language model trained with image-text question-and-answer sample data, image description sample data, and visual positioning sample data to construct image-text understanding and image-text generation visual language models. When processing image-text understanding tasks, the model analyzes the target image and understanding instructions input to obtain the answer text; when processing image generation tasks, it analyzes the image generation text instructions input to obtain the output image; and when processing visual positioning tasks, it analyzes the target image and positioning instructions input to obtain the image's positioning information. This invention, through a unified framework visual language model, can simultaneously complete image-text understanding, image generation, and image positioning tasks, improving the accuracy of image-text understanding.

[0032] Figure 3 This is the second flowchart illustrating the text-image comprehension method provided by this invention, as shown below. Figure 3 As shown, the method may further include: Step 10: With the image to be processed input into the image segmentation part, perform resolution conversion processing on the image to be processed to obtain the converted image; Step 20: Perform sliding window segmentation on the converted image based on the minimum segmentation window to obtain image units; the minimum segmentation window is determined based on the resolution conversion process; Step 30: Convert the converted image into a thumbnail; the resolution of the thumbnail is determined based on the resolution conversion process.

[0033] like Figure 2 As shown, the visual language model provided by this invention includes a high-resolution image dynamic resolution segmentation module, i.e., an image segmentation part. For example, the input image is converted to a multiple of 448 through padding and scaling to obtain the converted image in this embodiment. Then, a sliding window is used to segment the converted image using a minimum segmentation window (e.g., 448×448 resolution as the minimum segmentation window) to obtain a small pixel window of a certain length. At the same time, the image to be processed is converted into a 448×448 resolution thumbnail, and the preprocessing of the high-resolution image is completed through the image segmentation part.

[0034] This embodiment completes the preprocessing of the image to be processed by the image segmentation part of the visual language model. The resulting image units and thumbnails are used for analysis in tasks such as image and text understanding.

[0035] In one embodiment, the image and text understanding method provided by this invention may further include: Step 40: After stitching the image unit and the thumbnail together and inputting them into the visual processing part, the visual embedding to be processed output by the visual processing part is obtained. Step 50: Perform dimensional transformation on the visual embedding to be processed through the visual transformation part to obtain the target visual embedding.

[0036] The visual language model provided by the present invention also includes a visual processing module, namely the visual processing part (Vision Encoder, such as Text-Enhanced Vision Transformers, ViT model) in this embodiment, and a visual conversion module, namely the visual conversion part (Vision Adaptor, such as QFormer model) in this embodiment.

[0037] The ViT model is used to process the image units obtained from the segmentation process, and the QFormer model is used as a transformation module to perform dimensional transformation on the visual embedding output by the ViT model (i.e., the visual embedding to be processed in this embodiment), so that the dimensional transformation visual embedding (i.e., the target visual embedding in this embodiment) is adapted to the input of subsequent modules, and the length of the target visual embedding is controlled.

[0038] This embodiment uses the visual processing and visual transformation parts of the visual language model to further transform the segmented image units to obtain a target visual embedding that can be input into a large language model.

[0039] In one embodiment, the image and text understanding method provided by this invention may further include: Step 60: When the text instruction to be processed and the target visual embedding are input into the language understanding part, the embedding vector of the language tag output by the language understanding part is obtained.

[0040] The image and text understanding method provided in this embodiment of the invention may further include: Step 70: When the embedding vector of the language tag is input into the generation part, the text prompt information, visual positioning box and generated image output by the generation part are obtained.

[0041] The visual language model provided by this invention also includes a large language model understanding module (i.e., the language understanding part in this embodiment) and a Glip / Stable Diffusion conversion module and a generation module (i.e., the generation part in this embodiment). The large language model receives the input target visual embedding and text instructions (i.e., the text instructions to be processed in this embodiment), and outputs the embedding vector of the language token in an autoregressive manner.

[0042] The generation part receives the embedding vectors of language tags output by the large language model. Through two different Transformer structure transformation modules, the embedding vectors of language tags are converted into fixed-length vectors, so that they express the prompt information of the original GLIP model and Stable Diffusion model. Then, through the latter, the GLIP model and Stable Diffusion model generate the corresponding visual localization boxes and generated images, and maintain semantic consistency with the prompt information.

[0043] The visual language model provided by this invention can simultaneously complete the tasks of text and image understanding, image generation, and image localization through the coordinated cooperation of various modules, thereby improving the accuracy of text and image understanding.

[0044] In one embodiment, the image and text understanding method provided by this invention may further include: Step 71: If the text instruction to be processed is a text-image comprehension instruction, determine that the text prompt information output by the generating part is the text-image comprehension result of the image to be processed; Step 72: If the text instruction to be processed is an image generation instruction, determine that the generated image output by the generation part is the image generation result of the text instruction to be processed; Step 73: If the text instruction to be processed is a visual positioning instruction, determine that the visual positioning box output by the generating part is the image positioning result of the image to be processed.

[0045] Specifically, if the text instruction to be processed is a text-image comprehension instruction (e.g., the text-image comprehension instruction is "describe the content of the image"), the visual language model provided by this invention analyzes the image to be processed to obtain the text-image comprehension result corresponding to the text-image comprehension instruction; if the text instruction to be processed is an image generation instruction (e.g., the image generation instruction is "generate an image of a spring park"), the visual language model provided by this invention analyzes the image generation instruction to obtain the image generation result corresponding to the image generation instruction; if the text instruction to be processed is a visual positioning instruction (e.g., the visual positioning instruction is "locate a certain element in the image"), the visual language model provided by this invention analyzes the image to be processed to obtain the image positioning result of the image to be processed corresponding to the visual positioning instruction.

[0046] This embodiment analyzes different text instructions to obtain different graphic understanding results, thereby improving the comprehensiveness of graphic understanding.

[0047] The image and text understanding device provided by the present invention will be described below. The image and text understanding device described below can be referred to in correspondence with the image and text understanding method described above.

[0048] Please refer to Figure 4 The present invention also provides a graphic understanding device, comprising: The image and text understanding module 401 is used to input the target image and understanding instructions corresponding to the image and text understanding task into the visual language model when receiving an image and text understanding task, so as to obtain the answer text; The image generation module 402 is used to input the text instruction corresponding to the image generation task into the visual language model when an image generation task is received, and to obtain the output image. The image localization module 403 is used to input the target image and localization command corresponding to the visual localization task into the visual language model when a visual localization task is received to obtain image localization information. The visual language model is trained based on text-to-image question-and-answer sample data, image description sample data, and visual positioning sample data.

[0049] Optionally, the visual language model includes an image segmentation component; the image-text understanding device further includes: The resolution conversion module is used to perform resolution conversion on the image to be processed when the image to be processed is input into the image segmentation part, so as to obtain a converted image; A sliding window segmentation module is used to perform sliding window segmentation on the converted image based on a minimum segmentation window to obtain image units; the minimum segmentation window is determined based on the resolution conversion process. The image conversion module is used to convert the converted image into a thumbnail; the resolution of the thumbnail is determined based on the resolution conversion process.

[0050] Optionally, the visual language model further includes a visual processing part and a visual conversion part; the graphic and text understanding device further includes: The visual embedding output module is used to obtain the visual embedding to be processed output by the visual processing part when the image unit is input into the visual processing part. The dimension conversion module is used to perform dimension conversion on the visual embedding to be processed through the visual conversion part to obtain the target visual embedding.

[0051] Optionally, the visual language model further includes a language understanding component; the image and text understanding device further includes: The embedding vector output module is used to obtain the embedding vector of the language tag output by the language understanding part when the text instruction to be processed and the target visual embedding are input into the language understanding part.

[0052] Optionally, the visual language model further includes a generation part; the image and text understanding device further includes: The model output module is used to obtain the text prompt information, visual positioning box and generated image output by the generation part when the embedding vector of the language tag is input into the generation part.

[0053] Optionally, the model output module includes: The image-text understanding result determination unit is used to determine the text prompt information output by the generating part as the image-text understanding result of the image to be processed when the text instruction to be processed is an image-text understanding instruction. The image generation result determination unit is used to determine, when the text instruction to be processed is an image generation instruction, that the generated image output by the generation part is the image generation result of the text instruction to be processed; The image localization result unit is used to determine, when the text instruction to be processed is a visual localization instruction, the visual localization box output by the generating part is the image localization result of the image to be processed.

[0054] Figure 5 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 5 As shown, the electronic device may include a processor 410, a communications interface 420, a memory 430, and a communication bus 440, wherein the processor 410, communications interface 420, and memory 430 communicate with each other via the communication bus 440. The processor 410 can call logical instructions in the memory 430 to execute a text-image understanding method. This method includes: upon receiving a text-image understanding task, inputting the target image and understanding instructions corresponding to the text-image understanding task into a visual language model to obtain response text; upon receiving an image generation task, inputting the text instructions corresponding to the image generation task into the visual language model to obtain an output image; upon receiving a visual positioning task, inputting the target image and positioning instructions corresponding to the visual positioning task into the visual language model to obtain image positioning information; the visual language model is trained based on text-image question-and-answer sample data, image description sample data, and visual positioning sample data.

[0055] Furthermore, the logical instructions in the aforementioned memory 430 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0056] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the image and text understanding methods provided by the above methods. The method includes: upon receiving an image and text understanding task, inputting the target image and understanding instructions corresponding to the image and text understanding task into a visual language model to obtain response text; upon receiving an image generation task, inputting the text instructions corresponding to the image generation task into a visual language model to obtain an output image; upon receiving a visual positioning task, inputting the target image and positioning instructions corresponding to the visual positioning task into a visual language model to obtain image positioning information; the visual language model is trained based on image and text question-and-answer sample data, image description sample data, and visual positioning sample data.

[0057] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the image-text understanding method provided by the above methods. This method includes: upon receiving an image-text understanding task, inputting the target image and understanding instructions corresponding to the task into a visual language model to obtain response text; upon receiving an image generation task, inputting the text instructions corresponding to the task into the visual language model to obtain an output image; and upon receiving a visual positioning task, inputting the target image and positioning instructions corresponding to the task into the visual language model to obtain image positioning information. The visual language model is trained based on image-text question-and-answer sample data, image description sample data, and visual positioning sample data.

[0058] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0059] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0060] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for understanding text and images, characterized in that, include: Upon receiving a text-image comprehension task, the target image and comprehension instructions corresponding to the task are input into the visual language model to obtain the response text. Upon receiving an image generation task, the text instruction corresponding to the image generation task is input into the visual language model to obtain the output image; Upon receiving a visual positioning task, the target image and positioning instructions corresponding to the visual positioning task are input into the visual language model to obtain image positioning information; The visual language model is trained based on text-to-image question-and-answer sample data, image description sample data, and visual positioning sample data.

2. The image and text understanding method according to claim 1, characterized in that, The visual language model includes an image segmentation component; the image-text understanding method further includes: When the image to be processed is input into the image segmentation part, the resolution of the image to be processed is converted to obtain the converted image; The converted image is segmented using a sliding window based on the minimum segmentation window to obtain image units; the minimum segmentation window is determined based on the resolution conversion process. The converted image is converted into a thumbnail; the resolution of the thumbnail is determined based on the resolution conversion process.

3. The image and text understanding method according to claim 2, characterized in that, The visual language model further includes a visual processing component and a visual transformation component; the image and text understanding method further includes: When the image unit and the thumbnail are stitched together and then input into the visual processing part, the visual embedding to be processed output by the visual processing part is obtained. The visual transformation section performs dimensional transformation on the visual embedding to be processed to obtain the target visual embedding.

4. The image and text understanding method according to claim 3, characterized in that, The visual language model also includes a language understanding component; the image-text understanding method further includes: When the text instruction to be processed and the target visual embedding are input into the language understanding part, the embedding vector of the language tag output by the language understanding part is obtained.

5. The image and text understanding method according to claim 4, characterized in that, The visual language model further includes a generation component; the image-text understanding method further includes: When the embedding vector of the language tag is input into the generation part, the text prompt information, visual positioning box and generated image output by the generation part are obtained.

6. The image and text understanding method according to claim 5, characterized in that, The step of obtaining the text prompt information, visual positioning box, and generated image output by the generation part when the embedding vector of the language tag is input into the generation part includes: If the text instruction to be processed is a text-image comprehension instruction, the text prompt information output by the generating part is determined to be the text-image comprehension result of the image to be processed; If the text instruction to be processed is an image generation instruction, the generated image output by the generation part is determined to be the image generation result of the text instruction to be processed. If the text instruction to be processed is a visual positioning instruction, the visual positioning box output by the generating part is determined to be the image positioning result of the image to be processed.

7. A graphic and text understanding device, characterized in that, include: The image and text understanding module is used to input the target image and understanding instructions corresponding to the image and text understanding task into the visual language model when receiving an image and text understanding task, so as to obtain the answer text; The image generation module is used to input the text instructions corresponding to the image generation task into the visual language model when an image generation task is received, and to obtain the output image. The image localization module is used to input the target image and localization command corresponding to the visual localization task into the visual language model when a visual localization task is received, so as to obtain image localization information. The visual language model is trained based on text-to-image question-and-answer sample data, image description sample data, and visual positioning sample data.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the image and text understanding method as described in any one of claims 1 to 6.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the image and text understanding method as described in any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the image and text understanding method as described in any one of claims 1 to 6.

Images