Image processing method, electronic device and computer program product

Abstract

The invention provides an image processing method and device, electronic equipment, a computer readable storage medium and a computer program product. The method comprises the following steps: determining enhancement information of a first image, wherein the enhancement information comprises information for enhancing semantics of the first image; determining a second image belonging to the same category as the first image, and determining an information extraction example of the second image; combining the first image, the information extraction instruction for the first image, the enhanced information and the information extraction example to obtain a cue word; and inputting the cue word into a first model for text generation to obtain first information for describing the first image. According to the invention, the comprehensiveness and accuracy of information extraction for the image can be improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, and in particular to an image processing method, an electronic device, and a computer program product. BACKGROUND

[0002] In the current digital era, image information extraction and description are of great significance to many fields, such as image retrieval, intelligent album management, image content analysis and understanding, etc. Related technologies often focus on directly extracting limited and fixed mode information from images themselves or example samples. In the case where an image includes a large amount of redundant information, the efficiency of image processing is low, and the interference of redundant information leads to low accuracy of information extraction. SUMMARY

[0003] Embodiments of the present application provide an image processing method, device, electronic device, computer readable storage medium and computer program product, which can improve the comprehensiveness and accuracy of information extraction for images.

[0004] The technical solutions of the embodiments of the present application are implemented as follows:

[0005] An image processing method is provided in an embodiment of the present application, which includes:

[0006] determining enhancement information of a first image, the enhancement information including information for enhancing semantics of the first image;

[0007] determining a second image belonging to the same category as the first image, and determining information extraction examples of the second image;

[0008] combining the first image, information extraction instructions for the first image, the enhancement information and the information extraction examples to obtain a prompt word;

[0009] inputting the prompt word into a first model to generate text, and obtaining first information for describing the first image.

[0010] An image processing device is provided in an embodiment of the present application, which includes:

[0011] a first determining module configured to determine enhancement information of a first image, the enhancement information including information for enhancing semantics of the first image;

[0012] a second determining module configured to determine a second image belonging to the same category as the first image, and determine information extraction examples of the second image;

[0013] a data combining module, configured to combine the first image, the information extraction instruction for the first image, the enhanced information, and the information extraction example to obtain a prompt word;

[0014] The text generation module is used to input the prompt word into the first model to generate text and obtain first information for describing the first image.

[0015] An embodiment of the present application provides an electronic device, comprising:

[0016] a memory for storing computer-executable instructions or computer programs;

[0017] The processor is used to implement the image processing method provided in the embodiment of the present application when executing the computer executable instructions or computer program stored in the memory.

[0018] An embodiment of the present application provides a computer-readable storage medium storing a computer program or computer-executable instructions for implementing the image processing method provided in the embodiment of the present application when executed by a processor.

[0019] An embodiment of the present application provides a computer program product, including a computer program or computer-executable instructions. When the computer program or computer-executable instructions are executed by a processor, the image processing method provided in the embodiment of the present application is implemented.

[0020] The embodiments of the present application have the following beneficial effects:

[0021] By combining a first image, an information extraction instruction for the first image, enhanced information of the first image, and an information extraction example of a second image belonging to the same category as the first image into a prompt word, and calling a pre-trained first model for text generation based on the prompt word, first information used to describe the first image is obtained. By introducing enhanced information, semantic information not directly reflected in the first image itself can be supplemented, helping the first model to understand the image content more deeply, thereby generating a more comprehensive and accurate image description. By using the information extraction example of the second image of the same category, the information extraction experience of the same type of second image can be drawn on, so that the first model can quickly locate key information when processing the first image, reduce ineffective information processing processes, and improve the efficiency and accuracy of information extraction. Combining information of multiple dimensions into prompt words provides the first model with complete and targeted input during the text generation process, thereby improving the comprehensiveness and accuracy of the first information generated by the first model. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 1 is a schematic diagram of the architecture of an image processing system 100 provided in an embodiment of the present application;

[0023] Figure 2 FIG. 1 is a structural schematic diagram of an electronic device 500 provided by an embodiment of the present application;

[0024] Figure 3A FIG. 2 is a first flowchart of a processing method of an image provided by an embodiment of the present application;

[0025] Figure 3B FIG. 3 is a second flowchart of a processing method of an image provided by an embodiment of the present application;

[0026] Figure 3C FIG. 4 is a third flowchart of a processing method of an image provided by an embodiment of the present application;

[0027] Figure 3D FIG. 5 is a fourth flowchart of a processing method of an image provided by an embodiment of the present application;

[0028] Figure 3E FIG. 6 is a fifth flowchart of a processing method of an image provided by an embodiment of the present application;

[0029] Figure 3F FIG. 7 is a sixth flowchart of a processing method of an image provided by an embodiment of the present application;

[0030] Figure 3G FIG. 8 is a seventh flowchart of a processing method of an image provided by an embodiment of the present application;

[0031] Figure 3H FIG. 9 is an eighth flowchart of a processing method of an image provided by an embodiment of the present application;

[0032] Figure 4 FIG. 10 is a flowchart of a multi-modal inference provided by an embodiment of the present application;

[0033] Figure 5A FIG. 11 is a schematic diagram of a related technology for position encoding;

[0034] Figure 5B FIG. 12 is a schematic diagram of multi-modal information for position encoding.

[0035] It should be noted that the above-mentioned "first", "second" are only used to distinguish different schemes, and do not represent the degree of superiority or priority in the implementation process. DETAILED DESCRIPTION

[0036] In order to make the purpose, technical scheme and advantages of the present application clearer, the present application will be described in further detail below with reference to the drawings, and the described embodiments should not be regarded as limiting the present application, and all other embodiments obtained by a person of ordinary skill in the art without making creative labor shall fall within the scope of protection of the present application.

[0037] In the following description, reference is made to "some embodiments", which describe a subset of all possible embodiments, but it is understood that "some embodiments" can be the same subset or a different subset of all possible embodiments, and can be combined with each other, without conflict.

[0038] In the following description, the terms "first / second / third" are merely used to distinguish similar objects, and do not represent a specific order of the objects. It is understood that the "first / second / third" can be interchanged in a specific order or sequence as allowed, so that the embodiments described herein can be implemented in an order other than that illustrated or described herein.

[0039] In the embodiments of the present application, the term "module" or "unit" refers to a computer program or a part of a computer program with a predetermined function, and works together with other related parts to achieve a predetermined target, and can be implemented entirely or partially by using software, hardware (such as a processing circuit or a memory) or a combination thereof. Similarly, one processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be a part of an integral module or unit that includes the functions of the module or unit.

[0040] Unless otherwise specified, at least one of the following described below refers to one or more cases, and "multiple" can refer to two or more cases.

[0041] Unless otherwise defined, all technical and scientific terms used in the embodiments of the present application have the same meaning as commonly understood by one of ordinary skill in the art. The terms used in the embodiments of the present application are only for the purpose of describing the embodiments of the present application and are not intended to limit the present application.

[0042] The relevant data collection process in the embodiments of the present application should strictly comply with the requirements of relevant laws and regulations, obtain the informed consent or separate consent of the personal information subject, and within the scope of authorization of laws and regulations and the personal information subject, carry out subsequent data use and processing behavior.

[0043] Before further detailing the embodiments of the present application, the terms and terms involved in the embodiments of the present application are explained, and the terms and terms involved in the embodiments of the present application are applicable to the following explanations.

[0044] 1) In response to: used to represent the conditions or states on which the operations performed depend, when the dependent conditions or states are met, one or more operations performed can be real-time or have a set delay; in the absence of special instructions, there is no restriction on the execution order of multiple operations performed.

[0045] 2) Human-computer interaction interface, an interface for providing human-computer interaction functions / an interface for displaying first information.

[0046] For example, a graphical user interface (GUI) display, such as an augmented reality (AR) interface, a virtual reality (VR) interface, a voice user interface (VUI), an interactive projection interface (using projection technology to display information on a plane), an eye movement detection interface (an interface controlled by detecting the user's line of sight), a holographic interface (a three-dimensional hologram formed by holographic projection technology, without the need to wear special glasses to see a stereoscopic image), a multi-modal interface (an interactive interface that combines multiple interaction methods such as touch, vision, hearing, etc.), a brain-machine interface (BMI) interface, etc.

[0047] 3) Enhanced information, additional information that supplements the semantics of the first image, used to enrich the description dimension of the image or provide background, details, attributes, etc. that are not directly presented by the image itself, to assist the model in more comprehensively understanding the image and generating more accurate description information. The enhanced information includes scene information of the first image and knowledge related to the first image.

[0048] 4) Information extraction example, an example of historical information extraction of a second image belonging to the same category as the first image, including the standard description content corresponding to the second image, used to provide a reference paradigm for information extraction of other images (such as the first image) of the same category, helping the model to learn the description rules and key elements of images of a specific category.

[0049] 5) Position vector, in the text generation or image processing scene, used to represent the numerical representation of the relative position or coordinates of the image blocks in the image or the characters in the text in space, which can assist the model in understanding the spatial structure of the image or the arrangement order of the text, and improve the accuracy of information extraction or generation.

[0050] 6) Cross arrangement, an arrangement mode that alternately organizes character lines of text information and image block lines of images according to a certain rule, wherein the character lines are arranged in the text reading order, the image block lines are arranged in the image spatial order (such as from top to bottom, from left to right), and the image block lines of the same image are continuously arranged, aiming to realize the multi-modal content fusion of text and images, and provide a structured input sequence for the model, and enhance the model's understanding ability of the information associated with text and images.

[0051] 7) Probability distribution, used to describe the possibility distribution of the appearance of a certain word element, semantic unit or output result in the process of model processing tasks, reflecting the confidence of the model on different output options. For example, by adding a plurality of probability distributions, the total probability distribution of the final output result can be determined to assist the model in deciding the optimal solution.

[0052] 8) Prior probability, the initial probability of the occurrence of an event or output result before the model performs a specific task, determined according to historical experience, training data or preset knowledge. Prior probability can be fused with other probability distributions to adjust the rationality of the model output, and reflect the guiding role of prior knowledge on the current task.

[0053] In today's digital age, image information extraction and description are of great significance to many fields, such as image retrieval, intelligent album management, image content analysis and understanding, etc. Related technologies often focus on directly extracting limited and fixed mode information from images themselves or example samples. In the case of images including a large amount of redundant information, the efficiency of image processing is low, and the interference of redundant information leads to low accuracy of information extraction.

[0054] Based on the above analysis, the applicant found that the image processing method of the related technology cannot comprehensively and accurately extract the required information from the image. To solve the above problems, the embodiment of the present application provides an image processing method, which can improve the comprehensiveness and accuracy of information extraction for images.

[0055] The following describes an exemplary application of the electronic device provided by the embodiments of the present application. The electronic device provided by the embodiments of the present application can be implemented as a notebook computer, a tablet computer, a desktop computer, a set-top box, a smart phone, a smart speaker, a smart watch, a smart television, a vehicle-mounted terminal, etc. Various types of terminals, and can also be implemented as a server.

[0056] Referring to Figure 1 , Figure 1 is an architecture schematic diagram of the image processing system 100 provided by the embodiments of the present application. To realize the support of an image processing application, the terminal 400 is connected to the server 200 through the network 300. The network 300 can be a wide area network or a local area network, or a combination of the two.

[0057] The terminal 400 is configured to combine the first image, the information extraction instruction, the enhanced information and the information extraction example into a prompt word, call the first model through the prompt word to generate text, obtain the first information, and display the first information on the human-computer interaction interface 410.

[0058] The server 200 is configured to combine the first image, the information extraction instruction, the enhanced information and the information extraction example into a prompt word, call the first model through the prompt word to generate text, obtain the first information, and send the first information to the terminal 400 to display on the man-machine interaction interface 410.

[0059] Taking a legal scene as an example, the server 200 is configured to combine a legal image, an information extraction instruction, enhanced information and an information extraction example into a prompt word, call the first model through the prompt word to generate text, obtain legal-related information for describing the legal image, and send the legal-related information to the terminal 400 to display on the man-machine interaction interface 410.

[0060] Taking a medical scene as an example, the server 200 is configured to combine a medical image, an information extraction instruction, enhanced information and an information extraction example into a prompt word, call the first model through the prompt word to generate text, obtain medical-related information for describing the medical image, and send the medical-related information to the terminal 400 to display on the man-machine interaction interface 410.

[0061] Taking an education scene as an example, the server 200 is configured to combine a teaching material image, an information extraction instruction, enhanced information and an information extraction example into a prompt word, call the first model through the prompt word to generate text, obtain teaching material-related information for describing the teaching material image, and send the teaching material-related information to the terminal 400 to display on the man-machine interaction interface 410.

[0062] Taking a financial scene as an example, the server 200 is configured to combine a financial report image, an information extraction instruction, enhanced information and an information extraction example into a prompt word, call the first model through the prompt word to generate text, obtain amount-related information for describing the financial report image, and send the amount-related information to the terminal 400 to display on the man-machine interaction interface 410.

[0063] In some embodiments, the server 200 can be a stand-alone physical server, or a server cluster or a distributed system composed of multiple physical servers, or a cloud server providing cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDN), and basic cloud computing services such as big data and artificial intelligence platforms. The terminal and the server can be connected directly or indirectly through wired or wireless communication, which is not limited in the embodiments of the present application.

[0064] Referring to Figure 2 , Figure 2 is a structural schematic diagram of an electronic device 500 provided by an embodiment of the present application, Figure 2The illustrated electronic device 500 includes at least one processor 510, memory 550, at least one network interface 520, and a user interface 530. The various components of electronic device 500 are coupled together by a bus system 540, which is configured to permit communication between the components. The bus system 540 can include a power bus, a control signal bus, and status signal bus, among others. For the sake of clarity, the various buses are illustrated in Figure 5 as the bus system 540. The processor 510 can be implemented as one or more integrated circuit chips designed to process information including program instructions and data. The processor 510 can be a general-purpose processor, a Digital Signal Processor (DSP), or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination thereof designed to process information including program instructions and data. Figure 2 The various buses are illustrated in Figure 5 as the bus system 540 for clarity.

[0065] The processor 510 can be an integrated circuit chip that includes a processor core, a memory, and a programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination thereof designed to process information including program instructions and data. The processor 510 can be a general-purpose processor, a Digital Signal Processor (DSP), or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination thereof designed to process information including program instructions and data.

[0066] The user interface 530 includes one or more output devices 531 that enable presentation of media content, including one or more speakers and / or one or more visual display screens. The user interface 530 also includes one or more input devices 532 that facilitate user input, such as a keyboard, a mouse, a microphone, a touch screen display, a camera, other input buttons and controls.

[0067] The memory 550 can be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state memory, hard drives, optical drives, and the like. The memory 550 optionally includes one or more storage devices remotely located from the processor 510.

[0068] The memory 550 includes volatile memory or nonvolatile memory, or both, and can also include removable or non-removable memory. Nonvolatile memory can be read only memory (ROM), and volatile memory can be random access memory (RAM). The memory 550 described herein is intended to include any suitable type of memory.

[0069] In some embodiments, the memory 550 is capable of storing data to support various operations, examples of which include programs, modules, and data structures or a subset or superset thereof, which are described below.

[0070] The operating system 551 includes system programs for handling various basic system services and for performing hardware dependent tasks, such as a framework layer, a core library layer, a driver layer, and the like, for implementing various basic services and handling hardware-related tasks;

[0071] A network communication module 552 is used to reach other electronic devices via one or more (wired or wireless) network interfaces 520. Exemplary network interfaces 520 include Bluetooth, Wi-Fi, and Universal Serial Bus (USB).

[0072] a presentation module 553 for enabling presentation of information via one or more output devices 531 (e.g., a display screen, a speaker, etc.) associated with the user interface 530 (e.g., a user interface for operating peripheral devices and displaying content and information);

[0073] The input processing module 554 is configured to detect one or more user inputs or interactions from one of the one or more input devices 532 and to translate the detected inputs or interactions.

[0074] In some embodiments, the apparatus provided in the embodiments of the present application may be implemented in software. Figure 2 The image processing device 555 stored in the memory 550 is shown. This device can be software in the form of a program or plug-in, and includes the following software modules: a first determination module 5551, a second determination module 5552, a data combination module 5553, and a text generation module 5554. These modules are logical and can be arbitrarily combined or further separated according to the functions they implement. The functions of each module will be described below.

[0075] In some embodiments, the terminal or server can implement the image processing method provided in the embodiment of the present application by running various computer executable instructions or computer programs. For example, computer executable instructions can be microprogram-level commands, machine instructions or software instructions. The computer program can be a native program or software module in the operating system; it can be a local (Native) application (APPlication, APP), that is, a program that needs to be installed in the operating system to run, such as an intelligent question and answer APP; it can also be a small program that can be embedded in any APP, that is, a program that can be run only by downloading it to a browser environment. In short, the above-mentioned computer executable instructions can be instructions in any form, and the above-mentioned computer program can be an application, module or plug-in in any form.

[0076] The image processing method provided in the embodiment of the present application will be described in conjunction with the exemplary application and implementation of the electronic device provided in the embodiment of the present application.

[0077] The image processing method provided by the embodiments of the present application will be described below. As described above, the electronic device implementing the image processing method of the embodiments of the present application can be a terminal or a server, or a combination of the two. Therefore, the execution subject of each step will not be repeated below.

[0078] Reference is made to Figure 3A , Figure 3A FIG. 1 is a first flowchart of the image processing method provided by the embodiments of the present application, taking an electronic device as the subject, and the steps shown in FIG. 1 will be described in combination. Figure 3A

[0079] In step 101, the enhancement information of the first image is determined, and the enhancement information includes information for enhancing the semantics of the first image.

[0080] In some embodiments, the enhancement information includes the scene information of the first image and the knowledge related to the first image. The basic elements in the first image are identified by computer vision technology (such as object detection, image segmentation), including object categories, colors, shapes, positions, etc.; based on the identified basic elements, the scene information of the first image is determined in combination with a scene recognition model or a knowledge base; according to the extracted scene information and the key objects in the image, the knowledge related to the first image is retrieved through a knowledge graph or a knowledge base; the extracted scene information and the associated related knowledge are integrated to construct complete enhancement information.

[0081] For example, if the first image is a picture depicting a campus scene, the basic elements in the first image are first identified, such as teaching buildings, playgrounds, students, trees, etc. The combination mode, spatial layout and mutual relationship of the objects in the first image are specifically analyzed to determine the scene type, such as determining that the scene of the first image is “school campus” according to the combination of elements such as teaching buildings and playgrounds in the campus scene; at the same time, scene details are further supplemented, such as the time being “daytime” and the season being “summer”, etc. If there is a landmark building in the first image, the history and design features of the building can be queried; if there is a scene of students conducting experiments in the first image, relevant subject knowledge such as the principle and purpose of the experiment can be associated, and these knowledge is summarized and concluded as the related knowledge of the first image. The scene information of “school campus-daytime-summer” is combined with the related knowledge of the history of the landmark building and the subject knowledge of the experiment to form the supplementary description information of the first image, i.e., the enhancement information.

[0082] In step 102, a second image belonging to the same category as the first image is determined, and an information extraction example of the second image is determined.

[0083] ​In some embodiments, the image classification model is called to retrieve a set of images belonging to the same category as the first image from the knowledge base according to the visual features of the first image, such as object category, scene type, color style, etc. From the retrieved set of images of the same category, one or more images most representative are selected as the second image according to preset rules, such as image clarity, information integrity, and similarity to the first image. According to the record of information extraction from the second image, the key information that has been successfully extracted is extracted, such as object name, attribute feature, scene description, etc., and is arranged to form an information extraction example of the second image.

[0084] For example, the first image is a photo of students in a campus doing sports activities on a playground, and the picture has a plastic track, a basketball court, and students running. Through the image classification model, an image set with the theme of "campus learning and living scene" is found in the database, the similarity of the first image to each image in the image set is determined, and the multiple images in the image set are sorted in descending order according to the similarity. The top images of the descending order are selected as the second image according to a preset number or a preset proportion, for example, the photos of students borrowing books in the campus library and doing experiments in the laboratory are selected as the second image. The similarity can be determined by any of the following algorithms: cosine similarity, Euclidean distance, Manhattan distance, and Pearson correlation coefficient. Taking the cosine similarity as an example, the image vector of the first image and the image vector of the second image are determined, the vector product of the image vector of the first image and the image vector of the second image is determined, the length product of the length of the image vector of the first image and the length of the image vector of the second image is determined, and the ratio of the vector product and the length product is determined as the similarity. If the image vector of the first image is m and the image vector of the second image is n, the vector product is m*n, the length of the image vector of the first image is |m|, the length of the image vector of the second image is |n|, the length product is |m|*|n|, and the ratio of the vector product and the length product is (m*n) / (|m|*|n|), which is taken as the similarity. Looking at the historical record, it is found that the information "the library has 100,000 books", "the laboratory is equipped with microscopes and chemical experiment equipment", and "the students are conducting plant cell observation experiments" has been extracted from the second image. These information is arranged as an information extraction example of the second image.

[0085] In step 103, the first image, the information extraction instruction for the first image, the enhanced information, and the information extraction example are combined to obtain the prompt word.

[0086] In some embodiments, the information extraction instruction for the first image is received, which can be input by the user or automatically generated according to the first image input by the user; the enhanced information and the information extraction example are supplemented to the information extraction instruction to obtain the prompt word.

[0087] Taking the example of step 102, if the information extraction instruction is "extract the purpose of the campus facilities, the content of student activities, and the educational significance implied in the scene", the first image is a photo of a campus playground sports class activity, and the enhanced information is "this image shows a school playground scene, and the time is daytime. The playground as an important sports activity site in the campus undertakes functions such as students' daily physical exercise, sports course teaching, and sports meeting holding, which helps to enhance students' physical fitness and cultivate team cooperation spirit", and the information extraction example of the second image is "the library provides students with rich knowledge resources to meet the needs of autonomous learning and research; the laboratory is equipped with microscopes and chemical experiment equipment, and students are conducting plant cell observation experiments, aiming to cultivate students' scientific inquiry ability and practical operation ability", then the combined prompt word is "extract the purpose of the campus facilities, the content of student activities, and the educational significance implied in the scene. The first image is a photo of a campus playground sports class activity. This image shows a school playground scene, and the time is daytime. The playground as an important sports activity site in the campus undertakes functions such as students' daily physical exercise, sports course teaching, and sports meeting holding, which helps to enhance students' physical fitness and cultivate team cooperation spirit. At the same time, the library provides students with rich knowledge resources to meet the needs of autonomous learning and research; the laboratory is equipped with microscopes and chemical experiment equipment, and students are conducting plant cell observation experiments, aiming to cultivate students' scientific inquiry ability and practical operation ability".

[0088] In step 104, the prompt word is input into the first model for text generation to obtain the first information for describing the first image.

[0089] Here, the pre-trained first model can include but is not limited to any one of the following: Large Language Model (LLM), Diffusion Model, Autoregressive Model, and Generative Adversarial Networks (GAN). The first information can be "the campus facility in the image is a playground, its main purpose is to carry out sports course teaching and students' daily exercise, and the plastic track and basketball court provide students with a good sports venue. Students are conducting running activities, which helps to improve students' endurance and physical fitness. This scene implies important educational significance, reflecting the school's emphasis on students' physical health, through sports activities not only to enhance physical fitness, but also to cultivate students' competitive consciousness and team cooperation spirit, promote students' all-round development, and conform to the school's education concept of all-round development of morality, intelligence, body, and beauty".

[0090] In some embodiments, referring to Figure 3B , Figure 3BFIG. 2 is a second flowchart of the image processing method provided by the embodiment of the present application. Figure 3A The step 104 "inputting the prompt word into the first model for text generation to obtain the first information for describing the first image" of the method can be implemented by Figure 3B The steps 1041-1042 of the method can be implemented by the following specific description.

[0091] In the step 1041, the prompt word is encoded by the first model to obtain a prompt word vector.

[0092] In some embodiments, referring to Figure 3C , Figure 3C FIG. 3 is a third flowchart of the image processing method provided by the embodiment of the present application. Figure 3B The step 1041 "encoding the prompt word by the first model to obtain a prompt word vector" of the method can be implemented by Figure 3C The steps 10411-10417 of the method can be implemented by the following specific description.

[0093] In the step 10411, the characters in the text information are encoded to obtain a first text vector, and the text information includes information extraction instructions, augmented information and information extraction examples.

[0094] In some embodiments, the text information is split into multiple characters or words, and then each character or word is mapped to a fixed-dimension dense vector by a pre-trained word embedding model, the vectors include semantic information of the characters or words, and the vectors corresponding to all the characters or words are spliced to obtain the first text vector.

[0095] For example, the word embedding model can be a word vector model (Word to Vector, Word2Vec), a word embedding model based on global word frequency statistics (Global Vectors for word representation, GloVe) or a word embedding layer in a bidirectional encoder representation model (Bidirectional Encoder Representations from Transformers, BERT). Taking the information extraction instruction "extract the purpose of campus facilities, the content of student activities and the educational significance implied in the scene" as an example, the words "extract", "image", "campus", "facility", "purpose" and the like are obtained after tokenization, and "extract" is mapped to the vector [0.2, 0.1, 0.3,...] by the word embedding model, "image" is mapped to the vector [0.4, 0.3, 0.2,...], and so on. All word vectors are combined into a first text vector.

[0096] In the step 10412, the first image is encoded to obtain a first image vector.

[0097] In some embodiments, the first image is processed by an image feature extraction model to extract local features and global features of the first image layer by layer to obtain a vector of fixed dimension, which represents the visual features of the first image as the first image vector.

[0098] For example, the image feature extraction model includes Convolutional Neural Networks (CNN) or Vision Transformer (ViT). Local features include edges, textures, colors, etc. Global features include object categories, scene types, etc. The first image of the physical education class on the campus playground is input into the pre-trained CNN. After operations such as convolutional layers and pooling layers, the CNN extracts feature information such as the plastic track, basketball court, and students running in the first image, and finally outputs a vector including these features, such as [0.15, 0.22, 0.3, ...], as the first image vector.

[0099] In step 10413, the position of the characters in the text information is first encoded to obtain the position vector of the characters.

[0100] In some embodiments, see Figure 3D , Figure 3D This is a fourth flow chart of the image processing method provided in an embodiment of the present application. Figure 3C Step 10413 of "performing a first encoding on the position of the characters in the text information to obtain the position vector of the characters" can be performed by Figure 3D Steps 201 to 202 are implemented as described below.

[0101] In step 201 , characters are arranged into character rows, image blocks are arranged into image block rows, and character rows of text information and image block rows of the first image are arranged alternately.

[0102] In some embodiments, characters are arranged into character rows according to the reading order in the text information, image blocks are arranged into image block rows according to the reading order in the first image, and character rows of the text information and image block rows of the first image are arranged crosswise, wherein image rows belonging to the same first image are arranged continuously.

[0103] In an example, the text information is divided into a plurality of character lines according to a preset rule, and the preset rule can be any one of the following: each character line is of a fixed length; the information extraction instruction, the enhanced information, and the information extraction example each are a character line. For example, in a campus scenario, the text information "extract the use of campus facilities in the image. This is a daytime playground scene, containing a track and a basketball stand" is divided into two character lines, character line 1: extract the use of campus facilities in the image; and character line 2: this is a daytime playground scene, containing a track and a basketball stand. In a case where the number of first images is a plurality, for each first image, the first image (such as a campus playground image) is cut into a plurality of image blocks of the same size according to a size (such as 2*3 pixels), and then each row of continuous image blocks is grouped into an image block row in a top-to-bottom and left-to-right order, for example, the first image is divided into 6 image blocks of 1*1 pixels, and each 3 image blocks arranged side by side form an image block row.

[0104] In step 202, the order of the characters in the result of the cross arrangement is first encoded to obtain a position vector of the characters.

[0105] In some embodiments, referring to Figure 3E , Figure 3E FIG. 5 is a fifth flowchart of a method for processing an image according to an embodiment of the present application. Figure 3D Step 202 of the method 2000, "first encoding the order of the characters in the result of the cross arrangement to obtain a position vector of the characters", for each character, can perform Figure 3E Step 2021 to step 2023 of the method 2000 are implemented, which are specifically described as follows.

[0106] In step 2021, for the first character line in the result of the cross arrangement, the order of the characters in the character line is taken as a horizontal position component and a vertical position component, respectively, and the horizontal position component and the vertical position component are combined into a position vector of the characters.

[0107] In some embodiments, when processing the first character line in the result of the cross arrangement, the order of the characters in the character line is taken as a horizontal position component and a vertical position component, respectively, and the horizontal position component and the vertical position component are combined into a position vector of the characters.

[0108] In an example, for the first character line, the order of the first character in the character line is 1, and then the horizontal position component and the vertical position component are both 1, and the position vector of the first character is (1, 1). The order of the n-th character in the character line is n, n is a positive integer greater than 1 and less than or equal to the number of characters in the character line, and then the horizontal position component and the vertical position component are both n, and the position vector of the n-th character is (n, n).

[0109] Here, for the second and the second and subsequent character lines in the result of the cross arrangement, steps 2022 and 2023 are performed.

[0110] In step 2022, a position vector of a first character of a character line is determined based on a position vector of an image block before the first character of the character line and a size of a first image.

[0111] In some embodiments, a transverse position component of the position vector of the previous image block of the first character is determined by adding a first preset value to a width of the first image, a longitudinal position component of the position vector of the previous image block of the first character is determined by adding a second preset value to a length of the first image, and the position vector of the first character is determined based on the first addition and the second addition.

[0112] For example, the position vector of the previous image block of the first character is (13, 14), the transverse position component of the position vector of the previous image block of the first character is 13, the longitudinal position component of the position vector of the previous image block of the first character is 14, the size of the first image is 2*3, the length of the first image is 2, the width of the first image is 3, the first addition is determined by adding a preset value (e.g., 1) to the transverse position component of the position vector of the previous image block of the first character and the width of the first image, i.e., the first addition is 13+3+1=17, the second addition is determined by adding a preset value (e.g., 1) to the longitudinal position component of the position vector of the previous image block of the first character and the length of the first image, i.e., the second addition is 14+2+1=17, and the position vector of the first character is combined from the first addition and the second addition (17, 17).

[0113] In step 2023, for the second and the second and subsequent characters in a character line, a position vector of a character is determined based on an order of the character in the character line and the position vector of the first character.

[0114] In some embodiments, for the second and the second and subsequent characters in a character line, a difference between an order of a character in the character line and an order of the first character in the character line is determined, a transverse position component of the position vector of the first character is added to the difference to obtain a third addition, a longitudinal position component of the position vector of the first character is added to the difference to obtain a fourth addition, and the position vector of the character is combined from the third addition and the fourth addition.

[0115] For example, taking the second character in the second character line as an example, the order of the character in the character line is 2, the order of the first character in the character line is 1, the difference between the order of the character in the character line and the order of the first character in the character line is 1, the position vector of the first character in the second character line is (28, 28), the horizontal position component of the position vector of the first character is 28, the vertical position component of the position vector of the character is 28, the third sum of the horizontal position component of the position vector of the first character and the difference is 28 + 1 = 29, the fourth sum of the vertical position component of the position vector of the first character and the difference is 28 + 1 = 29, and the position vector of the character is (29, 29).

[0116] When encoding the position of the character, the calculation logic of the horizontal position component and the vertical position component of the position vector follows a fixed pattern, that is, the horizontal position component and the vertical position component are the same, the encoding logic is simple and clear, the model is easy to learn and understand, the understanding efficiency of the text structure is improved, and the explainability of the encoding is enhanced. The character lines and the image block lines are staggered, and through the fixed calculation logic, the positions of the characters during the staggering can be determined according to the fixed distance, the position encoding distance between different modal elements can be stably kept consistent, the model does not need to adapt to different distance rules when processing the position relationship between cross-modal elements, and the relative position between the multi-modal content can be more efficiently and accurately recognized, so that the spatial association during the cross-modal information fusion is more stable and predictable.

[0117] Referring back to Figure 3C In step 10414, the position of the image block in the first image is secondly encoded to obtain a position vector of the image block.

[0118] In some embodiments, the step 10414 "secondly encoding the position of the image block in the first image to obtain a position vector of the image block" performs the following different processing in different cases for each image block:

[0119] Case 1, in the case where the image block is the first image block in the first image block line, the position vector of the image block is determined according to the position vector of the character located before the image block.

[0120] In some embodiments, the position vector of the image block is determined based on the position vector of the character before the image block, and the width and length of the first image.

[0121] For example, a seventh sum of a horizontal coordinate component of the position vector of the image block in the same column and in the previous image block row, a width of the first image, and a preset value (e.g., 1) is determined, and a sixth sum of a vertical coordinate component of the position vector of the image block in the same column and in the previous image block row and a length of the first image, and a preset value (e.g., 1) is determined, and the seventh sum and the vertical coordinate component of the position vector of the image block in the same column and in the previous image block row are combined as the position vector of the image block. For example, the image block is in the first position of the second image block row, the image block in the same column and in the previous image block row is the first image block in the first image block row, the position vector of the image block in the same column and in the previous image block row is (9, 8), the horizontal coordinate component of the position vector of the image block in the same column and in the previous image block row is 9, the vertical coordinate component of the position vector of the image block in the same column and in the previous image block row is 8, the width of the first image is 3, and the seventh sum is 9 + 3 + 1 = 13, and thus the position vector of the image block is (13, 8).

[0122] Case 2, in the case where the image block is the first image block in the second and subsequent image block rows, the position vector of the image block is determined according to the position vectors of the image blocks in the same column.

[0123] In some embodiments, the position vector of the image block is determined based on the position vectors of the image blocks in the same column and the width of the first image.

[0124] For example, a seventh sum of a horizontal coordinate component of the position vector of the image block in the same column and in the previous image block row, a width of the first image, and a preset value (e.g., 1) is determined, and a sixth sum of a vertical coordinate component of the position vector of the image block in the same column and in the previous image block row and a length of the first image, and a preset value (e.g., 1) is determined, and the seventh sum and the vertical coordinate component of the position vector of the image block in the same column and in the previous image block row are combined as the position vector of the image block. For example, the image block is in the first position of the second image block row, the image block in the same column and in the previous image block row is the first image block in the first image block row, the position vector of the image block in the same column and in the previous image block row is (9, 8), the horizontal coordinate component of the position vector of the image block in the same column and in the previous image block row is 9, the vertical coordinate component of the position vector of the image block in the same column and in the previous image block row is 8, the width of the first image is 3, and the seventh sum is 9 + 3 + 1 = 13, and thus the position vector of the image block is (13, 8).

[0125] Case 3, in the case where the image block is the second and subsequent image block in the image block row, the position vector of the image block is determined according to the position vector of the previous image block of the image block.

[0126] In some embodiments, the position vector of the image block is determined based on the position vector of the previous image block and the length of the first image.

[0127] For example, the longitudinal position component of the position vector of the previous image block, the length of the first image, and a preset value (e.g., 1) are added together to obtain an eighth sum, and the eighth sum and the transverse position component of the position vector of the previous image block are combined to obtain the position vector of the image block. For example, the position vector of the previous image block is (9, 8), the transverse position component of the position vector of the previous image block is 9, the longitudinal position component of the position vector of the previous image block is 8, the size of the first image is 2*3, the length of the first image is 2, the eighth sum is 8+2+1=11, and the position vector of the image block is (9, 11).

[0128] The embodiment of the present application calculates the position of the first image block by the position vector of the previous character and the size of the first image, establishes the spatial correlation between the character row and the image block row, avoids the occurrence of position discontinuity in the multi-modal layout, and ensures the boundary connection logic of the model to perceive the character row and the image block row. The position vector of the first image block of the subsequent image block row is derived by the position vector of the image block in the same column and the previous row and the width of the first image, the position consistency of the image block in the longitudinal column is ensured, the model can recognize the semantic correlation of the image blocks in the same column through the position vector, and the structured understanding of the multi-modal content is strengthened. The position vector of the subsequent image block in the row is calculated based on the position vector of the previous image block and the length of the first image, the transverse continuous arrangement of the image blocks in the row is ensured, the model perceives the adjacent relationship between the image blocks through the coordinate difference, and the spatial structure of the image local and the whole is restored. Through the case-based rule design, the layout logic such as the connection of the image and the text, the alignment of the block column, and the continuity of the block row is converted into a standardized position vector calculation process, the position coding of the character and the image block in the unified coordinate system is realized, the learning cost of the model for the complex position relationship is reduced, and the interpretability and robustness of the multi-modal position information are improved.

[0129] Continuing to refer to Figure 3C In step 10415, the first text vector and the position vector of the character are combined into a second text vector.

[0130] In some embodiments, the vector of each character in the first text vector is added or spliced with the position vector of the character, the addition operation is to add the numerical values of the corresponding dimensions, and the splicing operation is to connect two vectors in sequence to obtain a sub-vector corresponding to the character, and the sub-vectors of multiple characters are fused into a second text vector.

[0131] For example, if the first text vector is [0.2, 0.1, 0.3,...], for any character, if the position vector of the character is [0.3, 0.2, 0.1,...], the vector of the character in the first text vector is added to the position vector of the character to obtain a sub-vector corresponding to the character, and the sub-vectors of all characters are fused into a second text vector according to the order in the text information.

[0132] In step 10416, the first image vector and the position vector of the image block are combined into a second image vector.

[0133] In some embodiments, the vector of each image block in the first image vector is added or concatenated with the position vector of the image block to obtain a sub-image vector of the image block, and the sub-image vectors of multiple image blocks are fused to obtain a second image vector.

[0134] For example, the first image vector is [0.15, 0.22, 0.3, ...]. For any image block, if the position vector of the image block is [0.05, 0.08, 0.1, ...], the vector of the image block in the first image vector is added to the position vector of the image block to obtain the sub-image vector of the image block, and the sub-image vectors of all image blocks are fused to obtain the second image vector.

[0135] In step 10417, the second text vector and the second image vector are combined into a prompt word vector.

[0136] In some embodiments, the second text vector and the second image vector are concatenated or fused through a specific fusion layer (such as a fully connected layer) to form a unified vector including all information of the image and text, namely, a prompt word vector, which serves as input for a subsequent decoding process.

[0137] For example, the second text vector is [0.5, 0.3, 0.4, ...], the second image vector is [0.2, 0.3, 0.4, ...], and the concatenation results in [0.5, 0.3, 0.4, ..., 0.2, 0.3, 0.4, ...] as the prompt word vector.

[0138] The embodiment of the present application encodes text semantics, image vision, text order and image spatial position respectively to ensure that the prompt word vector completely covers the visual content, information extraction instructions, enhanced knowledge and information extraction examples of the first image, avoiding the understanding deviation caused by the lack of single-dimensional information. The text vector and the image vector are deeply integrated to construct a multimodal unified representation, so that the model can accurately capture the semantic relationship between the image and the text. By encoding the position of the characters in the text information and the position of the image blocks in the first image, the model's perception of the sequence order and spatial structure is improved.

[0139] Continue to see Figure 3B In step 1042, the prompt word vector is decoded to obtain first information for describing the first image.

[0140] Here, the prompt word vector can be decoded in any of the following ways: Bayesian decoding, Maximal Spanning Tree Decoding, Greedy Decoding, Beam Search Decoding, Transformer decoder, Recurrent Neural Network (RNN) decoder.

[0141] In some embodiments, referring to Figure 3F , Figure 3F is a sixth flowchart of the image processing method provided in the embodiments of the present application. Figure 3B Step 1042 "decoding the prompt word vector to obtain first information for describing the first image" of the first model can be implemented by Figure 3F Steps 10421-10427 of the first model are implemented, as described below.

[0142] In step 10421, the first probability distribution of the first token to be output by the first model is determined when the input of the first model is the image vector of the first image.

[0143] Here, the image vector of the first image is input into the first model, the first model extracts the image features of the first image through the visual encoder, and then maps them to the vocabulary space through the linear layer to calculate the probability of each first token to form the first probability distribution.

[0144] In step 10422, the second probability distribution of the second token to be output by the first model is determined when the input of the first model is the information extraction instruction vector of the information extraction instruction.

[0145] Here, the instruction vector of the information extraction instruction is determined, the instruction vector is input into the first model, the semantic features of the instruction vector are extracted through the text encoder, and the probability of each second token is determined to form the second probability distribution.

[0146] In step 10423, the third probability distribution of the third token to be output by the first model is determined when the input of the first model is the enhanced information vector of the enhanced information.

[0147] Here, the enhanced information vector of the enhanced information is determined, the enhanced information vector is input into the first model, the semantic features of the enhanced information are extracted through the text encoder, and the probability of each third token is determined to form the third probability distribution, which focuses on the semantic features of the scene knowledge and the world knowledge.

[0148] In step 10424, in a case where the input of the first model is the information extraction example vector of the information extraction example, a fourth probability distribution of a fourth word piece to be output by the first model is determined.

[0149] Here, the information extraction example vector of the information extraction example is determined, the information extraction example vector is input into the first model, the semantic features of the information extraction example vector are extracted through the text encoder, the probability of each fourth word piece is determined based on the structured information of the historical example to form the fourth probability distribution, and the first model is guided to reuse the extraction mode of the history.

[0150] In step 10425, a total probability distribution is determined based on the first probability distribution, the second probability distribution, the third probability distribution, and the fourth probability distribution.

[0151] In some embodiments, referring to Figure 3G , Figure 3G is a seventh flowchart of the image processing method provided in the embodiments of the present application. Figure 3F The step 10425 of determining the total probability distribution based on the first probability distribution, the second probability distribution, the third probability distribution, and the fourth probability distribution can be implemented by Figure 3G The steps 301 to 303 of the method 300 can be implemented by

[0152] In step 301, the probability distribution sum of the first probability distribution, the second probability distribution, the third probability distribution, and the fourth probability distribution is determined.

[0153] In some embodiments, for each first word piece, the sum of the first probability distribution of the first word piece, the second probability distribution of the second word piece, the third probability distribution of the third word piece, and the fourth probability distribution of the fourth word piece is determined as the probability distribution sum. In order to avoid numerical overflow, the logarithm of the first probability distribution, the logarithm of the second probability distribution, the logarithm of the third probability distribution, and the logarithm of the fourth probability distribution can be determined respectively, the logarithm sum of the plurality of logarithms is determined, and the logarithm sum is normalized to obtain the probability distribution sum.

[0154] For example, the logarithm of the first probability distribution is 0.5, the logarithm of the second probability distribution is 0.3, the logarithm of the third probability distribution is 0.4, and the logarithm of the fourth probability distribution is 0.2. The logarithm sum is 1.4, and the logarithm sum is normalized to obtain the probability distribution sum 0.55.

[0155] In step 302, the prior probability of the first model for the information extraction instruction is determined.

[0156] In some embodiments, the first model is pre-determined with a prior probability of certain word pieces according to semantics of the information extraction instruction itself, and the prior probability is controlled by an adjustable intensity parameter to determine how much the prior accounts for in the total probability.

[0157] For example, assuming that the base probability of the word piece "use" is 0.6 in the prior of the information extraction instruction "extract the use of campus facilities", if the adjustment parameter is set to "medium intensity", the prior will increase the probability of the word piece "use"; if it is set to "weak intensity", the influence of the prior will be reduced.

[0158] In step 303, the probability distribution is added and fused with the prior probability to obtain a total probability distribution.

[0159] In some embodiments, the probability distribution is weighted and added with the prior probability to obtain a total probability distribution.

[0160] For example, the first weight corresponding to the probability distribution addition 0.55 is 0.8, and the second weight corresponding to the prior probability 0.3 is 0.2, so the total probability distribution is 0.55*0.8+0.3*0.2=0.5.

[0161] The embodiments of the present application fuse and sum the probability distributions corresponding to the first image, the information extraction instruction, the enhanced information, and the information extraction example, integrate the advantages of information of different modalities, mutually supplement the multi-dimensional information such as the visual details of the image, the task orientation of the instruction, the background knowledge of the enhanced information, and the structural template of the example, realize the deep cooperation of multi-modal information, improve the comprehensiveness and accuracy of the probability prediction of the word piece, and guarantee that the generated content can cover rich and closely related information. The introduction of the prior probability of the information extraction instruction gives the specific word piece a natural tendency according to the semantics of the instruction, and through the preset weight, the generation direction of the first model can be constrained to ensure that the generated first information is closely related to the task requirements, avoid task deviation caused by single modal interference or model spontaneous tendency, and make the output first information fit the target of the information extraction instruction.

[0162] Continuing to refer to Figure 3F In step 10426, a fifth word piece is selected from the first word piece, the second word piece, the third word piece, and the fourth word piece according to the total probability distribution.

[0163] In some embodiments, the fifth word piece is selected from the first word piece, the second word piece, the third word piece, and the fourth word piece according to the total probability distribution by greedy decoding (selecting the word piece with the highest probability), beam search (retaining the first K word pieces), or temperature sampling (introducing randomness).

[0164] For example, the first word item, the second word item, the third word item and the fourth word item are sorted in descending order according to the total probability distribution, to obtain a descending order result; and a preset number or a preset proportion of word items are selected from the top of the descending order result as the fifth word item. Alternatively, the word item corresponding to the highest total probability distribution is taken as the fifth word item.

[0165] In step 10427, the fifth word item is combined to obtain the first information for describing the first image.

[0166] In some embodiments, after each fifth word item is generated, the generated fifth word item is added to the word item sequence as a condition for generating the next fifth word item, and the next fifth word item is predicted based on the current word item sequence and the total probability distribution of the next word item. By using the attention mechanism of the model, the newly generated fifth word item is semantically associated with the fifth word item in the word item sequence, and the first information for describing the first image is obtained by combining the information extraction instruction and the sentence structure constraint.

[0167] For example, the first image includes a scene of students running in a campus playground, the information extraction instruction is "extract the purpose of campus facilities and the educational significance", the enhanced information is "the playground is the core site of physical education, promoting all-round development", and the information extraction example is "the library is used for knowledge learning, and the laboratory is used for practical operation". The word item sequence is initially empty, the fifth word item "racecourse" is selected according to the total probability distribution, the word item sequence is updated to [racecourse], the next fifth word item "for" is selected according to the total probability distribution, the word item sequence is updated to [racecourse, for], and so on. The final word item sequence is [racecourse, for, student, daily, run, train, help, physique, improve], and the first information can be "the playground is the core site of physical education in the campus, which is used for students' daily running training and physical education practice, helps to improve physique and cultivate team cooperation ability, and embodies the educational goal of all-round development".

[0168] The embodiment of the present application provides visual clues through image vectors, and supplements semantic depth through text vectors, so as to avoid missing abstract concepts by a pure visual model or misinterpreting image details by a pure text model. The information extraction instruction ensures that the generated content is closely related to the instruction, the historical example provides a structured template, and the normativeness of the generated result is improved. By adjusting the weights of each modality, different tasks can be adapted.

[0169] In some embodiments, referring to Figure 3H , Figure 3H is the eighth flowchart of the image processing method provided by the embodiment of the present application. After step 104 "inputting the prompt word into the first model to generate text and obtaining the first information for describing the first image", steps 105 to 109 are executed. Figure 3H are specifically described below.

[0170] In step 105, the first information is subjected to format detection, and a format detection result is obtained.

[0171] In some embodiments, it is checked whether the first information conforms to a preset format. The preset format can include at least one of the following: sentence structure, element integrity, punctuation, and line specification.

[0172] For example, if the first information is "The playground is used for students to run and improve their physical fitness", and the preset format requires that the "educational significance" element must be included, then the format detection result of the first information is that it does not conform to the preset format. If the first information is "The playground is the core sports venue on campus, used for students' daily running training, helping to improve physical fitness and teamwork, and embodying the goal of comprehensive development education", then the format of the first information conforms to the preset format.

[0173] In step 106, in response to the format detection result indicating that the format of the first information conforms to the preset format, the information extraction instruction and the first image are input into a second model for text generation to obtain second information for describing the first image.

[0174] In some embodiments, when the format detection result of the first information indicates that the format of the first information conforms to the preset format, the same information extraction instruction and the first image are used to call a pre-trained second model, the second model being a different model from the first model, and the second model can also be used for text generation to obtain the second information for describing the first image.

[0175] For example, the information extraction instruction is "extract the purpose of campus facilities and educational significance", the first information generated by the first model is "The playground is a sports education venue used for running training to promote student health", and the second information generated by the second model is "The campus playground is a core space for sports courses, carrying out daily exercise and team competition for students, helping to improve physical fitness and teamwork, and implementing the concept of comprehensive development".

[0176] In step 107, the first information and the second information are subjected to illusion detection, and an illusion detection result is obtained.

[0177] In some embodiments, by comparing the first information and the second information, it is checked whether there is illusion content. The detection logic includes: fact consistency, if the first information does not conform to the fact, then the illusion detection result indicates that the first information has illusion; instruction fit, if the first information does not conform to the information extraction instruction, then the illusion detection result indicates that the first information has illusion; and knowledge compatibility, if the first information contradicts common sense, then the illusion detection result indicates that the first information has illusion.

[0178] For example, if the first information is "playground for student astronomical observation" and the second information is "playground for sports courses", the hallucination detection result indicates that the first information has hallucination; if the first information is "playground for running training", the hallucination detection result indicates that the first information does not have hallucination relative to the second information.

[0179] In step 108, in response to the hallucination detection result indicating that the first information does not have hallucination relative to the second information, the quality parameter of the first information is determined based on the pre-trained third model.

[0180] In some embodiments, when the hallucination detection result indicates that the first information does not have hallucination relative to the second information, a third model (e.g., a reward model) is called to determine the quality parameter of the first information based on multiple quality dimensions of the first information.

[0181] For example, the multiple quality dimensions include at least one of relevance, completeness, logic, and conciseness. If the first information is "playground, running", the completeness of the first information is poor, and the quality parameter of the first information is low; if the first information is "playground is the core site of campus sports education, used for student daily running training and sports course practice, helping to improve physical fitness and team cooperation ability, and embodying the education goal of overall development", the first information meets the requirements of multiple quality dimensions, and the quality parameter of the first information is high.

[0182] In step 109, third information with a quality parameter higher than a quality parameter threshold is selected from the first information, and the third information and the first image corresponding to the third information are written into a knowledge base, and the knowledge base includes the second image and the information extraction example of the second image.

[0183] In some embodiments, a quality parameter threshold is set, and third information with a quality parameter higher than the quality parameter threshold is selected from the first information, and the third information and the first image corresponding to the third information are written into a knowledge base, and the knowledge base is used to store high-quality images and description information corresponding to the images for subsequent model training or query.

[0184] For example, there are multiple first information, which are first information A, first information B and first information C, if the quality parameter of the first information A is 0.8, the quality parameter of the first information B is 0.6, the quality parameter of the first information C is 0.9, and the quality parameter threshold is 0.85, the first information C is selected as the third information, and the first information C and the first image corresponding to the first information C are added to the knowledge base.

[0185] The embodiment of the application controls the compliance of the first information through format detection, ensures that the first information meets the preset format requirements, performs hallucination detection through a second model, eliminates the first information that contradicts facts or instructions, ensures the authenticity of the description, determines the quality parameter of the first information through a third model, filters high-quality information according to the quality parameter threshold, and improves the generation quality of subsequent information. The third information and the corresponding first image are stored in the knowledge base, high-quality data is provided for subsequent model training, complex image-text understanding tasks are supported, and the accuracy of text generation of the model is improved. Multiple models cooperatively detect and process the first information, objective scores are used to replace manual screening, the reliability of the generation result is improved by means of multi-model balance.

[0186] In the following, an exemplary application of the embodiment of the application in an intelligent question answering application scenario will be described.

[0187] In a retrieval-augmented generation (RAG) generation scenario involving pictures, the redundant information included in the pictures is often greater than the text, and it is more likely that the picture content does not match the expected output due to the input picture including a large amount of noise. For example, in the scenario of extracting the amount of case goods from the price determination provided by an institution, the price determination usually includes multiple pictures, such as “price determination matter description”, “determination basis”, “determination process and method”, “detail table”, “determination conclusion”, etc. However, in the RAG process, the required goods price table can be directly obtained from the “detail table” or “determination conclusion” picture, and taking the entire price determination book as the output will result in the output including too much redundant information. The repeated amount of money will make the model have the illusion of multiple determination results, or other descriptions about the amount of goods in the entire price determination book will easily affect the efficiency and accuracy of the model in performing the task. In addition, how to extract the required result from the price table requires the model to understand the instructions, which includes business knowledge of the specific scenario. How to automatically filter and reduce the noise in the input through a filtering enhancement method is the key to improving the accuracy of the multi-modal generation task.

[0188] The related art includes the following ways of filtering and enhancing input: rule method, which is a filtering and enhancing method of manually setting screening and strengthening standards, converts unstructured information such as images into text through optical character recognition (OCR) technology, extracts key-value pairs from the text, screens the key-value pairs according to pre-set structured data extraction matching criteria, removes contents that do not meet the requirements, and strengthens key information that meets the standards, to filter out noise and enhance effective information; image classification model or enhancement model, which uses a convolutional neural network or a Transformer model to accumulate data in a specific scenario, and after training, can filter or enhance through image classification or clarity supplement; vector similarity, which calculates the similarity between the vector of the example sample and the vector of the input this time based on the historical accumulation of example samples in the database, and filters or supplements samples based on the similarity threshold.

[0189] The related art lacks automation and generalization ability, such as manually configuring rules, training image classification models or enhancement models, which often takes a long time and lacks generalization ability in new task scenarios or inputs that have not been involved, requiring retraining and configuration.

[0190] The embodiments of the present application strengthen and improve the underlying position perception reasoning ability of the multi-modal large model, enhance the association between images and text, and take the position of the image in the original input as a hidden perception to enhance filtering and denoising during reasoning and strengthen the generation ability. An overall model framework is proposed, which can better integrate information between multi-modal inputs based on the world knowledge learned by pre-training the text model, and provide better background knowledge for understanding images and completing downstream tasks. Through iteration and maintenance of the knowledge base and the extraction samples and details corresponding to the examples, the generalization ability and adaptability of the entire framework in different tasks are improved, and through feedback-based iterative maintenance, the subsequent manual maintenance cost is reduced.

[0191] For the proposed overall model framework, scene understanding is performed through the framework, for example, dangerous driving traffic cases, evidence sorting, and report generation scenarios. The input pictures are usually blurred, and the input pictures are usually scanned or handwritten and of various types. The evidence pictures usually include driver's license scans, responsibility identification papers, loss appraisal opinions, agreement papers, and insurance copies. These pictures are usually not classified, but directly provided to the model, and the task instructions are usually detailed sub-tasks, such as extracting the insurance effective end time to determine whether it is within the insurance period, judging the loss caused by the accident, etc. These sub-tasks only require part of the pictures, and when a large number of pictures are input, the input data will exceed the maximum context length of the model, and there will be interference items and noise, causing the model to hallucinate.

[0192] Referring to Figure 4 , Figure 4 is a flowchart of a multi-modal reasoning process provided by an embodiment of the present application. First, a number of input pictures and specific task instructions described in text form are received. Second, a classification service is called based on the input pictures. The classification service allocates a knowledge base corresponding to the category and a prompt word example corresponding to the category according to the category of the input pictures, where the prompt word example is a structured data, for example, a JavaScript Object Notation (Json), which is a prompt word for providing scene knowledge generated in advance by a language large model. For example, information needs to be extracted from a complaint picture. Even if different complaints include different features and have different clarity, through a Json, the model is guided to extract the date, name, and case handling unit of the complaint picture, which can reduce the possibility of model hallucination and help better perform subsequent tasks. If the template guide is not added, the model only extracts the page number, signature, seal, or other blurred features due to resolution, which is easy to produce hallucinations. For example, in the scene of price determination, the knowledge base is all pictures of price determination, and in the scene of dangerous driving cases, the knowledge base is all pictures related to the case and evidence. In actual business, whether multiple knowledge bases are needed is determined according to the scene and the content covered by the task. The classification service is used to reduce the number of candidates recalled later. According to the logic of multiple knowledge bases, such as keywords, pre-allocated categories, small models trained, etc. The knowledge base includes pictures and corresponding prompt word examples for each picture, such as price determination, insurance policy, and medical appraisal report. There are many types of picture styles, and most scenes will have blurred and difficult-to-identify handwritten situations. If the model is directly asked to extract information from the picture, the model cannot identify the key points in the picture. Since the picture is blurred, the model may hallucinate. Sometimes there is a gap between the task instruction and the picture input, for example, the picture does not contain the content that needs to be extracted, which causes hallucination and instruction compliance errors. The prompt word template corresponding to the picture is a more general extraction template. For example, for a blurred and handwritten picture, if the type of the picture is an insurance policy, the focus is on the effective end time, object, etc. of the picture. If the type of the picture is a price determination, the focus is on the unit price and the name of the goods, etc.

[0193] Then, the text large model is called based on the input task instruction, so that the text large model provides scene understanding and world knowledge (text enhancement content, i.e., enhancement information) by using the knowledge learned through pre-training, for example, helping to predict key elements in the image, providing background information, and arranging structured forms. The text enhancement content refers to the knowledge described in the text paragraph, which is generated by the large model through a prompt word, for example, describing what a certain type of book evidence usually is, what important information it includes, and what it is usually used for in the current scene.

[0194] Next, the multi-modal reasoning model is called based on the task instruction, and a picture with a similar theme (i.e., a second image) is matched from the knowledge base by calculating the similarity, to serve as a standard for filtering noise from the input picture. The extraction template example corresponding to the picture with a similar theme is used as a supplementary input to enhance the instruction following ability, wherein the task instruction and the input picture can both be used as inputs for calculating the similarity with the example picture (i.e., the second image) in the knowledge base. Specifically, the text and the picture are encoded simultaneously by the multi-modal model, and the similarity with the example picture is calculated through the encoded feature vectors.

[0195] Again, in the multi-modal reasoning large model service, the task instruction, the text enhancement content, the input picture, and the example template of the similar picture are input together, and the reinforced distance-aware position encoding and the multi-modal Bayesian decoding are used to complete the reasoning task.

[0196] Finally, after the reasoning task is completed, the task quality is determined through the post-hallucination inspection modification of the text large model and the trained reward model. The post-inspection is to preliminarily determine whether the content generated by the model (i.e., the first information) conforms to the format agreement (i.e., the preset format) through format analysis (i.e., format detection), such as Json, Markdown. The failure of format alignment often means a higher probability of hallucination. By comparing the instructions and results through the text large model (i.e., the second model), it is determined whether hallucination occurs. The text large model is often a large model of different sources from the multi-modal model (i.e., the first model) that performs reasoning. The reward model (i.e., the third model) is usually a public process reward model (PRM) or an object relational mapping (ORM) model. In the reward model, the input text (i.e., the first information) directly returns the score. By controlling the threshold, the high-quality output picture without hallucination (i.e., the first image) is used as an example picture, and the answer (i.e., the third information) is used as its example template, which is updated to the knowledge base. That is, the picture (i.e., the first image) and the corresponding answer (i.e., the third information) are summarized and supplemented to the knowledge base. In the future, similar pictures can be recalled and used as examples. For example, in the price identification paper, the name of the identification agency, the unit price, the type of goods, the identification time, etc. are extracted. Such output structure can guide the model to focus on these information and reduce the influence of irrelevant noise interference.

[0197] The embodiments of the present application can help to supplement the knowledge that the user and the model have not introduced in advance in specific cases. Through the pre-maintained knowledge base, the instruction following ability of the model is improved, and the user experience is improved. After completing the task, the task quality is determined and the knowledge base is updated. In the case of dynamic changes in online environment, the generalization ability is improved.

[0198] For the process of Bayesian decoding, the noise introduced increases due to the introduction of a large amount of background content of different modalities (world knowledge, scene knowledge, picture examples, extraction templates, etc.), which affects the generation quality of the model. The training objective of the large model training stage is to predict the probability distribution P(t+1|T) of the next token t+1 for a given token T(0:t). The training data is a large amount of text collected from the Internet and the probability distribution of these texts T. Since the collected texts are far away from the input T(S1 (world knowledge), S2 (scene template), S3 (picture example)) simulated by the current task, the generation quality P(t+1|S1, S2, S3) simulated by the task will also be affected. If these simulated texts are split, the probability distribution P(t+1|S1), P(t+1|S2) corresponding to different subtexts can improve the learning ability of the model in the training stage. In the inference stage of the multi-modal large model, the input T is disassembled into S1, S2, S3 (input of different sources and different modalities), the probability P(t|Sn=1, 2, 3…) is calculated, and then the decoding operation is performed in the decoding process through the Bayesian formula (see formula 1 below).

[0199]

[0200] wherein, log p(T|A1, A2, …, A n ) is the total probability distribution of the next word, β is a preset weight, logsumexp is a summation operation, which is the sum of the probabilities after taking the exponential and then taking the logarithm, top is selected from the top of the sorting result, and p(T|S i) is the probability distribution corresponding to different subtexts, that is, the probability distribution after adding certain additional content, log is the logarithmic operation, log p(T) is the probability distribution obtained by directly generating the original problem with the large model, which is actually the probability judgment given by the large model to each word among all candidate words, constant is a constant. By predicting all possible next words and giving each word a score, the higher the score, the more likely the word will be selected. The model calculates the entropy value of each possible next word to find out which option has a more concentrated "probability distribution". If the model has previously selected a word, it will slightly reduce the entropy value of the related options around it (through eta = 0.5). For example, the model is more likely to continue generating content around "apple" rather than suddenly jumping to unrelated words. From all the words, select the top 3 words with the lowest entropy value, such as "apple", "banana", and "orange". Combine the scores of the top 3 selected words into a comprehensive score, similar to "voting" or "weighted average". Because the word table is large when the large model generates, it may cause a computer floating point overflow problem in probability calculation. By using the logsumexp operation, the probabilities are summed by exponentiation and then logarithm is taken to avoid numerical overflow. At the same time, multiply the comprehensive score by the entropy of each sampled probability distribution (the entropy of the probability distribution is the certainty of the probability distribution) to give a greater weight to the generation path of more certain words. The parameter β (set to 0 here) is used to balance the weights of the two scores, and finally a comprehensive score is obtained. Among them, log p(T) is the prior knowledge of the model for the problem, that is, the answer given by the model when the input is not brought into any context. In the RAG task, if you want to avoid the prior bias of the model itself for this task, you can set it to 0. In other tasks, such as long text question answering, or scenarios where you believe the model itself can handle better, you can increase β.

[0201] Due to the involvement of multi-modal input, the model has increased difficulty in perceiving different modal data. Since the attention mechanism used by the large model is a parallel operation, its perception ability for input positions completely depends on the design of position encoding. See Figure 5A , Figure 5A is a schematic diagram of related technologies for position encoding. Taking position encoding of multi-modal literary works as an example, the related technology large model position encoding mainly sets all text to (x, x), and the input dimension is increased by 1. Such encoding may cause unclear boundaries between different modalities, especially when inputting multi-modal content and the order is random, which will affect the model's judgment.

[0202] When different modal boundaries appear, the position coding mode designed in the embodiments of the present application calculates the size of each dimension of the picture. Assuming that the length and width of the picture are a and b, the position coding of the picture starts with x+=b+1 and y+=a+1, and the position coding is subsequently calculated through x+=b+1 and y+=a+1. This is a solution that ensures that all text tokens fall on (x, y) and x=y under the condition that each picture is equal before and after the text is separated. The positions of the text are all coded as (x, y) and x=y, and are sequentially added by 1 from left to right. When the picture appears, the positions where the picture and the text meet are calculated as x+=b+1 and y+=a+1. The second and subsequent pictures in each row are coded as x+=b+1 and y+=a+1. The positions where the text and the picture are converted are also coded starting from x+=b+1 and y+=a+1. If the text is converted from the picture again, the text is continuously coded as (x, y) and x=y. The pictures are separated by a special token img.

[0203] Referring to Figure 5B , Figure 5B is a schematic diagram of position coding of multi-modal information provided by the embodiments of the present application. The first row is text, and the position coding is performed as (x, y) and x=y. The position vectors of different texts are sequentially added by 1 from left to right. The second and third rows are a plurality of image blocks obtained by dividing a same image with a pixel of 2*3. At this time, a=2 and b=3. Since the text is converted into the picture from the first row to the second row, x=x+b+1=5+3+1=9 and y=y+a+1=5+2+1=8. That is, the coding of the first image of the second row is (9, 8). The horizontal coordinates of the images in the same row are unchanged, and the vertical coordinates are separated by a+1. For example, the vertical coordinates between (9, 8) and (9, 11) are separated by a+1=2+1=3. The second and third rows are divided from the same image, the vertical coordinates of the images in the same column are unchanged, and the horizontal coordinates are separated by b+1. For example, the vertical coordinates between (9, 8) and (13, 8) are separated by b+1=3+1=4. The token is text, and the third row is converted from the image to the text, x=x+b+1=13+3+1=17 and y=y+a+1=14+2+1=17. The fourth row is a picture with a pixel of 1*4. At this time, a=1 and b=4. When the third row is converted to the fourth row, x=x+b+1=18+4+1=23 and y=y+a+1=18+1+1=20. That is, the position coding of the first image block of the fourth row is (23, 20). The position coding in this manner can keep the position coding distance consistent when different modal boundaries appear.

[0204] The embodiment of the present application combines the world knowledge of the language large model and the scene understanding to help the multi-modal large model better understand the picture, close to common sense knowledge, extract and answer the framework of specific questions, introduce the world knowledge obtained in the pre-training stage of the language large model, which can help improve the performance of the subsequent model in the case of input picture noise, user instruction is not clear and the like. Through task classification and template recall based on picture similarity, compared with multi-modal recall based on text similarity, the picture processing template and logic provided in this way can better help the subsequent task, and at the same time, through the post-test and output example, the link of the knowledge base can be updated, the knowledge base can be learned and updated online, and the generalization ability of the framework can be increased. Through the decoding process based on the Bayes probability formula, the generation reasoning logic of multi-path input source splicing and merging can be easily simulated, the problem of insufficient training of the model in the training stage can be solved, and at the same time, the influence of noise in the input can be automatically reduced. By introducing new position encoding, the input division between different orders of multiple modalities is more obvious, and the model is more sensitive. In the scene of long input and multiple modal content switching frequently, the reasoning ability of the model can be improved.

[0205] The following continues to illustrate an exemplary structure of the image processing apparatus 555 provided by the embodiment of the present application as a software module. In some embodiments, as shown in Figure 2 The software module stored in the image processing apparatus 555 of the memory 550 can include:

[0206] The first determination module 5551 is configured to determine the enhancement information of the first image, and the enhancement information includes information for enhancing the semantics of the first image.

[0207] The second determination module 5552 is configured to determine the second image belonging to the same category as the first image, and determine the information extraction example of the second image.

[0208] The data combination module 5553 is configured to combine the first image, the information extraction instruction for the first image, the enhancement information and the information extraction example to obtain the prompt word.

[0209] The text generation module 5554 is configured to input the prompt word into the first model to generate text and obtain the first information for describing the first image.

[0210] In some embodiments, the text generation module 5554 is further configured to encode the prompt word through the first model to obtain a prompt word vector, and decode the prompt word vector to obtain the first information for describing the first image.

[0211] In some embodiments, the text generation module 5554 is further configured to encode the characters in the text information to obtain a first text vector, the text information comprising the information extraction instruction, the augmented information, and the information extraction example; encode the first image to obtain a first image vector; first encode the positions of the characters in the text information to obtain a character position vector; second encode the positions of the image blocks in the first image to obtain an image block position vector; combine the first text vector and the character position vector into a second text vector; combine the first image vector and the image block position vector into a second image vector; and combine the second text vector and the second image vector into the prompt word vector.

[0212] In some embodiments, the text generation module 5554 is further configured to arrange the characters into character lines, arrange the image blocks into image block lines, and cross-arrange the character lines of the text information and the image block lines of the first image; first encode the order of the characters in the cross-arranged result to obtain the character position vector.

[0213] In some embodiments, the text generation module 5554 is further configured to, for a first character line in the cross-arranged result, take the order of the characters in the character line as a horizontal position component and a vertical position component respectively, and combine the horizontal position component and the vertical position component into the character position vector; and for a second and subsequent character line in the cross-arranged result, determine the position vector of a first character in the character line based on the position vector of the image block before the first character and the size of the first image, and determine the position vector of a character in the character line based on the order of the character in the character line and the position vector of the first character.

[0214] In some embodiments, the text generation module 5554 is further configured to, in a case where the image block is a first image block in a first image block line, determine the position vector of the image block according to the position vector of the character before the image block; in a case where the image block is a first image block in a second and subsequent image block line, determine the position vector of the image block according to the position vector of the image block in the same column as the image block; and in a case where the image block is a second and subsequent image block in an image block line, determine the position vector of the image block according to the position vector of the previous image block of the image block.

[0215] In some embodiments, the text generation module 5554 is further configured to determine the position vector of the image block based on the position vector of the character before the image block and the width and length of the first image; determine the position vector of the image block based on the position vector of the image block in the same column as the image block and the width of the first image; and determine the position vector of the image block based on the position vector of the previous image block and the length of the first image.

[0216] In some embodiments, the text generation module 5554 is further configured to, in a case where the input of the first model is an image vector of the first image, determine a first probability distribution of a first word piece to be output by the first model; in a case where the input of the first model is an information extraction instruction vector of the information extraction instruction, determine a second probability distribution of a second word piece to be output by the first model; in a case where the input of the first model is an enhanced information vector of the enhanced information, determine a third probability distribution of a third word piece to be output by the first model; in a case where the input of the first model is an information extraction example vector of the information extraction example, determine a fourth probability distribution of a fourth word piece to be output by the first model; determine a total probability distribution based on the first probability distribution, the second probability distribution, the third probability distribution, and the fourth probability distribution; select a fifth word piece from the first word piece, the second word piece, the third word piece, and the fourth word piece according to the total probability distribution; and combine the fifth word piece into the first information for describing the first image.

[0217] In some embodiments, the text generation module 5554 is further configured to determine a probability distribution sum of the first probability distribution, the second probability distribution, the third probability distribution, and the fourth probability distribution; determine a prior probability of the first model for the information extraction instruction; and fuse the probability distribution sum and the prior probability to obtain the total probability distribution.

[0218] In some embodiments, the text generation module 5554 is further configured to perform format detection on the first information to obtain a format detection result; in response to the format detection result representing that a format of the first information conforms to a preset format, input the information extraction instruction and the first image into a second model to perform text generation, and obtain second information for describing the first image; perform hallucination detection on the first information and the second information to obtain a hallucination detection result; in response to the hallucination detection result representing that the first information does not have hallucination relative to the second information, determine a quality parameter of the first information based on a pre-trained third model; select third information with a quality parameter higher than a quality parameter threshold from the first information, and write the third information and a first image corresponding to the third information into a knowledge base, the knowledge base including a second image and an information extraction example of the second image.

[0219] The embodiment of the present application provides a computer program product, which comprises a computer program or computer executable instructions stored in a computer readable storage medium. A processor of an electronic device reads the computer executable instructions from the computer readable storage medium, and the processor executes the computer executable instructions, so that the electronic device executes the image processing method provided in the embodiment of the present application.

[0220] The embodiment of the present application provides a computer readable storage medium storing computer executable instructions, wherein the computer executable instructions or computer programs are stored, and when the computer executable instructions or computer programs are executed by a processor, the processor executes the image processing method provided by the embodiment of the present application, for example, as shown in the following. Figure 3A The image processing method is shown.

[0221] In some embodiments, the computer readable storage medium can be RAM, ROM, flash memory, magnetic surface memory, optical disc, or CD-ROM memory, etc.; and can also be various devices including one or any combination of the above storage.

[0222] In some embodiments, the computer executable instructions can be in the form of programs, software, software modules, scripts or codes, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and can be deployed in any form, including being deployed as independent programs or being deployed as modules, components, subroutines or other units suitable for use in a computing environment.

[0223] As an example, the computer executable instructions can but not necessarily correspond to files in a file system, can be stored in a part of a file storing other programs or data, for example, stored in one or more scripts in a Hyper Text Markup Language (HTML) document, stored in a single file dedicated to the program in question, or stored in multiple cooperative files (for example, files storing one or more modules, subroutines or code parts).

[0224] As an example, the computer executable instructions can be deployed to be executed on one electronic device, or executed on multiple electronic devices located in one place, or executed on multiple electronic devices distributed in multiple places and interconnected through a communication network.

[0225] To sum up, by the embodiments of the present application, the first image, the information extraction instruction for the first image, the enhancement information of the first image, and the information extraction example of the second image belonging to the same category as the first image are combined into a prompt word, and the pre-trained first model is called based on the prompt word to generate text to obtain the first information for describing the first image. By introducing the enhancement information, the information on the semantic level that is not directly embodied in the first image itself can be supplemented, and the first model can understand the image content more deeply to generate more comprehensive and accurate image descriptions. Through the information extraction example of the second image of the same category, the experience of information extraction of the second image of the same category can be learned, so that the first model can quickly locate the key information when processing the first image, reduce the invalid information processing process, and improve the efficiency and accuracy of information extraction. Combining information of multiple dimensions into a prompt word provides complete and targeted input for the first model in the process of text generation, and improves the comprehensiveness and accuracy of the first model in generating the first information.

[0226] The above merely describes the embodiments of the present application, but is not intended to limit the protection scope of the present application. Any modification, equivalent replacement, and improvement within the spirit and scope of the present application shall be included in the protection scope of the present application.

Claims

1. A method for processing an image, characterized in that: The method comprises: determining enhancement information of the first image, the enhancement information comprising information for enhancing semantics of the first image; determining a second image that belongs to the same category as the first image, and determining an information extraction example for the second image; combining the first image, the information extraction instruction for the first image, the enhanced information, and the information extraction example to obtain a prompt word; The prompt word is input into a first model to generate text, thereby obtaining first information for describing the first image.

2. The method according to claim 1, characterized in that The step of inputting the prompt word into the first model to generate text to obtain first information for describing the first image includes: Encoding the prompt word through the first model to obtain a prompt word vector; The prompt word vector is decoded to obtain first information for describing the first image.

3. The method according to claim 2, characterized in that The encoding of the prompt word by the first model to obtain a prompt word vector includes: Encoding characters in text information to obtain a first text vector, wherein the text information includes the information extraction instruction, the enhanced information, and the information extraction example; Encoding the first image to obtain a first image vector; Performing a first encoding on the position of the character in the text information to obtain a position vector of the character; performing a second encoding on the position of the image block in the first image to obtain a position vector of the image block; Combining the first text vector and the position vector of the character into a second text vector; combining the first image vector and the position vector of the image block into a second image vector; The second text vector and the second image vector are combined into a prompt word vector.

4. The method according to claim 3, characterized in that The first encoding of the position of the character in the text information to obtain the position vector of the character includes: Arranging the characters into character rows, arranging the image blocks into image block rows, and arranging the character rows of the text information and the image block rows of the first image alternately; Performing a first encoding on the order of the characters in the result of the cross arrangement to obtain a position vector of the characters; The performing a second encoding on the position of the image block in the first image to obtain the position vector of the image block includes: In a case where the image block is the first image block in the first image block row, determining the position vector of the image block according to the position vector of the character located before the image block; When the image block is the first image block in the second or subsequent image block rows, determining the position vector of the image block based on position vectors of image blocks in the same column as the image block; When the image block is the second or subsequent image block in the image block row, the position vector of the image block is determined based on the position vector of the previous image block.

5. The method according to claim 4, characterized in that The first encoding of the order of the characters in the result of the cross arrangement to obtain the position vector of the characters includes: For the first character row in the result of the cross arrangement, taking the order of the characters in the character row as a horizontal position component and a vertical position component, respectively, and combining the horizontal position component and the vertical position component to form a position vector of the character; For the second and subsequent character rows in the cross-arrangement result, the following processing is performed: determining a position vector of the first character based on a position vector of an image block preceding the first character of the character row and a size of the first image; For the second and subsequent characters in the character row, position vectors of the characters are determined based on the order of the characters in the character row and the position vector of the first character.

6. The method according to claim 4, characterized in that The determining the position vector of the image block according to the position vector of the character located before the image block includes: determining a position vector of the image block based on a position vector of the character before the image block and a width and a length of the first image; The determining the position vector of the image block according to the position vector of the image block in the same column as the image block includes: determining a position vector of the image block based on a position vector of the image block in the same column as the image block and a width of the first image; The determining the position vector of the image block according to the position vector of a previous image block of the image block includes: The position vector of the image block is determined based on the position vector of the previous image block and the length of the first image.

7. The method according to claim 2, characterized in that The decoding of the prompt word vector to obtain first information for describing the first image includes: determining a first probability distribution of a first word-gram to be output by the first model when the input of the first model is the image vector of the first image; determining a second probability distribution of a second word-gram to be outputted by the first model when the input of the first model is the information extraction instruction vector of the information extraction instruction; determining a third probability distribution of a third word-gram to be output by the first model when the input of the first model is the enhanced information vector of the enhanced information; determining a fourth probability distribution of a fourth word-gram to be output by the first model when the input of the first model is the information extraction example vector of the information extraction example; determining an overall probability distribution based on the first probability distribution, the second probability distribution, the third probability distribution, and the fourth probability distribution; selecting a fifth word-gram from the first word-gram, the second word-gram, the third word-gram, and the fourth word-gram according to the total probability distribution; The fifth word-gram is combined into first information for describing the first image.

8. The method according to claim 7, characterized in that The determining of a total probability distribution based on the first probability distribution, the second probability distribution, the third probability distribution, and the fourth probability distribution includes: Determining a probability distribution sum of the first probability distribution, the second probability distribution, the third probability distribution, and the fourth probability distribution; determining a priori probability of the first model for the information extraction instruction; The sum of the probability distributions is combined with the prior probability to obtain a total probability distribution.

9. The method according to any one of claims 1 to 8, characterized in that After inputting the prompt word into the first model to generate text and obtaining first information for describing the first image, the method further includes: Performing format detection on the first information to obtain a format detection result; In response to the format detection result indicating that the format of the first information conforms to a preset format, inputting the information extraction instruction and the first image into a second model for text generation to obtain second information for describing the first image; performing hallucination detection on the first information and the second information to obtain a hallucination detection result; In response to the hallucination detection result indicating that the first information is not hallucinated relative to the second information, determining a quality parameter of the first information based on a pre-trained third model; Selecting third information whose quality parameter is higher than a quality parameter threshold from the first information, writing the third information and the first image corresponding to the third information into a knowledge base, wherein the knowledge base includes the second image and the information extraction example of the second image.

10. An electronic device, characterized in that: The electronic device comprises: a memory for storing computer-executable instructions or computer programs; A processor, configured to implement the image processing method according to any one of claims 1 to 9 when executing the computer executable instructions or computer program stored in the memory.

11. A computer program product comprising computer executable instructions or a computer program, characterized in that When the computer executable instructions or computer program are executed by a processor, the image processing method according to any one of claims 1 to 9 is implemented.

Images