Information extraction method, model training method and information extraction system
By using a multimodal information extraction model to extract information from images and employing a reflective verification mechanism to verify and adjust the initial information, the problem of insufficient accuracy of traditional models is solved, and high-precision information extraction results are achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-11-28
- Publication Date
- 2026-03-10
AI Technical Summary
Traditional information extraction models have errors in image information extraction, resulting in poor accuracy and making it difficult to meet the needs of high-precision application scenarios.
A multimodal information extraction model is used to extract information from the image to be processed, and the initial information is verified and adjusted through a reflection and verification mechanism to ensure the accuracy of the extraction results.
It improves the accuracy and interpretability of information extraction, and optimizes the initial information through a self-reflection mechanism to output more realistic target information.
Smart Images
Figure CN121640201A_ABST
Abstract
Description
Technical Field
[0001] This specification relates to one or more embodiments in the field of computer technology, and in particular to an information extraction method, a model training method, and an information extraction system. Background Technology
[0002] With the rapid development of the digital age, image data is growing explosively. Image information extraction technology, as the core means of processing image data, has important application value in many fields such as intelligent security, medical diagnosis, and autonomous driving. By efficiently extracting text, target, and semantic information contained in images, it can realize the structured analysis of image content and provide data support for subsequent decision-making and applications. Its technological progress is of great significance to promoting the intelligent development of various industries.
[0003] However, traditional information extraction models usually output extraction results directly based on the feature information of the image. However, the information extracted in this way often contains errors, resulting in poor accuracy. This makes it difficult to meet the needs of high-precision application scenarios and limits the widespread application and further development of image information extraction technology. Summary of the Invention
[0004] In view of the above, one or more embodiments of this specification provide the following technical solutions: According to a first aspect of one or more embodiments of this specification, an information extraction method is proposed, comprising: The image to be processed and information extraction prompts are input into a pre-trained multimodal information extraction model to instruct the multimodal information extraction model to: extract information from the image to be processed based on the information extraction prompts to obtain initial information; and reflect and verify the initial information according to the target features corresponding to the image to be processed to verify the matching between the initial information and the image to be processed, and process the initial information into corresponding target information based on the reflection and verification results. Obtain the target information output by the multimodal information extraction model.
[0005] According to a second aspect of one or more embodiments of this specification, a model training method is proposed, comprising: The sample image and sample information extraction prompts are input into the multimodal information extraction model to be trained. The information extraction model extracts information from the sample image based on the sample information extraction prompts to obtain initial information. The initial information is then subjected to reflection verification based on the target features corresponding to the sample image to verify the matching between the initial information and the sample image. Based on the reflection verification results, the initial information is processed into corresponding output information. The model parameters of the multimodal information extraction model are adjusted based on the deviation between the actual label corresponding to the sample image and the output information.
[0006] According to a third aspect of one or more embodiments of this specification, an information extraction system is proposed, the information extraction system comprising: an encoding module, a feature fusion module, and a decoding module; The encoding module is used to determine the image features of the image to be processed and the text features of the information extraction prompts input into the multimodal information extraction model; The feature fusion module is used to fuse the image features and the text features of the information extraction prompts to obtain fused features; The decoding module is used to decode the fused features to extract information from the image to be processed to obtain initial information, and to perform reflection verification on the initial information based on the target features of the image to be processed to verify the matching between the initial information and the image to be processed, and to process the initial information into corresponding target information based on the reflection verification result.
[0007] According to a fourth aspect of one or more embodiments of this specification, an information extraction apparatus is provided, comprising: An extraction unit is used to input the image to be processed and information extraction prompts into a pre-trained multimodal information extraction model, so as to instruct the multimodal information extraction model to: extract information from the image to be processed based on the information extraction prompts to obtain initial information, and to reflect and verify the initial information according to the target features corresponding to the image to be processed, so as to verify the matching between the initial information and the image to be processed, and process the initial information into corresponding target information based on the reflection and verification results; The acquisition unit is used to acquire the target information output by the multimodal information extraction model.
[0008] According to a fifth aspect of one or more embodiments of this specification, a model training apparatus is provided, comprising: The input unit is used to input sample images and sample information extraction prompts into the multimodal information extraction model to be trained, so that the information extraction model can extract information from the sample images based on the sample information extraction prompts to obtain initial information, and perform reflection verification on the initial information according to the target features corresponding to the sample images to verify the matching between the initial information and the sample images, and process the initial information into corresponding output information based on the reflection verification results; The adjustment unit is used to adjust the model parameters of the multimodal information extraction model based on the deviation between the actual label corresponding to the sample image and the output information.
[0009] According to a sixth aspect of one or more embodiments of this specification, an electronic device is provided, comprising: a processor; a memory for storing processor-executable instructions; wherein the processor executes the executable instructions to implement the steps of the information extraction method or model training method described above.
[0010] According to a seventh aspect of one or more embodiments of this specification, a computer-readable storage medium is provided that stores computer instructions thereon, which, when executed by a processor, implement the steps of the information extraction method or model training method described above.
[0011] According to an eighth aspect of one or more embodiments of this specification, a computer program product is provided, comprising a computer program / instructions that, when executed by a processor, implement the steps of the information extraction method or model training method described above.
[0012] As can be seen from the above embodiments, this specification instructs the multimodal information extraction model to: extract information from the image to be processed based on the information extraction prompts to obtain initial information; and reflect and verify the initial information according to the target features corresponding to the image to be processed to verify the matching between the initial information and the image to be processed, and process the initial information into the corresponding target information based on the reflection and verification results; and obtain the target information output by the multimodal information extraction model.
[0013] In this method, the information extraction model does not directly output the extraction result during the information extraction process of the image to be processed. Instead, after extracting the initial information, it uses the target features corresponding to the image to reflect on and verify the initial information. In this way, if the verification result shows that the initial information does not meet the requirements, it can be further adjusted to obtain more accurate target information. Compared with existing methods that directly output the extraction result, this method makes full use of the model's self-reflection mechanism to further optimize the initial information, thereby fully ensuring the accuracy of information extraction. Attached Figure Description
[0014] Figure 1 This is a flowchart illustrating an information extraction method provided in an exemplary embodiment; Figure 2 This is a schematic diagram of the overall process of information extraction provided in an exemplary embodiment; Figure 3 This is a schematic flowchart of a model training method provided in an exemplary embodiment; Figure 4This is a schematic diagram of an information extraction system provided in an exemplary embodiment; Figure 5 This is a schematic diagram of the structure of a device provided in an exemplary embodiment; Figure 6 This is a schematic diagram of an information extraction device provided in an exemplary embodiment; Figure 7 This is a schematic diagram of a model training device provided in an exemplary embodiment. Detailed Implementation
[0015] To make the objectives, technical solutions, and advantages of this specification clearer, the technical solutions of this specification will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this specification, and not all of them. Based on the embodiments in this specification, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this specification.
[0016] With the rapid iteration and updates of large-scale models, their importance in various fields is increasing daily. In the field of image information extraction, early small models, represented by Donut, relied on large amounts of high-quality training data and were prone to overfitting. In recent years, multimodal large-scale models have emerged, significantly improving the model's ability to perceive images by fusing image and text information. However, traditional multimodal large-scale models mostly rely on surface content recognition in images and lack verification and review mechanisms for the results, resulting in room for improvement in the accuracy of information extraction in complex scenarios.
[0017] Based on this, this specification provides an information extraction method that fully utilizes the self-reflection mechanism of a multimodal information extraction model, allowing the model to reflect on and verify the initial extraction results, thereby improving the accuracy of information extraction. At the same time, the reflection and error correction process can be used for interpretation, improving the interpretability of the information.
[0018] The technical solutions provided in the various embodiments of this specification are described in detail below with reference to the accompanying drawings.
[0019] Figure 1 This is a flowchart illustrating an information extraction method provided in an exemplary embodiment, including: S100: Input the image to be processed and the information extraction prompt words into a pre-trained multimodal information extraction model to instruct the multimodal information extraction model to: extract information from the image to be processed based on the information extraction prompt words to obtain initial information; and reflect and verify the initial information according to the target features corresponding to the image to be processed to verify the matching between the initial information and the image to be processed, and process the initial information into corresponding target information based on the reflection and verification results.
[0020] S102: Obtain the target information output by the multimodal information extraction model.
[0021] In this specification, the execution subject for performing an information extraction method can be a designated device such as a server, or it can be a terminal device such as a mobile phone, tablet computer, laptop computer, or desktop computer, or a client installed on these terminal devices. For ease of understanding, the following will use a server as the execution subject to explain the information extraction method provided in this specification.
[0022] The server can acquire images to be processed. These images can be rich text images from which information needs to be extracted, such as scanned contracts, screenshots of meeting presentation slides, product posters, and charts from academic papers. The information to be extracted can include key data from contract terms, core viewpoints from presentation slides, promotional information from posters, and statistical conclusions from charts. It's important to note that this information can include not only the original text in the image but also information obtained after recognizing and processing the text and image content. For example, extracting a summary of the rights and obligations of both parties from a scanned contract, extracting applicable rules for promotional activities from a product poster, and analyzing data from academic charts to generate trend summaries.
[0023] In addition, the server can pre-construct information extraction prompts for the multimodal information extraction model. These prompts indicate the content that the model needs to extract. The prompts may also include verification enable flags or verification hints.
[0024] Among them: the verification enable flag is used to indicate that the multimodal information extraction model has enabled reflection verification. This verification enable flag can be represented by a special token, such as "<verify_start> Custom symbols or strings such as “[validation_enable]” and “CHECK_ON” can be used to initiate the reflexive validation process when the multimodal information extraction model recognizes the validation enable flag.
[0025] The verification prompt message contains text content instructing the multimodal information extraction model on how to perform reflective verification. The multimodal information extraction model can then execute the reflective verification process based on this text content. For example, this text content could be: "Check whether there are any contradictions between the extracted text information and the visual elements in the image." It should be noted that the text example above can be the text content instructing the multimodal information extraction model on how to perform reflective verification. In addition to this text content, the input information extraction prompt can also include text content instructing the model to extract initial information, such as "Extract text information related to xxx from the image and summarize it." This yields the complete prompt message: "Extract text information related to xxx from the image and summarize it, and check whether there are any contradictions between the extracted text information and the visual elements in the image."
[0026] Of course, the above prompts may not include a verification enable flag or verification prompts. Instead, the model's network structure or parameters may be configured to enable reflective verification functionality, such as by executing reflective verification logic through specific network layers or modules.
[0027] In the actual information extraction process, the server can input the image to be processed and information extraction prompts into a pre-trained multimodal information extraction model to instruct the multimodal information extraction model to extract information from the image to be processed based on the information extraction prompts to obtain initial information.
[0028] The multimodal information extraction model can then further reflect on and verify the initial information to check the matching between the initial information and the image to be processed, and process the initial information into the corresponding target information based on the reflection and verification results, and then output the target information.
[0029] Specifically, the multimodal information extraction model includes a visual encoder, a text encoder, a transformer layer, and a decoder.
[0030] After the image to be processed is input into the multimodal information extraction model, key visual information in the image can be identified and extracted through a visual encoder, thereby obtaining image features.
[0031] In this specification, the visual encoder can extract features from the image to be processed based on a dynamic encoding vector, where the dynamic encoding vector is used to determine the length of the image feature encoding vector. Within a preset resolution range, the resolution of the image to be processed is positively correlated with the encoding vector length: when the image resolution is high, it means that more detailed information needs to be extracted. In this case, by adaptively increasing the encoding vector length, the richness of image feature representation can be improved. In addition, to avoid the encoding vector length from growing indefinitely, a resolution range threshold can be set in advance. When the image resolution is within this range, the encoding vector length increases with the resolution until it reaches the fixed length upper limit corresponding to the maximum resolution. This effectively solves the problem of wasted computational resources and model performance degradation caused by the explosion of encoding dimensions.
[0032] The resolution range can be set according to the performance and storage space of the device deploying the multimodal information extraction model. For example, the higher the performance of the device and the larger the storage space, the higher the upper limit of the resolution range. In addition, the resolution range can also be set according to the business type or business requirements of the current business. For example, the upper limit of the resolution range is lower for businesses with high efficiency requirements and higher for businesses with high accuracy requirements. Of course, it can also be set based on experience. This manual does not specify the method of determination.
[0033] Meanwhile, the multimodal information extraction model can use a text encoder to map the text of information extraction prompts to an ID in a vocabulary and then vectorize and encode these IDs to obtain the text features of the information extraction prompts.
[0034] Then, text features and image features can be input into the transformer layer, so that the transformer layer can fuse the text features and image features through the mutual learning of multiple perceptual layers, thereby obtaining fused features.
[0035] The fused features output from the transformer layer are further input into the decoder, which can decode the fused features to obtain the initial information.
[0036] After determining the initial information, the decoder can further determine the text features contained in the initial information, and, based on the positional information of the initial information in the image to be processed, determine the local features corresponding to the initial information in the target features.
[0037] In this specification, the target information mentioned above can be image features or fused features determined based on image features and text features of information extraction prompts. The server can determine the target features corresponding to the image at the position corresponding to the extracted initial information in the image to be processed, and then compare the text features contained in the initial information with the target features to perform feature verification on the initial information, thereby verifying the matching between the initial information and the image to be processed.
[0038] In practical applications, the server can also determine the target features based on the feature representation of the location corresponding to the extracted initial information in the image to be processed, as well as the context feature representation adjacent to that location. Of course, the server can also use the feature representation of the entire image to be processed as the target feature; here, the feature representation can refer to image features or fused features.
[0039] In this specification, the matching between the initial information and the image to be processed can be determined based on the deviation between the two. That is, when the deviation between the initial information and the image to be processed is greater than a preset deviation, it indicates that the matching degree between the two is low, and the content of the initial information may be unreasonable. Therefore, the decoder can adjust the initial information and use the adjusted information as the target information. The preset deviation can be set according to the actual situation, and this specification does not impose specific limitations on it.
[0040] When the deviation between the initial information and the image to be processed is less than the preset deviation, it indicates that the matching degree between the two is high, and the initial information can be directly output as the target information.
[0041] It should be added that, in addition to ambiguity between the content in the initial information and the image to be processed, the deviation between the initial information and the image to be processed being greater than the preset deviation can also include other situations such as the initial information containing content that does not exist at the corresponding position in the image to be processed, and the content at the corresponding position in the image to be processed not being reflected in the initial information.
[0042] Furthermore, during the process of the decoder adjusting the initial information, the adjustment objective can be to minimize the deviation between the text features contained in the initial information and the target features. The text features contained in the initial information can be adjusted, and the adjusted information can be determined based on the adjusted text features.
[0043] For example, the decoder can determine a loss value based on the deviation between text features and target features, and then determine the adjustment direction for adjusting the text features contained in the initial information based on this loss value, and further adjust the information based on this adjustment direction. Of course, in practical applications, the decoder can also skip the text feature adjustment environment and directly adjust the initial information based on the deviation between the two.
[0044] Furthermore, in practical applications, the decoder can adjust the initial information multiple times until the deviation between the text features contained in the adjusted information and the target features is less than a preset threshold, at which point the final target information is output. For ease of understanding, this specification provides a schematic diagram of the overall information extraction process, as follows: Figure 2 As shown.
[0045] Figure 2 This is a schematic diagram of the overall process of information extraction provided in an exemplary embodiment.
[0046] The image to be processed may include multiple images, and the CoT inference content of the multimodal information extraction model includes the following steps: Initial information: The model first obtains an initial response based on the information to be extracted through its first perception; Reflection and verification: Use the information in the image (target features) to reflect on and verify the initial information to determine whether the initial information needs to be corrected; Content correction: If the initial information needs to be corrected based on the results of the reflection and verification, the initial information shall be adjusted. Target information: Output the target information obtained after correction.
[0047] The server can then obtain the target information extracted by the multimodal information extraction model and use this target information to perform subsequent business operations such as archiving key contract terms in the business system, automatically generating meeting minutes from PPT core viewpoints, synchronizing poster promotional information to the marketing and promotion platform, and using academic charts and conclusions to assist in the statistical analysis of paper data.
[0048] It should be added that, in addition to outputting target information, the multimodal information extraction model can also output the reasoning process of the target information (including the reasoning process of initial information as well as the process of reflection, verification and correction). This reasoning process can be displayed in the form of a thought chain, thereby improving the interpretability of the target information.
[0049] Furthermore, before using the aforementioned multimodal information extraction model, it needs to be trained. Therefore, this specification provides a model training method, such as... Figure 3 As shown.
[0050] Figure 3 This is a flowchart illustrating a model training method provided in an exemplary embodiment, including the following steps: S300: Input the sample image and sample information extraction prompts into the multimodal information extraction model to be trained, so that the information extraction model can extract information from the sample image based on the sample information extraction prompts to obtain initial information, and perform reflection verification on the initial information according to the target features corresponding to the sample image to verify the matching between the initial information and the sample image, and process the initial information into corresponding output information based on the reflection verification result; S302: Adjust the model parameters of the multimodal information extraction model based on the deviation between the actual label corresponding to the sample image and the output information.
[0051] The visual encoder extracts image features from sample images, while the text encoder extracts text features from information extraction prompts. Target features include image features or fused features determined based on image and text features. The visual encoder extracts image features from sample images based on dynamic encoding vectors. The dynamic encoding vector determines the length of the encoding vector for the image features; within a preset resolution range, the resolution of the sample image is positively correlated with the length of the encoding vector.
[0052] During the reflection and verification process, the multimodal information extraction model can determine the text features contained in the initial information, and, based on the location information corresponding to the initial information in the sample image, determine the local features corresponding to the initial information in the target features; then, based on the deviation between the text features and local features contained in the initial information, the initial information is reflected and verified.
[0053] Furthermore, this specification also provides an information extraction system, such as... Figure 4 As shown.
[0054] Figure 4 This is a schematic diagram of an information extraction system provided in an exemplary embodiment.
[0055] The information extraction system includes an encoding module, a feature fusion module, and a decoding module. The encoding module is used to determine the image features of the image to be processed and the text features of the information extraction prompts for the input multimodal information extraction model; The feature fusion module is used to fuse image features and text features of information extraction prompts to obtain fused features; The decoding module is used to decode the fused features to extract information from the image to be processed and obtain initial information, and to perform reflection verification on the initial information based on the target features of the image to be processed to verify the matching between the initial information and the image to be processed, and to process the initial information into the corresponding target information based on the reflection verification results.
[0056] Furthermore, the above encoding modules include: a visual encoding module and a text encoding module; the visual encoding module is used to: extract image features from the image to be processed; the text encoding module is used to: extract text features from the image to be processed; the target features include image features or fused features determined based on image features and text features.
[0057] The visual encoding module can extract features from the image to be processed based on the dynamic encoding vector to obtain image features. The dynamic encoding vector is used to determine the length of the encoding vector for the image features. Within a preset resolution range, the resolution of the image to be processed is positively correlated with the length of the encoding vector.
[0058] The decoding module can adjust the text features contained in the initial information with the goal of minimizing the deviation between the text features contained in the initial information and the target features, and determine the adjusted information based on the adjusted text features.
[0059] As can be seen from the above, in this scheme, the information extraction model does not directly output results when processing the image to be processed. Its workflow is as follows: first, initial information is extracted; then, the target features corresponding to the image to be processed are called to reflectively verify the initial information. Once the verification finds that the initial information has deviations or does not meet the accuracy requirements, the initial information is iteratively adjusted based on the verification feedback, ultimately outputting more accurate target information.
[0060] This process leverages the model's built-in self-reflection mechanism to refine and optimize the initial extraction results, ensuring the accuracy of information extraction through process design and making the output target information more aligned with actual needs.
[0061] Figure 5 This is a schematic structural diagram of a device provided in an exemplary embodiment. Please refer to... Figure 5 At the hardware level, the device includes a processor, internal bus, network interface, memory, and non-volatile memory, and may also include other hardware required for its functions. One or more embodiments of this specification can be implemented in software, such as by the processor reading the corresponding computer program from non-volatile memory into memory and then running it. Of course, besides software implementation, one or more embodiments of this specification do not exclude other implementation methods, such as logic devices or a combination of hardware and software, etc. That is to say, the execution entity of the following processing flow is not limited to individual logic units, but can also be hardware or logic devices.
[0062] Figure 6 This is a schematic diagram of an information extraction device provided in an exemplary embodiment.
[0063] Please refer to Figure 6 Information extraction devices can be applied to, for example Figure 5 The device shown is used to implement the technical solution described in this specification. The information extraction device may include: The extraction unit 600 is used to input the image to be processed and information extraction prompts into a pre-trained multimodal information extraction model, so as to instruct the multimodal information extraction model to: extract information from the image to be processed based on the information extraction prompts to obtain initial information, and to reflect and verify the initial information according to the target features corresponding to the image to be processed, so as to verify the matching between the initial information and the image to be processed, and process the initial information into corresponding target information based on the reflection and verification results; The acquisition unit 602 is used to acquire the target information output by the multimodal information extraction model.
[0064] Optionally, the information extraction prompt includes a verification activation identifier or verification prompt information; wherein: the verification activation identifier is used to instruct the multimodal information extraction model to activate reflective verification; the verification prompt information includes text content that prompts the multimodal information extraction model on how to perform reflective verification.
[0065] Optionally, the multimodal information extraction model includes a visual encoder and a text encoder; the visual encoder is used to extract image features from the image to be processed, and the text encoder is used to extract text features of the information extraction prompts; the target features include the image features or fused features determined based on the image features and the text features of the information extraction prompts.
[0066] Optionally, the visual encoder is used to extract features from the image to be processed based on a dynamic encoding vector to obtain the image features; wherein, the dynamic encoding vector is used to determine the length of the encoding vector of the image features, and within a preset resolution range, the resolution of the image to be processed is positively correlated with the length of the encoding vector.
[0067] Optionally, the extraction unit 600 is specifically used to: determine the text features contained in the initial information; and, based on the position information corresponding to the initial information in the image to be processed, determine the local features corresponding to the initial information in the target features; and perform reflection verification on the initial information based on the deviation between the text features contained in the initial information and the local features.
[0068] Optionally, the extraction unit 600 is specifically used to determine the deviation between the text features contained in the initial information and the target features; if the deviation is greater than a preset deviation, the initial information is adjusted and the adjusted information is used as the target information; otherwise, the initial information is determined as the target information.
[0069] Optionally, the extraction unit 600 is specifically used to adjust the text features contained in the initial information with the goal of minimizing the deviation between the text features contained in the initial information and the target features, and to determine the adjusted information based on the adjusted text features.
[0070] Figure 7 This is a schematic diagram of a model training device provided in an exemplary embodiment.
[0071] Please refer to Figure 7 Model training devices can be applied to, for example Figure 5 The device shown is used to implement the technical solution of this specification. The model training device may include: The input unit 700 is used to input a sample image and sample information extraction prompts into a multimodal information extraction model to be trained, so that the information extraction model can extract information from the sample image based on the sample information extraction prompts to obtain initial information, and perform reflection verification on the initial information according to the target features corresponding to the sample image to verify the matching between the initial information and the sample image, and process the initial information into corresponding output information based on the reflection verification result; The adjustment unit 702 is used to adjust the model parameters of the multimodal information extraction model according to the deviation between the actual label corresponding to the sample image and the output information.
[0072] Optionally, the multimodal information extraction model includes a visual encoder and a text encoder; the visual encoder is used to extract image features from the sample image, and the text encoder is used to extract text features of the information extraction prompt; the target features include the image features or fused features determined based on the image features and the text features of the information extraction prompt.
[0073] Optionally, the visual encoder is used to extract features from the sample image based on a dynamic encoding vector to obtain the image features; wherein, the dynamic encoding vector is used to determine the length of the encoding vector of the image features, and within a preset resolution range, the size of the sample image resolution is positively correlated with the length of the encoding vector.
[0074] Optionally, the input unit is specifically used to: determine the text features contained in the initial information; and, based on the position information corresponding to the initial information in the sample image, determine the local features corresponding to the initial information in the target features; and perform reflection verification on the initial information based on the deviation between the text features contained in the initial information and the local features.
[0075] Based on the same concept as the methods described above, this specification also provides a computer-readable storage medium having computer instructions stored thereon that, when executed by a processor, implement the steps of the methods as described in any of the above embodiments.
[0076] Based on the same concept as the methods described above, this specification also provides a computer program product, including a computer program / instructions that, when executed by a processor, implement the steps of the methods as described in any of the above embodiments.
Claims
1. An information extraction method, comprising: inputting a to-be-processed image and an information extraction prompt into a pre-trained multi-modal information extraction model to instruct the multi-modal information extraction model to: perform information extraction on the to-be-processed image based on the information extraction prompt to obtain initial information, and perform reflection verification on the initial information according to a target feature corresponding to the to-be-processed image to verify a matching condition between the initial information and the to-be-processed image, and process the initial information into corresponding target information based on a reflection verification result. obtaining the target information output by the multi-modal information extraction model.
2. The method of claim 1, wherein the information extraction prompt contains a verification start identifier or verification prompt information; and wherein: the verification start identifier is used to instruct the multi-modal information extraction model to start the reflection verification; and the verification prompt information contains text content for prompting the multi-modal information extraction model how to perform the reflection verification.
3. The method of claim 1, wherein the multi-modal information extraction model is provided with a visual encoder and a text encoder; and wherein: the visual encoder is used to extract image features from the to-be-processed image, and the text encoder is used to extract text features of the information extraction prompt; and the target feature includes the image features or fusion features determined based on the image features and the text features of the information extraction prompt. 4.The method of claim 3, wherein the visual encoder is configured to extract features of the image to be processed based on a dynamic encoding vector to obtain the image features. The dynamic coding vector is used to determine the length of the coding vector of the image features, and in a preset resolution range, the size of the to-be-processed image resolution is positively correlated with the length of the coding vector.
5. The method of claim 1, wherein the reflection verification on the initial information according to the target feature corresponding to the to-be-processed image specifically includes: determining text features contained in the initial information, and determining local features corresponding to the initial information in the target feature based on position information of the initial information in the to-be-processed image; and performing reflection verification on the initial information according to a deviation between the text features contained in the initial information and the local features.
6. The method of claim 1, wherein processing the initial information into corresponding target information based on the reflection verification result specifically includes: determining a deviation between the text features contained in the initial information and the target feature; and if the deviation is greater than a preset deviation, adjusting the initial information, and taking the adjusted information as the target information, otherwise, determining the initial information as the target information.
7. The method of claim 6, wherein processing the initial information into corresponding target information based on the reflection verification result specifically includes: minimizing the deviation between the text features contained in the initial information and the target feature as an adjustment target, adjusting the text features contained in the initial information, and determining the adjusted information based on the adjusted text features.
8. A model training method, comprising: inputting the sample image and sample information extraction prompt word into the to-be-trained multi-modal information extraction model, so as to perform information extraction on the sample image based on the sample information extraction prompt word through the information extraction model to obtain initial information, and performing reflection verification on the initial information according to a target feature corresponding to the sample image, so as to verify a matching condition between the initial information and the sample image, and processing the initial information into corresponding output information based on a reflection verification result; adjusting model parameters of the multi-modal information extraction model according to a deviation between an actual label corresponding to the sample image and the output information.
9. The model training method of claim 8, wherein the multi-modal information extraction model is provided with a visual encoder and a text encoder; the visual encoder is configured to extract image features from the sample image, and the text encoder is configured to extract text features of the information extraction prompt word; the target feature includes the image features or fusion features determined based on the image features and the text features of the information extraction prompt word. 10.The method of claim 9, wherein the visual encoder is configured to extract features of the sample image based on a dynamic encoding vector to obtain the image features; and wherein, The dynamic encoding vector is used to determine the length of the encoding vector of the image features, and in a preset resolution range, the size of the sample image resolution is positively correlated with the length of the encoding vector.
11. The method of claim 8, wherein the reflection verification on the initial information according to the target feature corresponding to the sample image specifically comprises: determining text features contained in the initial information, and determining local features corresponding to the initial information in the target feature based on position information of the initial information in the sample image; performing reflection verification on the initial information according to a deviation between the text features contained in the initial information and the local features.
12. An information extraction system, the information extraction system comprising: an encoding module, a feature fusion module, and a decoding module; the encoding module is configured to determine image features of a to-be-processed image input into the multi-modal information extraction model and text features of an information extraction prompt word; the feature fusion module is configured to perform feature fusion on the image features and the text features of the information extraction prompt word to obtain fusion features; the decoding module is configured to decode the fusion features to perform information extraction on the to-be-processed image to obtain initial information, and perform reflection verification on the initial information according to a target feature of the to-be-processed image to verify a matching condition between the initial information and the to-be-processed image, and process the initial information into corresponding target information based on a reflection verification result.
13. The information extraction system of claim 12, the encoding module comprising: a visual encoding module and a text encoding module; the visual encoding module is configured to extract image features from the to-be-processed image; the text encoding module is configured to extract text features from the information extraction prompt word; the target feature includes the image features or fusion features determined based on the image features and the text features of the information extraction prompt word.
14. The information extraction system of claim 13, wherein the visual encoding module is specifically configured to: extracting features of the image to be processed based on the dynamic coding vector, to obtain the image features; wherein The dynamic coding vector is used to determine the length of the coding vector of the image feature, and within a preset resolution range, the size of the resolution of the to-be-processed image is positively correlated with the length of the coding vector.
15. The information extraction system of claim 13, wherein the decoding module is specifically configured to: adjust the text feature contained in the initial information with the adjustment target of minimizing the deviation between the text feature contained in the initial information and the target feature, and determine the adjusted information based on the adjusted text feature.
16. An information extraction apparatus, comprising: an extraction unit configured to input a to-be-processed image and an information extraction prompt word into a pre-trained multi-modal information extraction model to instruct the multi-modal information extraction model to: perform information extraction on the to-be-processed image based on the information extraction prompt word to obtain initial information, and perform reflection verification on the initial information according to a target feature corresponding to the to-be-processed image to verify the matching between the initial information and the to-be-processed image, and process the initial information into corresponding target information based on the reflection verification result; an acquisition unit configured to acquire the target information output by the multi-modal information extraction model.
17. A model training apparatus, comprising: an input unit configured to input a sample image and a sample information extraction prompt word into a multi-modal information extraction model to be trained to perform information extraction on the sample image based on the sample information extraction prompt word by the information extraction model to obtain initial information, and perform reflection verification on the initial information according to a target feature corresponding to the sample image to verify the matching between the initial information and the sample image, and process the initial information into corresponding output information based on the reflection verification result; an adjustment unit configured to adjust model parameters of the multi-modal information extraction model according to the deviation between the actual label corresponding to the sample image and the output information.
18. An electronic device, comprising: a processor; a memory for storing processor-executable instructions; wherein the processor implements the steps of the method of any one of claims 1-11 by running the executable instructions.
19. A computer-readable storage medium having computer instructions stored thereon, the instructions being executed by a processor to implement the steps of the method of any one of claims 1-11.
20. A computer program product comprising computer program / instructions, the computer program / instructions being executed by a processor to implement the steps of the method of any one of claims 1-11.