Image element information identification method and device based on large model

Abstract

The invention provides an image element information identification method and device based on a large model, and the method comprises the steps: obtaining a target image; cutting the target image, and removing an invalid area in the target image; based on an OCR character recognition model, multiple pieces of character information and multiple pieces of character coordinate information in the target image are determined; on the basis of pre-defined template configuration information and an MMIE multi-mode question and answer extraction model, element recognition is carried out on the target image according to the multiple pieces of character information and the multiple pieces of character coordinate information in the target image, and an element recognition result is obtained; on the basis of an LKE algorithm, operator post-processing is carried out on the element recognition result, an operator post-processing result is obtained, and the operator post-processing is used for verifying the multiple pieces of character information, the multiple pieces of character coordinate information and the element recognition result; and determining an element extraction result of the target image according to the multiple pieces of character information, the multiple pieces of character coordinate information, the element recognition result and the operator post-processing result. The operation complexity is reduced, and the user experience is improved.

Description

Technical Field

[0001] The present application relates to the field of image recognition technology, and more specifically, to a method and device for recognizing image element information based on a large model. Background Art

[0002] The scene of identifying various types of element information in image files is widely used in many fields of daily production and life, including but not limited to: element recognition of bill information, element recognition of certificate information, recognition of key information in documents and other image recognition scenes with obvious element characteristics. The image recognition technology can quickly form the ability to extract image information, improve the accuracy of elements, and reduce the difficulty of user operation. It can be well adapted and play a great value in fields such as invoice recognition in travel systems, document entry in transaction systems, document recognition in the financial field, and identification of certificate information.

[0003] However, currently most methods in the field of image document recognition use model training or image layout positioning. The model training method has problems such as high labeling cost, long development cycle, and low online efficiency. The image layout positioning method improves the speed and reduces the cost. However, the image layout positioning method has poor support for complex images or images without obvious layouts and layouts. The defects of the above two image recognition methods have had a great impact on the application of image file recognition. Summary of the invention

[0004] In view of this, the purpose of this application is to provide a method and device for image element information recognition based on a large model, through the MMIE multimodal question and answer extraction model, a large model specially trained in the field of image recognition, for semantic understanding. The platform capability formed based on the MMIE multimodal question and answer extraction model can enable users to complete the definition of a type of image recognition with less configuration, reduce the complexity of operation, improve the user experience, and provide a highly scalable zero-code image recognition platform in the field of image recognition.

[0005] At the same time, this application can also define template configuration information and use operator post-processing to construct image recognition templates. Compared with templates defined in other ways, the recognition accuracy is higher and the service adaptability is stronger.

[0006] In a first aspect, an embodiment of the present application provides a method for identifying image element information based on a large model, characterized in that the method includes: acquiring a target image; cropping the target image to remove invalid areas in the target image; determining multiple text information and multiple text coordinate information in the target image based on an OCR text recognition model; based on pre-defined template configuration information and an MMIE multimodal question and answer extraction model, performing element recognition on the target image according to the multiple text information and multiple text coordinate information in the target image to obtain an element recognition result; performing operator post-processing on the element recognition result based on an LKE algorithm to obtain the operator post-processing result, wherein the operator post-processing is used to verify multiple text information, multiple text coordinate information and element recognition results; determining the element extraction result of the target image according to the multiple text information, multiple text coordinate information, element recognition results and operator post-processing results.

[0007] Optionally, the method further includes: identifying seal information in the target image to determine the seal information in the target image.

[0008] Optionally, based on predefined template configuration information and the MMIE multimodal question and answer extraction model, the step of performing element recognition on the target image according to multiple text information and multiple text coordinate information in the target image includes: determining element definition semantic configuration information according to the predefined template configuration information; controlling the MMIE multimodal question and answer extraction model to determine the element recognition result according to the multiple text information and multiple text coordinate information in the target image according to the element definition semantic configuration information.

[0009] Optionally, the step of performing operator post-processing on the element recognition result includes: performing correctness verification on the multiple text information based on the LKE engine to obtain a verification result, the correctness verification including text recognition verification and format verification; performing text correction on the multiple text information according to the text recognition verification result in the verification result; performing format correction on the multiple text information according to the format verification result in the verification result.

[0010] Optionally, the MMIE multimodal question and answer extraction model uses an XY-CUT algorithm to sort multiple text information and multiple text coordinate information, and uses a prompt method to ask and answer questions to the model, wherein the MMIE multimodal question and answer extraction model is a pre-trained model based on the MMIE large language model, and extracts information from the target image by embedding text, text strips, one-dimensional position information, and two-dimensional position information in sequence.

[0011] Optionally, the basic pre-training model uses LayoutLMV3 as the backbone network, based on MMLM, PBD, and SLP as pre-training schemes, and is trained with multiple construction data and multiple domain data. By strengthening the spatial relative position attention, increasing the spatial position attention weight, and modifying the original LayoutLMV3's attention calculation method, the ability to understand the document's spatial layout can be improved.

[0012] Optionally, the template configuration information is defined by the following steps: performing template definition, the template definition is used to describe multiple information of the image, the multiple information includes element identification information, classification definition information, MMIE extraction configuration information, operator post-processing configuration information, and parameter mapping information; performing classification information definition, the classification information definition is used in the image recognition process to identify whether the target image is the image type defined by the template; performing prompt pre-extraction and prompt configuration, the prompt pre-extraction is used to provide pre-extraction capabilities for the MMIE multimodal question and answer extraction model to identify available element information from the target image and form a configuration; performing operator configuration, the operator configuration is used to configure the LKE algorithm.

[0013] In a second aspect, an embodiment of the present application further provides an image element information recognition device based on a large model, the device comprising:

[0014] An image acquisition module, used for acquiring a target image;

[0015] An image cropping module, used to crop the target image to remove invalid areas in the target image;

[0016] An information recognition module, used for determining a plurality of text information and a plurality of text coordinate information in the target image based on an OCR text recognition model;

[0017] An element recognition module is used to recognize elements of the target image based on predefined template configuration information and the MMIE multimodal question-answering extraction model according to multiple text information and multiple text coordinate information in the target image to obtain an element recognition result;

[0018] A result verification module, used for performing operator post-processing on the element recognition result based on the LKE algorithm to obtain the operator post-processing result, wherein the operator post-processing is used for verifying the multiple text information, the multiple text coordinate information and the element recognition result;

[0019] The result determination module is used to determine the element extraction result of the target image according to the multiple text information, multiple text coordinate information, element recognition results and operator post-processing results.

[0020] In the third aspect, an embodiment of the present application also provides an electronic device, comprising: a processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor communicates with the memory through the bus, and the processor executes the machine-readable instructions to perform any step of the above-mentioned large model-based image element information recognition method.

[0021] In a fourth aspect, the embodiment of the present application further provides a computer-readable storage medium, characterized in that a computer program is stored on the computer-readable storage medium, and when the computer program is executed by a processor, the steps of any of the above-mentioned image element information recognition methods based on a large model are executed.

[0022] The large-model-based image element information recognition method and device provided in the embodiments of the present application perform semantic understanding through the MMIE multimodal question and answer extraction model, a large model specially trained in the field of image recognition. The platform capabilities formed based on the MMIE multimodal question and answer extraction model can enable users to complete the definition of a type of image recognition with less configuration, reduce operational complexity, improve user experience, and provide a highly scalable zero-code image recognition platform in the field of image recognition.

[0023] At the same time, this application can also define template configuration information and use operator post-processing to construct image recognition templates. Compared with templates defined in other ways, the recognition accuracy is higher and the service adaptability is stronger.

[0024] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, preferred embodiments are specifically cited below and described in detail with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0026] Figure 1 A flowchart of a method for identifying image element information based on a large model provided in an embodiment of the present application;

[0027] Figure 2 A schematic diagram of the principle of the MMIE multimodal question-answering extraction model provided in an embodiment of the present application;

[0028] Figure 3 A schematic diagram of the element extraction principle provided in the embodiment of the present application;

[0029] Figure 4 A flowchart of another large model-based image element information recognition method provided in an embodiment of the present application;

[0030] Figure 5 A schematic diagram of the structure of an image element information recognition device based on a large model provided in an embodiment of the present application;

[0031] Figure 6 A schematic structural diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0032] To make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical scheme in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all of the embodiments. The components of the embodiments of the present application usually described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the application claimed for protection, but merely represents the selected embodiments of the present application. Based on the embodiments of the present application, each other embodiment obtained by those skilled in the art without making creative work belongs to the scope of protection of the present application.

[0033] First, the application scenarios to which the present application is applicable are introduced. The present application can be applied in the field of image recognition technology.

[0034] Research has found that the scenarios of identifying various types of element information in image files are widely used in many fields of daily production and life, including but not limited to: element recognition of bill information, element recognition of certificate information, recognition of key information in documents, and other image recognition scenarios with obvious element characteristics. Image recognition technology can quickly form image information extraction capabilities, improve the accuracy of elements, and reduce the difficulty of user operation. It can be well adapted and play a great role in fields such as invoice recognition in travel systems, document entry in transaction systems, document recognition in the financial field, and identification of certificate information.

[0035] However, currently most methods in the field of image document recognition use model training or image layout positioning. The model training method has problems such as high labeling cost, long development cycle, and low online efficiency. The image layout positioning method improves the speed and reduces the cost. However, the image layout positioning method has poor support for complex images or images without obvious layouts and layouts. The defects of the above two image recognition methods have had a great impact on the application of image file recognition.

[0036] Based on this, the embodiments of the present application provide a method and device for identifying image element information based on a large model, which performs semantic understanding through the MMIE multimodal question and answer extraction model, a large model specially trained in the field of image recognition. The platform capabilities formed based on the MMIE multimodal question and answer extraction model can enable users to complete the definition of a type of image recognition with less configuration, reduce the complexity of operation, improve the user experience, and provide a highly scalable zero-code image recognition platform in the field of image recognition.

[0037] At the same time, this application can also define template configuration information and use operator post-processing to construct image recognition templates. Compared with templates defined in other ways, the recognition accuracy is higher and the service adaptability is stronger.

[0038] See also Figure 1 , Figure 1 The following is a flow chart of a method for identifying image element information based on a large model provided in an embodiment of the present application. Figure 1 As shown in , the image element information recognition method based on the large model provided in the embodiment of the present application includes:

[0039] S101, acquiring a target image.

[0040] Here, the target image may be an invoice image, a receipt image, a certificate image, etc.

[0041] S102: cropping the target image to remove invalid areas in the target image.

[0042] Here, the target image is cropped to remove invalid areas in the target image, thereby filtering the invalid areas in the file and eliminating interference factors.

[0043] S103 . Determine a plurality of text information and a plurality of text coordinate information in the target image based on an OCR text recognition model.

[0044] S104: Based on the predefined template configuration information and the MMIE multimodal question-answering extraction model, perform element recognition on the target image according to the multiple text information and multiple text coordinate information in the target image to obtain an element recognition result.

[0045] Specifically, based on the predefined template configuration information and the MMIE multimodal question answering extraction model, the step of performing element recognition on the target image according to the multiple text information and multiple text coordinate information in the target image includes:

[0046] Determine element definition semantic configuration information based on predefined template configuration information;

[0047] The MMIE multimodal question-answering extraction model is controlled to determine a factor recognition result according to the factor definition semantic configuration information and the multiple text information and the multiple text coordinate information in the target image.

[0048] Specifically, the MMIE multimodal question-answering extraction model uses an XY-CUT algorithm to sort multiple text information and multiple text coordinate information, and uses a prompt method to ask and answer questions for the model.

[0049] Among them, the MMIE multimodal question-answering extraction model is a pre-trained model based on the MMIE large language model. It embeds text, text strips, one-dimensional position information, and two-dimensional position information in sequence to extract information from the target image.

[0050] Specifically, the basic pre-training model uses LayoutLMV3 as the backbone network, based on MMLM, PBD, and SLP as pre-training schemes, and is trained with multiple construction data and multiple domain data. By strengthening the spatial relative position attention, increasing the spatial position attention weight, and modifying the original LayoutLMV3's attention calculation method, the ability to understand the document's spatial layout can be improved.

[0051] As an example, see Figure 2 , Figure 2 This is a schematic diagram of the principle of the MMIE multimodal question-answering extraction model provided in the embodiment of the present application. Figure 2 As shown in the figure, the MMIE multimodal question answering extraction model principle provided in the embodiment of the present application

[0052] Among them, the MMIE multimodal question-answering extraction model is based on the OCR text recognition results, uses the XY-CUT algorithm to sort the OCR text strips, and uses the prompt method to ask questions to the model. The model is pre-trained based on the MMIE large language model, and achieves the purpose of extraction by embedding text, text strips, one-dimensional position information, and two-dimensional position information in sequence.

[0053] For the basic pre-trained model: use LayoutLMV3 as the backbone network (model parameter volume 200 million), based on MMLM+PBD+SLP as the pre-training scheme, and train with 1 million+ construction data and 10,000+ domain data. By strengthening the spatial relative position attention, enhancing the spatial position attention weight, and modifying the original LayoutLMV3's attention calculation method, the model's ability to understand the document spatial layout is improved.

[0054] Exemplarily, the pre-trained model can use factor key information, factor type information (such as date, amount), and spatial location information to acquire factor information.

[0055] Specifically, the template configuration information can be defined by the following steps:

[0056] Performing template definition, the template definition is used to describe multiple information of the image, the multiple information including element identification information, classification definition information, MMIE extraction configuration information, operator post-processing configuration information, and parameter mapping information;

[0057] Defining classification information, wherein the classification information definition is used in the image recognition process to identify whether the target image is an image type defined by the template;

[0058] Perform prompt pre-extraction and prompt configuration, wherein the prompt pre-extraction is used to provide a pre-extraction capability for the MMIE multimodal question-answering extraction model to identify available element information from the target image and form a configuration;

[0059] Perform operator configuration, where the operator configuration is used to configure the LKE algorithm.

[0060] As an example, see Figure 3 , Figure 3 This is a schematic diagram of the principle of factor extraction provided by the embodiment of the present application. Figure 3 As shown in , the element extraction principle provided by the embodiment of the present application includes: target image 301, MMIE model extraction step 302 and LKE algorithm step 303.

[0061] Among them, after acquiring the target image, the OCR text recognition model is first used to extract the text information and coordinate information in the image; secondly, the user defines the prompt information for the elements to be recognized (prompt is a way to start the machine learning model, usually a question or a task description, which can be called intelligent semantic configuration information), and the platform organizes it and sends it to the MMIE multimodal extraction model. The model analyzes the OCR text information, including position information (Position), element Key information (Key), and element type information (Type), and returns the corresponding result of the element information; finally, by configuring the operator logic for each element result, using the LKE engine for text post-processing, the recognition result content is detected, corrected, and the output format Format is performed to improve the recognition accuracy and make the result meet user needs.

[0062] S105. Based on the LKE algorithm, perform operator post-processing on the element recognition result to obtain the operator post-processing result, wherein the operator post-processing is used to verify the multiple text information, the multiple text coordinate information and the element recognition result.

[0063] Specifically, the step of performing operator post-processing on the element recognition result includes: performing correctness verification on the multiple text information based on the LKE engine to obtain a verification result, the correctness verification includes text recognition verification and format verification; performing text correction on the multiple text information according to the text recognition verification result in the verification result; performing format correction on the multiple text information according to the format verification result in the verification result.

[0064] S106. Determine the element extraction result of the target image according to the multiple text information, the multiple text coordinate information, the element recognition result and the operator post-processing result.

[0065] As an example, see Figure 4 , Figure 4 This is a flow chart of another method for identifying image element information based on a large model provided in an embodiment of the present application. Figure 4 As shown, another large-model-based image element information recognition method provided in an embodiment of the present application includes: template definition, classification information definition, prompt pre-extraction, prompt configuration, operator configuration, element information encapsulation, MMIE Model calculation, LKE Engine calculation, image input, image cropping, OCR recognition, prompt recognition, operator post-processing, and element recognition completion.

[0066] Among them, template definition, classification information definition, prompt pre-extraction, prompt configuration, operator configuration, and element information encapsulation constitute the configuration workflow.

[0067] Among them, image input, image cropping, OCR recognition, prompt recognition, operator post-processing, and feature recognition complete the image recognition workflow.

[0068] Here, through the above-mentioned workflows, a zero-code platform for customizable image element recognition can be built. The platform can form a set of "templates" for each type of image and publish them as services, which can provide image recognition capabilities to the outside world.

[0069] Among them, the template definition step requires defining a set of templates for the image. The template will describe all the information of the image, including feature identification information, classification definition information, MMIE extraction configuration information, operator post-processing configuration information, parameter mapping information, etc.

[0070] Among them, the classification information definition step is an extended capability of the platform, which can provide classification capabilities for the template, that is, in the image recognition process, it automatically identifies whether the image uploaded by the user is the image type defined by the template.

[0071] Here, in order to improve the user operation experience, the MMIE Model provides a pre-extraction capability that can automatically identify available element information from the image and form a configuration.

[0072] Among them, the platform provides a more convenient QA configuration method for prompt configuration, and can be adjusted at any time, and finally selects the optimal solution as the configuration information for image recognition

[0073] like Figure 4 The embodiment shown in also combines the operator processing capabilities provided by LKE Engine to adjust the model output content or format, thereby improving recognition accuracy. Operator processing can also be adjusted at any time to meet the needs of external service capabilities.

[0074] Specifically, during the recognition process, the user provides the image to be recognized and inputs the image information by calling the external service capability;

[0075] The platform crops the image files, filters out invalid areas in the files, and eliminates interference factors;

[0076] Call the OCR text recognition capability model to obtain the text information and text coordinate information in the image;

[0077] The platform obtains the configuration information of the defined template, combines it with the OCR text information and passes it into the MMIE Model multimodal extraction model, which then performs feature recognition.

[0078] Perform operator post-processing on the results of model recognition, such as checking the correctness of the results, correcting erroneous results, and formatting the results that need to be adjusted in output format;

[0079] Encapsulate the final result for output.

[0080] Among them, the operator chain is composed of operator logic operation units. Each operator processes the basic operation logic once. In the actual extraction of bill elements, according to the actual situation, the operators are freely matched to form one or more operator chains. These operator chains will run in the LKE engine to complete the element extraction operation.

[0081] In this way, combined with the LKE operator template capabilities, the template can simultaneously have multimodal extraction and LKE operator extraction, with complementary advantages.

[0082] In addition, the MMIE Model multimodal extraction model can be combined with small sample training capabilities to perform image recognition using a basic pre-trained model + Lora model (a small Lora model is formed by training with a small number of samples to improve the recognition accuracy of a certain type of image), which can greatly improve the recognition accuracy of images.

[0083] Optionally, the method further includes: identifying seal information in the target image to determine the seal information in the target image.

[0084] The large-model-based image element information recognition method provided in the embodiment of the present application performs semantic understanding through the MMIE multimodal question and answer extraction model, a large model specially trained in the field of image recognition. The platform capability formed based on the MMIE multimodal question and answer extraction model allows users to complete the definition of a type of image recognition with less configuration, reduces the complexity of operation, improves the user experience, and provides a highly scalable zero-code image recognition platform in the field of image recognition.

[0085] At the same time, this application can also define template configuration information and use operator post-processing to construct image recognition templates. Compared with templates defined in other ways, the recognition accuracy is higher and the service adaptability is stronger.

[0086] Based on the same inventive concept, the embodiment of the present application also provides a large model-based image element information recognition device corresponding to the large model-based image element information recognition method. Since the principle of solving the problem by the device in the embodiment of the present application is similar to the above-mentioned large model-based image element information recognition method in the embodiment of the present application, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be repeated.

[0087] See also Figure 5 , Figure 5 The following is a schematic diagram of the structure of a large model-based image element information recognition device provided in an embodiment of the present application. Figure 5 As shown in , the image element information recognition device 500 based on the large model includes:

[0088] An image acquisition module 501 is used to acquire a target image;

[0089] An image cropping module 502 is used to crop the target image to remove invalid areas in the target image;

[0090] An information recognition module 503, used to determine a plurality of text information and a plurality of text coordinate information in the target image based on an OCR text recognition model;

[0091] The element recognition module 504 is used to perform element recognition on the target image according to the multiple text information and multiple text coordinate information in the target image based on the predefined template configuration information and the MMIE multimodal question answering extraction model to obtain an element recognition result;

[0092] A result verification module 505 is used to perform operator post-processing on the element recognition result based on the LKE algorithm to obtain the operator post-processing result, wherein the operator post-processing is used to verify the multiple text information, the multiple text coordinate information and the element recognition result;

[0093] The result determination module 506 is used to determine the element extraction result of the target image according to the multiple text information, multiple text coordinate information, element recognition results and operator post-processing results.

[0094] The large-model-based image element information recognition device provided in the embodiment of the present application performs semantic understanding through the MMIE multimodal question and answer extraction model, a large model specially trained in the field of image recognition. The platform capability formed based on the MMIE multimodal question and answer extraction model allows users to complete the definition of a type of image recognition with less configuration, reduces the complexity of operation, improves the user experience, and provides a highly scalable zero-code image recognition platform in the field of image recognition.

[0095] At the same time, this application can also define template configuration information and use operator post-processing to construct image recognition templates. Compared with templates defined in other ways, the recognition accuracy is higher and the service adaptability is stronger.

[0096] See also Figure 6 , Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. Figure 6 As shown in , the electronic device 600 includes a processor 610 , a memory 620 and a bus 630 .

[0097] The memory 620 stores machine-readable instructions executable by the processor 610. When the electronic device 600 is running, the processor 610 communicates with the memory 620 via the bus 630. When the machine-readable instructions are executed by the processor 610, the above-mentioned Figure 1 The specific implementation of the steps of the large model-based image element information recognition method in the method embodiment shown can be found in the method embodiment, and will not be repeated here.

[0098] The present application also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the computer program can execute the above-mentioned Figure 1 The specific implementation of the steps of the large model-based image element information recognition method in the method embodiment shown can be found in the method embodiment, and will not be repeated here.

[0099] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0100] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical or other forms.

[0101] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0102] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0103] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a non-volatile computer-readable storage medium that is executable by a processor. Based on this understanding, the technical solution of the present application can essentially be embodied in the form of a software product, or in other words, the part that contributes to the prior art or the part of the technical solution. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0104] Finally, it should be noted that the above-described embodiments are only specific implementation methods of the present application, which are used to illustrate the technical solutions of the present application, rather than to limit them. The protection scope of the present application is not limited thereto. Although the present application is described in detail with reference to the above-mentioned embodiments, ordinary technicians in the field should understand that any technician familiar with the technical field can still modify the technical solutions recorded in the above-mentioned embodiments within the technical scope disclosed in the present application, or can easily think of changes, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. A method for identifying image element information based on a large model, characterized in that: The method comprises: Acquire target images; Cropping the target image to remove invalid areas in the target image; Based on the OCR text recognition model, determine multiple text information and multiple text coordinate information in the target image; Based on the predefined template configuration information and the MMIE multimodal question answering extraction model, the target image is subjected to element recognition according to the multiple text information and multiple text coordinate information in the target image to obtain an element recognition result; Based on the LKE algorithm, the element recognition result is subjected to operator post-processing to obtain the operator post-processing result, wherein the operator post-processing is used to verify the multiple text information, the multiple text coordinate information and the element recognition result; The element extraction result of the target image is determined according to the multiple text information, the multiple text coordinate information, the element recognition result and the operator post-processing result.

2. The method according to claim 1, characterized in that The method further comprises: The seal information in the target image is identified to determine the seal information in the target image.

3. The method according to claim 1, characterized in that Based on the predefined template configuration information and the MMIE multimodal question answering extraction model, the step of performing element recognition on the target image according to the multiple text information and multiple text coordinate information in the target image includes: Determine element definition semantic configuration information based on predefined template configuration information; The MMIE multimodal question-answering extraction model is controlled to determine a factor recognition result according to the factor definition semantic configuration information and the multiple text information and the multiple text coordinate information in the target image.

4. The method according to claim 1, characterized in that: The step of performing operator post-processing on the element recognition result comprises: Performing a correctness check on the plurality of text messages based on the LKE engine to obtain a check result, wherein the correctness check includes a text recognition check and a format check; Performing text correction on the plurality of text messages according to the text recognition verification result in the verification result; According to the format verification result in the verification result, the format of the plurality of text messages is corrected.

5. The method according to claim 1, characterized in that The MMIE multimodal question-answering extraction model uses the XY-CUT algorithm to sort multiple text information and multiple text coordinate information, and uses the prompt method to ask and answer questions for the model. Among them, the MMIE multimodal question-answering extraction model is a pre-trained model based on the MMIE large language model. It embeds text, text strips, one-dimensional position information, and two-dimensional position information in sequence to extract information from the target image.

6. The method according to claim 5, characterized in that The basic pre-training model uses LayoutLMV3 as the backbone network, based on MMLM, PBD, and SLP as pre-training schemes, and is trained with multiple construction data and multiple domain data. By strengthening the spatial relative position attention, increasing the spatial position attention weight, and modifying the original LayoutLMV3's attention calculation method, the ability to understand the document's spatial layout is improved.

7. The method according to claim 3, characterized in that The template configuration information is defined by the following steps: Performing template definition, the template definition is used to describe multiple information of the image, the multiple information including element identification information, classification definition information, MMIE extraction configuration information, operator post-processing configuration information, and parameter mapping information; Defining classification information, wherein the classification information definition is used in the image recognition process to identify whether the target image is an image type defined by the template; Perform prompt pre-extraction and prompt configuration, wherein the prompt pre-extraction is used to provide a pre-extraction capability for the MMIE multimodal question-answering extraction model to identify available element information from the target image and form a configuration; Perform operator configuration, where the operator configuration is used to configure the LKE algorithm.

8. A large-model-based image element information recognition device, characterized in that: The device comprises: An image acquisition module, used for acquiring a target image; An image cropping module, used to crop the target image to remove invalid areas in the target image; An information recognition module, used for determining a plurality of text information and a plurality of text coordinate information in the target image based on an OCR text recognition model; An element recognition module is used to recognize elements of the target image based on predefined template configuration information and the MMIE multimodal question-answering extraction model according to multiple text information and multiple text coordinate information in the target image to obtain an element recognition result; A result verification module, used for performing operator post-processing on the element recognition result based on the LKE algorithm to obtain the operator post-processing result, wherein the operator post-processing is used for verifying the multiple text information, the multiple text coordinate information and the element recognition result; The result determination module is used to determine the element extraction result of the target image according to the multiple text information, multiple text coordinate information, element recognition results and operator post-processing results.

9. An electronic device, characterized in that: include: A processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor communicates with the memory via the bus, and the processor executes the machine-readable instructions to perform the steps of the large model-based image element information recognition method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the large model-based image element information recognition method as claimed in any one of claims 1 to 7 are executed.

Images