Image information recognition method and device based on multi-modal large model

Abstract

The application discloses an image information recognition method and device based on a multi-modal large model, relates to the technical field of artificial intelligence, and aims to solve the problem of low recognition accuracy of image information, and comprises the following steps: acquiring certificate image information; extracting a certificate type in the certificate image information based on a multi-modal visual language model, and performing visual conversion based on a structured prompt word of the certificate type to obtain a visual representation; performing fusion processing on the visual representation and the certificate image information to obtain a structured text representation, and performing structured relation extraction on the structured text representation according to the certificate type to obtain a structured image information recognition result.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to an image information recognition method and apparatus based on a multimodal large model. Background Technology

[0002] With the rapid integration and development of digital technology and financial services, multi-type document recognition and information extraction have become core technologies for scenarios such as identity verification and business review. By automatically identifying document types from various document images and extracting structured information, complex relationships between people can be analyzed to support automated business processing.

[0003] Currently, image information recognition for various types of identification documents typically relies on a combination of traditional Optical Character Recognition (OCR) and Named Entity Recognition (NER). This involves first using OCR to recognize the text, and then using an NER model to extract entity information. However, this combination of OCR and NER methods lacks prior understanding of the document type, cannot adjust the information recognition strategy according to the document type, and struggles to handle information involving multiple individuals and relationships, resulting in poor accuracy in image information recognition. Summary of the Invention

[0004] One technical problem that this invention aims to solve is the low accuracy of image information recognition.

[0005] According to one aspect of this application, an image information recognition method based on a multimodal large model is provided, comprising: acquiring document image information;

[0006] The document type is extracted from the document image information based on a multimodal visual language model, and a visual representation is obtained based on the structured prompts of the document type.

[0007] The visual representation and the document image information are fused to obtain a structured text representation. The structured relationship of the structured text representation is then extracted according to the document type to obtain the structured image information recognition result.

[0008] Furthermore, before performing visual conversion on the structured prompts based on the document type to obtain a visual representation, the method further includes:

[0009] Generate structured prompt words that match the document type, the structured prompt words including field definition words, format requirement words, and relation definition words;

[0010] The visual transformation of the structured prompts based on the document type to obtain a visual representation includes:

[0011] Based on a preset visual transformation component, the field definition words, the format requirement words, and the relationship definition words are visualized and transformed to obtain labeled image representations, chart representations, and relationship graph representations.

[0012] Furthermore, after generating the structured prompt word matching the document type, the method further includes:

[0013] Determine the word similarity between any two words among the field definition words, the format requirement words, and the relation definition words;

[0014] When the word similarity is greater than a preset similarity threshold, the two words are deduplicated so that the deduplicated structured prompt words can be used for visual transformation.

[0015] Furthermore, the step of extracting the document type from the document image information based on the multimodal visual language model includes:

[0016] Obtain type detection prompts, which are used to characterize the definition content and judgment criteria of different document types;

[0017] A multimodal classification input is constructed based on the type detection prompts and the encoded information of the document image.

[0018] The multimodal classification input is used as the input parameter of the multimodal visual language model for classification processing to obtain the document type and classification confidence. The multimodal visual language model is constructed based on an encoder-decoder network with an attention mechanism.

[0019] Furthermore, the acquisition type detection prompt words include:

[0020] In response to the prompt word type configuration instruction, receive uploaded type detection prompt words that match different document types; or,

[0021] The system identifies target reference words in the input reference document image information and retrieves type detection prompt words that match the target reference words based on a preset reference mapping relationship. The preset reference mapping relationship is configured with type detection prompt words corresponding to different target reference words.

[0022] The document types include ID card only, ID card in hand, household registration book, birth certificate, marriage certificate, relationship certificate, and other types.

[0023] Furthermore, after classifying the multimodal classification input as input parameters of the multimodal visual language model to obtain the document type and classification confidence, the method further includes:

[0024] If the classification confidence level is less than a preset threshold, then optical character recognition is performed on the document image information to obtain an optical recognition result.

[0025] Furthermore, the process of fusing the visual representation and the document image information to obtain a structured text representation includes:

[0026] A multimodal recognition input is constructed based on the visual representation, the structured prompts, and the document image information;

[0027] The multimodal visual language model is invoked, and the multimodal recognition input is used as the input parameter of the multimodal visual language model for recognition processing to obtain the structured text representation.

[0028] Furthermore, the step of extracting structured relationships from the structured text representation according to the document type to obtain the structured image information recognition result includes:

[0029] Retrieve the relationship extraction strategy that matches the document type and create relationship extraction prompts;

[0030] According to the relation extraction strategy and the relation extraction prompts, the structured text representation is extracted using a large language model to obtain the structured image information recognition result.

[0031] Furthermore, the method also includes:

[0032] If the structured image information recognition result matches the extraction failure condition, then the entity information represented by the structured text is output; or,

[0033] If the structured text indicates a failure to match the recognition condition, then optical character recognition is performed on the document image information to obtain the optical recognition result.

[0034] According to another aspect of this application, an image information recognition device based on a multimodal large model is provided, comprising: an acquisition module for acquiring document image information;

[0035] The extraction module is used to extract the document type from the document image information based on a multimodal visual language model, and to perform a visual transformation based on the structured prompts of the document type to obtain a visual representation;

[0036] The extraction module is used to fuse the visual representation and the document image information to obtain a structured text representation, and to extract the structured relationship of the structured text representation according to the document type to obtain the structured image information recognition result.

[0037] Furthermore, the device also includes: a generation module,

[0038] The generation module is used to generate structured prompt words that match the document type. The structured prompt words include field definition words, format requirement words, and relationship definition words.

[0039] The extraction module is specifically used to perform visual transformations on the field definition words, the format requirement words, and the relationship definition words based on a preset visual transformation component, to obtain labeled image representations, chart representations, and relationship graph representations.

[0040] Furthermore, the generation module is also used to determine the word similarity between any two words among the field definition words, the format requirement words, and the relationship definition words; when the word similarity is greater than a preset similarity threshold, the two words are deduplicated so as to use the deduplicated structured prompt words for visual conversion.

[0041] Furthermore,

[0042] The extraction module is specifically used to obtain type detection prompts, which are used to characterize the definition content and judgment criteria of different document types; construct a multimodal classification input based on the type detection prompts and the encoded information of the document image information; and use the multimodal classification input as the input parameter of the multimodal visual language model for classification processing to obtain the document type and classification confidence. The multimodal visual language model is constructed based on an encoder-decoder network with an attention mechanism.

[0043] Furthermore, the acquisition module is also used to respond to the prompt word type configuration instruction by receiving uploaded type detection prompt words that match different document types; or, identifying target benchmark words in the entered benchmark document image information, and retrieving type detection prompt words that match the target benchmark words based on a preset benchmark mapping relationship, wherein the preset benchmark mapping relationship is configured with type detection prompt words corresponding to different target benchmark words; wherein, the document types include only ID card type, handheld ID card type, household registration book type, birth certificate type, marriage certificate type, relationship certificate type, and other types.

[0044] Furthermore, the device also includes:

[0045] If the classification confidence level is less than a preset threshold, the recognition module performs optical character recognition on the document image information to obtain an optical recognition result.

[0046] Furthermore,

[0047] The extraction module is specifically used to construct a multimodal recognition input based on the visual representation, the structured prompts, and the document image information; to retrieve the multimodal visual language model, and to use the multimodal recognition input as the input parameters of the multimodal visual language model for recognition processing, thereby obtaining the structured text representation.

[0048] Furthermore,

[0049] The extraction module is further configured to retrieve a relation extraction strategy that matches the document type and create relation extraction prompts; according to the relation extraction strategy and the relation extraction prompts, a large language model is used to extract relations from the structured text representation to obtain the structured image information recognition result.

[0050] Furthermore,

[0051] The recognition module is further configured to output the entity information represented by the structured text if the recognition result of the structured image information matches the extraction failure condition; or, if the structured text representation matches the recognition failure condition, perform optical character recognition on the document image information to obtain an optical recognition result.

[0052] According to another aspect of this application, a computer-readable storage medium is provided, wherein at least one executable instruction is stored therein, the executable instruction causing a processor to perform operations corresponding to the above-described image information recognition method based on a multimodal large model.

[0053] According to another aspect of this application, a computer device is provided, comprising: a processor, a memory, a communication interface, and a communication bus, wherein the processor, the memory, and the communication interface communicate with each other through the communication bus;

[0054] The memory is used to store at least one executable instruction, which causes the processor to perform the operation corresponding to the above-described image information recognition method based on a multimodal large model.

[0055] By employing the above technical solutions, the technical solutions provided in the embodiments of this application have at least the following advantages:

[0056] This application provides an image information recognition method and apparatus based on a multimodal large model. Compared with the prior art, the embodiments of this application acquire document image information; extract the document type from the document image information based on a multimodal visual language model, and perform visual transformation based on the structured prompt words of the document type to obtain a visual representation; fuse the visual representation and the document image information to obtain a structured text representation, and extract structured relationships from the structured text representation according to the document type to obtain a structured image information recognition result. This achieves the goal of accurately recognizing the relationship content in the document image, improves the understanding ability of image and text recognition, demonstrates the high performance of relationship recognition in document images, and also greatly improves the system data processing performance, thereby improving the recognition accuracy of image information.

[0057] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0058] The accompanying drawings, which form part of this specification, illustrate embodiments of the invention and, together with the description, serve to explain the principles of the invention.

[0059] The invention will be more clearly understood with reference to the accompanying drawings and the following detailed description, wherein:

[0060] Figure 1 This invention provides a flowchart of an image information recognition method based on a multimodal large model according to an embodiment of the present invention.

[0061] Figure 2 This invention provides a flowchart of another image information recognition method based on a multimodal large model, according to an embodiment of the present invention.

[0062] Figure 3 This diagram illustrates a structured text fusion method provided by an embodiment of the present invention.

[0063] Figure 4 This diagram illustrates a complete process for document image recognition according to an embodiment of the present invention.

[0064] Figure 5 This diagram illustrates a block diagram of an image information recognition device based on a multimodal large model, according to an embodiment of the present invention.

[0065] Figure 6 A schematic diagram of the structure of a computer device provided in an embodiment of the present invention is shown. Detailed Implementation

[0066] Various exemplary embodiments of the present invention will now be described in detail with reference to the accompanying drawings. It should be noted that, unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps set forth in these embodiments do not limit the scope of the invention.

[0067] At the same time, it should be understood that, for ease of description, the dimensions of the various parts shown in the accompanying drawings are not drawn according to actual scale.

[0068] The following description of at least one exemplary embodiment is merely illustrative and is in no way intended to limit the invention or its application or use.

[0069] Techniques, methods, and equipment known to those skilled in the art may not be discussed in detail, but where appropriate, such techniques, methods, and equipment should be considered part of the specification.

[0070] It should be noted that similar labels and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be discussed further in subsequent figures.

[0071] Embodiments of this invention can be applied to computer systems / servers that can operate with a wide range of other general-purpose or special-purpose computing system environments or configurations. Examples of well-known computing systems, environments, and / or configurations suitable for use with computer systems / servers include, but are not limited to: personal computer systems, server computer systems, thin clients, thick clients, handheld or laptop devices, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputer systems, mainframe computer systems, and distributed cloud computing environments that include any of the above systems, etc.

[0072] Computer systems / servers can be described in the general context of computer system executable instructions (such as program modules) executed by the computer system. Typically, program modules can include routines, programs, object programs, components, logic, data structures, etc., which perform specific tasks or implement specific abstract data types. Computer systems / servers can be implemented in distributed cloud computing environments, where tasks are performed by remote processing devices linked through a communication network. In distributed cloud computing environments, program modules can reside on local or remote computing system storage media, including storage devices.

[0073] This application provides an image information recognition method based on a multimodal large model, such as... Figure 1 As shown, the method includes:

[0074] 101. Obtain image information of the certificate.

[0075] In this embodiment, the current execution end, as the subject of image information recognition, can be a terminal computer or a remote server, etc., in order to obtain document image information. The document image information can be obtained by taking pictures of different types of documents. Different types of documents may include, but are not limited to, ID cards, handheld ID cards, household registration books, birth certificates, marriage certificates, relationship certificates, etc. This embodiment does not make specific limitations.

[0076] It should be noted that the document image information in this embodiment can be pre-shot and stored, or it can be uploaded by the user in real time. Furthermore, since the document image information is in image format, to ensure the standardization of subsequent processing, the current execution terminal first encodes the document image information into base64 format and performs size normalization processing in order to execute step 102.

[0077] 102. Extract the document type from the document image information based on a multimodal visual language model, and perform a visual transformation based on the structured prompts of the document type to obtain a visual representation.

[0078] In this embodiment, the multimodal visual language model refers to a large open-source model with processing functions such as recognizing common objects, analyzing text, charts, and other elements in images, and accurately locating objects in images. It is preferably a multi-version model such as Qwen-VL, but this embodiment does not impose specific limitations. The document types can include, but are not limited to, ID cards, handheld ID cards, household registration books, birth certificates, marriage certificates, relationship certificates, etc., which are extracted by inputting the document image information into the multimodal visual language model.

[0079] It should be noted that structured prompts refer to word content generated for different document types, used to prompt large-scale models to convert into visual images. This can include extracted fields, format requirements, relationship definitions, etc., which are not specifically limited in this embodiment. Simultaneously, after receiving the structured prompts, the current execution end performs a visualization conversion, that is, converts the text content into a visual image representation, i.e., a visual representation, such as a chart, labeled image, etc., which are not specifically limited in this embodiment.

[0080] 103. The visualization representation and the document image information are fused to obtain a structured text representation, and the structured relationship of the structured text representation is extracted according to the document type to obtain the structured image information recognition result.

[0081] In this embodiment, since the visual representation is in image form and the original document image information is also in image form, to enhance the recognition effectiveness of the image information content, the current execution end performs fusion processing on the visual representation and the document image information to obtain a structured text representation. At this time, OCR recognition can be used followed by text fusion to obtain the structured text representation. Furthermore, since different document types have different text structures, structured relation extraction is performed on the structured text representation for different document types to obtain the final structured image information recognition result.

[0082] It should be noted that when the current execution end performs structured relation extraction, it means extracting the text content in the structured text representation according to the extraction strategy that matches the document type, so as to form an image recognition result with a specific structure and correlation.

[0083] In this application embodiment, the scenarios used for image information recognition include the financial, government, medical, and enterprise sectors. The financial sector may include scenarios such as identity verification, account opening review, and loan approval, requiring the recognition of various documents and extraction of structured information. The government sector may include scenarios such as government service processing and document verification, requiring the recognition of multiple types of documents and the parsing of personnel relationships. The medical sector may include scenarios such as patient identity verification and medical record management, requiring the recognition of documents and the extraction of patient and family information. The enterprise sector may include scenarios such as employee onboarding and customer identity verification, requiring the recognition of documents and the extraction of structured information.

[0084] In another embodiment of this application, for further definition and explanation, before the step of performing a visual conversion based on the structured prompts of the document type to obtain a visual representation, the method further includes:

[0085] Generate structured prompts that match the document type.

[0086] To incorporate prompts into visual recognition and improve the accuracy of image information recognition, the current execution end generates matching structured prompts for different document types. In this case, a mapping relationship between different document types and different structured prompts can be pre-configured so that structured prompts are generated based on the document type. The structured prompts include field definition terms, format requirement terms, and relationship definition terms. Field definition terms represent the list of fields to be extracted, including but not limited to name, ID number, address, and date of birth. Format requirement terms represent field format specifications, including but not limited to date formats and JSON structures. Relationship definition terms represent the type of relationship between individuals, including head of household-member, parent-child, and spouse, etc. This embodiment does not impose specific limitations.

[0087] For example, generating a structured prompt P based on document type T can be done based on the prompt project Prompt, as follows:

[0088] P = GeneratePrompt(T, FieldDefinitions, FormatRequirements);

[0089] Among them, FieldDefinitions are field definition terms (such as name, ID number, address, date of birth), and FormatRequirements are format requirement terms (such as date format, JSON structure).

[0090] Correspondingly, the step of performing a visual transformation based on the structured prompts of the document type to obtain a visual representation includes:

[0091] Based on a preset visual transformation component, the field definition words, the format requirement words, and the relationship definition words are visualized and transformed to obtain labeled image representations, chart representations, and relationship graph representations.

[0092] Since structured prompts include field definition words, format requirement words, and relation definition words, in order to achieve effective visualization transformation and efficient encoding of prompt information, thereby improving the recognition of text content based on visual representations, the current execution end uses a preset visual transformation component to perform visualization transformations on field definition words, format requirement words, and relation definition words respectively, resulting in labeled image representations, chart representations, and relation graph representations. The visualization representations include labeled image representations, chart representations, and relation graph representations. The preset visual transformation component can be the VisualizePrompt component, which converts the prompt word P into a visual representation V_prompt.

[0093] V_prompt = VisualizePrompt(P);

[0094] In some embodiments, field definition terms are converted into labeled images, i.e., the field positions are labeled on example images; format requirement terms are converted into charts, such as JSON structure diagrams; and relationship definition terms are converted into relationship diagrams, such as family member relationship diagrams. This application embodiment does not impose specific limitations. Specifically, the labeled image represents the labeling of field positions on the example image, generating a labeled image I_annotated; the chart represents the conversion of the JSON structure into a visual chart, generating a structure diagram Chart_structure; and the relationship diagram represents the conversion of relationship definitions into a relationship diagram, generating a relationship diagram Graph_relation.

[0095] In another embodiment of this application, for further definition and explanation, after generating the structured prompt word matching the document type, the method further includes:

[0096] Determine the word similarity between any two words among the field definition words, the format requirement words, and the relation definition words;

[0097] When the word similarity is greater than a preset similarity threshold, the two words are deduplicated so that the deduplicated structured prompt words can be used for visual transformation.

[0098] To ensure that structured prompts are simplified and accurate, thereby improving the effectiveness of visual transformation using structured prompts, the current execution end performs deduplication and simplification on the structured prompts after receiving them. Specifically, the word similarity between any two words among field definition words, format requirement words, and relation definition words is first determined. That is, any two words are selected sequentially from the field definition words, format requirement words, and relation definition words as the words whose similarity needs to be calculated. At this time, if there are multiple words for each of the field definition words, format requirement words, and relation definition words, any two words can be selected from multiple field definition words, or any two words can be selected from multiple format requirement words, or any two words can be selected from multiple relation definition words. This application embodiment does not make specific limitations. Furthermore, after calculating the similarity of each word, it is compared with a preset similarity threshold. When the word similarity is greater than the preset similarity threshold, it indicates that there are similar words among the field definition words, format requirement words, and relation definition words that can be deduplicated. Therefore, deduplication is performed on any two words to utilize the simplified structured prompt words for visual conversion. The preset similarity threshold can be configured based on different deduplication requirements. Furthermore, different field definition words, format requirement words, and relation definition words can be configured with corresponding preset similarity thresholds; this embodiment does not impose specific limitations.

[0099] In another embodiment of this application, for further definition and explanation, such as Figure 2 As shown, the steps for extracting the document type from the document image information based on a multimodal visual language model include:

[0100] 201. Obtain type detection prompts;

[0101] 202. Construct a multimodal classification input based on the type detection prompt words and the encoded information of the document image information;

[0102] 203. The multimodal classification input is used as the input parameter of the multimodal visual language model for classification processing to obtain the document type and classification confidence.

[0103] To achieve accurate extraction of document types and improve the effectiveness of information extraction using document types, the current execution end first obtains type detection prompts. These prompts characterize the definition and judgment criteria of different document types. For example, the seven document types include ID card only, handheld ID card, household registration book, birth certificate, marriage certificate, relationship certificate, and others, clearly defining the visual features and key information of each type. These prompts can be configured based on different type detection needs, and this embodiment does not impose specific limitations. Furthermore, a multimodal classification input is constructed based on the type detection prompts and the encoded information of the document image, represented as Input = {image: I_base64, prompt: P_type_detection}.

[0104] Where P_type_detection is the type detection cue word, and I_base64 is the document image information. Then, the multimodal classification input is used as the input parameters of the multimodal visual language model for classification processing to obtain the document type and classification confidence, expressed as:

[0105] T, C_type = MultimodalClassifier(Input);

[0106] Among them, MultimodalClassifier is a visual language model based on the Transformer architecture, preferably the open-source Qwen-VL model. It outputs the document type label T and the classification confidence C_type, where T∈{id_card, handheld_id_card, household_register, birth_certificate, marriage_certificate, relationship_proof, other}, corresponding to the types of only ID card, handheld ID card, household registration book, birth certificate, marriage certificate, relationship proof, and other types, respectively.

[0107] It should be noted that the multimodal visual language model in this application embodiment is constructed based on an encoder-decoder network that introduces an attention mechanism, preferably an open-source wen-VL series model, including but not limited to Qwen-VL-Plus, Qwen-VL-Max, Qwen2.5-VL, Qwen3-VL, etc., and this application embodiment does not make specific limitations.

[0108] In some embodiments, an open-source multimodal vision-language model pre-trained by the current execution end is set with temperature = 0.01 to ensure output stability, max_tokens = 2048, and a timeout of 120 seconds. During training, the JSON format results returned by the multimodal vision-language model are parsed to extract pic_type (type name), confidence, reason (basis for judgment), and key_features (key features). Finally, type mapping refers to mapping Chinese type names to English identifiers, such as mapping "household register" to "household_register" for subsequent processing. In the specific training process, the Qwen-VL series of models can understand images and text simultaneously and comprehensively judge the document type based on visual and text features.

[0109] In another embodiment of the present application, for further limitation and explanation, the step of obtaining the type detection prompt word includes:

[0110] In response to a prompt word type configuration instruction, receive uploaded type detection prompt words matching different document types; or,

[0111] Identify the target reference word in the input reference document image information, and retrieve the type detection prompt word matching the target reference word based on a preset reference mapping relationship.

[0112] To meet the flexible detection requirements for document images and achieve the purpose of classifying document images based on type detection prompt words, in a specific embodiment, the current execution end can obtain type detection prompt words by receiving a user's instruction. That is, when the user triggers a prompt word type configuration instruction, after the current execution end receives this instruction, it can receive the uploaded type detection prompt words matching different document types. At this time, the document types include only ID card type, ID card held in hand type, household register type, birth certificate type, marriage certificate type, relationship certificate type, and other types, which are not specifically limited in the embodiments of the present application.

[0113] In another specific embodiment, the current execution terminal can also obtain type detection prompts through automatic recognition. Specifically, the user can pre-enter multiple baseline document image information for identifying target baseline words. For example, baseline document image information such as "ID card" and "household registration" can be entered for different document types. Then, image feature extraction can be performed based on image recognition technology, such as using a pre-trained convolutional neural network (CNN). This application embodiment does not make specific limitations. After obtaining the target baseline word, the current execution terminal retrieves the type detection prompt words that match the target baseline word based on a preset baseline mapping relationship. At this time, the preset baseline mapping relationship is configured with type detection prompt words corresponding to different target baseline words. The type detection prompt words can be queried according to the preset baseline mapping relationship based on the identified target baseline word, thereby achieving the purpose of automatically obtaining type detection prompt words. This application embodiment does not make specific limitations.

[0114] In another embodiment of this application, for further definition and explanation, after the step of using the multimodal classification input as input parameters of the multimodal visual language model for classification processing to obtain the document type and classification confidence, the method further includes:

[0115] If the classification confidence level is less than a preset threshold, then optical character recognition is performed on the document image information to obtain an optical recognition result.

[0116] To ensure the continued effectiveness of document image recognition even after document type detection failure, the current execution end uses classification confidence to determine the issue. This involves comparing the classification confidence with a preset threshold. If the classification confidence is less than the preset threshold, it indicates document type classification failure. In this case, traditional optical character recognition (OCR) technology is used to recognize the document image, yielding an optical recognition result. The preset threshold can be 0.5, 0.7, etc., preferably 0.5. If the classification confidence C_type < 0.5, an error message is returned, or a general OCR strategy is employed. This embodiment does not impose specific limitations.

[0117] In another embodiment of this application, for further definition and explanation, the step of fusing the visual representation and the document image information to obtain a structured text representation includes:

[0118] A multimodal recognition input is constructed based on the visual representation, the structured prompts, and the document image information;

[0119] The multimodal visual language model is invoked, and the multimodal recognition input is used as the input parameter of the multimodal visual language model for recognition processing to obtain the structured text representation.

[0120] To improve the accuracy of image information extraction and achieve the extraction of structured text content using visual fusion, the fusion process specifically involves first constructing a multimodal recognition input based on visual representations, structured prompts, and document image information, as follows:

[0121] Input_enhanced = {image: I, visual_prompt: V_prompt, text_prompt: P_text};

[0122] Where P_text is the structured prompt word of the simplified text prompt, I is the document image information, and V_prompt is the visual representation, V_prompt = {I_annotated, Chart_structure, Graph_relation}.

[0123] It should be noted that after constructing the multimodal recognition input, the current execution end calls the multimodal visual language model and uses the multimodal recognition input as the input parameter of the multimodal visual language model for recognition processing to obtain the structured text representation. At this time, the recognition of the structured text representation can be expressed as:

[0124] T_text = MultimodalOCR(Input_enhanced);

[0125] Among them, the MultimodalOCR model is a multimodal visual language model with image recognition capabilities, such as the Qwen-VL series model. This application does not make specific limitations on the embodiments. The final output is a structured text representation T_text, which makes full use of the visual understanding capabilities of the visual language model, greatly reduces the length of the text prompt, reduces the computational overhead, and improves the comprehensibility and executability of the prompt information.

[0126] In some specific embodiments, such as Figure 3 As shown, the original document image information I and its visual representation V_prompt are used to construct a multimodal input consisting of the original image I (document content), the visual prompt V_prompt (task guidance), and the simplified structured prompt P_text (providing basic recognition requirements). Then, an open-source multimodal visual language model trained on local data is invoked. Specifically, temperature=0.01, max_tokens=4096, and the timeout is 300 seconds. The multimodal visual language model generates structured text T_text based on visual features and visual prompts, maintaining the original text order and format. Simultaneously, the generated structured text representation can be formatted and cleaned to ensure text quality.

[0127] In another embodiment of this application, for further definition and explanation, the step of extracting structured relationships from the structured text representation according to the document type to obtain the structured image information recognition result includes:

[0128] Retrieve the relationship extraction strategy that matches the document type and create relationship extraction prompts;

[0129] According to the relation extraction strategy and the relation extraction prompts, the structured text representation is extracted using a large language model to obtain the structured image information recognition result.

[0130] To enhance the ability to extract relational content from document images and thus improve the accuracy of image information recognition, the current execution terminal, when performing structured relation extraction, first invokes a relation extraction strategy matching the document type. This relation extraction strategy characterizes the method or rule for extraction based on different relations and can be pre-configured based on different document types; this application embodiment does not impose specific limitations. For example, for the document type corresponding to an ID card / handheld ID card, the relation extraction strategy can be set to extract single-person information, with no relation extraction; for the document type corresponding to a household registration book, the relation extraction strategy can be set to extract information on the head of household and members, identifying the relationship between members and the head of household; for the document type corresponding to a birth certificate, the relation extraction strategy can be set to extract information on the newborn and parents, identifying the parent-child relationship; for the document type corresponding to a marriage certificate, the relation extraction strategy can be set to extract information on both spouses, identifying the marital relationship; and for the document type corresponding to a birth certificate, the relation extraction strategy can be set to extract information on the individual and related personnel, identifying complex relationship networks.

[0131] It should be noted that relation extraction prompts are used to describe trigger words or instruction words that direct the large language model to perform relation extraction, and can be represented as:

[0132] P_relation = BuildRelationPrompt(T, T_text, RelationTemplates);

[0133] Among them, RelationTemplates are used to characterize the mapping rules for prompt words extracted by the indicator word extraction project Prompt, and can be, for example, a list of relation templates.

[0134] Furthermore, relation extraction is performed using a large language model, represented as follows:

[0135] S_info = LLMRelationExtractor(P_relation);

[0136] Output the structured image information recognition result S_info, which includes a list of personnel information and a relationship network.

[0137] In some embodiments, corresponding relationship extraction templates can be configured for different relationship extraction strategies to illustrate the extraction methods applicable to different document types. For example, there are templates for extracting single-person information for ID cards / handheld ID cards, templates for extracting head-of-household-member relationships for household registration books, templates for extracting parent-child relationships for birth certificates, templates for extracting spousal relationships for marriage certificates, and templates for extracting complex relationship networks for relationship certificates. This application embodiment does not impose specific limitations on these templates. Furthermore, the structured image information recognition results can be represented in JSON format, including a list of personnel information and a relationship network. The personnel information list includes names, ID numbers, addresses, dates of birth, etc., and the relationship network includes the types of relationships between personnel, etc. This application embodiment does not impose specific limitations on these templates. The output JSON structure can be represented as follows:

[0138] {

[0139] "owner": {...}, / / Owner / Personal Information

[0140] "members": [...], / / Member / related personnel information

[0141] "relationships": [...] / / List of relationships

[0142] }

[0143] In another embodiment of this application, for further definition and explanation, the steps also include:

[0144] If the structured image information recognition result matches the extraction failure condition, then the entity information represented by the structured text is output; or,

[0145] If the structured text indicates a failure to match the recognition condition, then optical character recognition is performed on the document image information to obtain the optical recognition result.

[0146] To ensure the effectiveness of image information content recognition and avoid the helplessness caused by recognition failures, in one specific embodiment, after the current execution end obtains the structured image information recognition result, it first identifies whether the structured image information recognition result matches the extraction failure condition. In this case, the extraction failure condition can be set to the absence of extracted relationships in the structured image information recognition result. Therefore, the entity information represented by the structured text is output. The entity information can be text content represented by text; this embodiment does not impose specific limitations. In another specific embodiment, the obtained structured text representation is matched with the recognition failure condition. In this case, the recognition failure condition can be pre-configured to the absence of recognized text content. Therefore, traditional optical character recognition is performed on the document image information to obtain optical recognition results. Alternatively, partially recognized results can be returned or a failure prompt message can be directly output; this embodiment does not impose specific limitations.

[0147] In a specific example of the embodiments of this application, as the identification execution end, such as Figure 4 As shown, the image information recognition steps based on a multimodal large model may include:

[0148] Step S1: The current execution terminal receives the document image input. The user uploads the image through the API interface or provides the image URL. The current execution terminal receives the image and performs preprocessing.

[0149] Step S2: Call the document type detection module, input the image into the multimodal model, identify the document type T, and obtain the confidence score C_type. If C_type < 0.5, return an error message.

[0150] Step S3: The current execution end prompts for project optimization, generates a structured prompt word P based on the document type T, and converts P into a visual representation V_prompt;

[0151] Step S4: The current execution end performs multimodal OCR recognition, fuses the original image I with the visual cue V_prompt, calls the multimodal model to perform text recognition, and outputs structured text T_text;

[0152] Step S5: The current execution end performs relation extraction and structured parsing. Based on the document type T, it calls the large language model to perform semantic understanding and relation extraction on the text T_text and outputs structured information S_info.

[0153] Step S6: Result Validation and Formatting. Validate the structured information S_info to ensure field integrity and relationship correctness, and output the final structured JSON result.

[0154] In the above example, the verification was performed on a real multi-document dataset. The verification results can be analyzed from the following perspectives: the image information recognition method in this application embodiment greatly improves the accuracy of document classification, increases the recognition capability of image text, demonstrates the high performance of relationship recognition in document images, and also greatly improves the system data processing performance.

[0155] This application provides an image information recognition method based on a multimodal large model. Compared with the prior art, this application obtains document image information; extracts the document type from the document image information based on a multimodal visual language model, and performs a visual transformation based on the structured prompt words of the document type to obtain a visual representation; fuses the visual representation and the document image information to obtain a structured text representation, and extracts structured relationships from the structured text representation according to the document type to obtain a structured image information recognition result. This achieves the goal of accurately recognizing the relationship content in the document image, enhances the understanding ability of image and text recognition, demonstrates the high performance of relationship recognition in document images, and greatly improves the system's data processing performance, thereby improving the accuracy of image information recognition.

[0156] Furthermore, as a response to the above Figure 1 The implementation of the method shown in this application provides an image information recognition device based on a multimodal large model, such as... Figure 5 As shown, the device includes:

[0157] Module 31 is used to acquire document image information;

[0158] Extraction module 32 is used to extract the document type from the document image information based on a multimodal visual language model, and to perform a visual transformation based on the structured prompt words of the document type to obtain a visual representation;

[0159] The extraction module 33 is used to fuse the visual representation and the document image information to obtain a structured text representation, and to extract the structured relationship of the structured text representation according to the document type to obtain the structured image information recognition result.

[0160] Furthermore, the device also includes: a generation module,

[0161] The generation module is used to generate structured prompt words that match the document type. The structured prompt words include field definition words, format requirement words, and relationship definition words.

[0162] The extraction module is specifically used to perform visual transformations on the field definition words, the format requirement words, and the relationship definition words based on a preset visual transformation component, to obtain labeled image representations, chart representations, and relationship graph representations.

[0163] Furthermore, the generation module is also used to determine the word similarity between any two words among the field definition words, the format requirement words, and the relationship definition words; when the word similarity is greater than a preset similarity threshold, the two words are deduplicated so as to use the deduplicated structured prompt words for visual conversion.

[0164] Furthermore,

[0165] The extraction module is specifically used to obtain type detection prompts, which are used to characterize the definition content and judgment criteria of different document types; construct a multimodal classification input based on the type detection prompts and the encoded information of the document image information; and use the multimodal classification input as the input parameter of the multimodal visual language model for classification processing to obtain the document type and classification confidence. The multimodal visual language model is constructed based on an encoder-decoder network with an attention mechanism.

[0166] Furthermore, the acquisition module is also used to respond to the prompt word type configuration instruction by receiving uploaded type detection prompt words that match different document types; or, identifying target benchmark words in the entered benchmark document image information, and retrieving type detection prompt words that match the target benchmark words based on a preset benchmark mapping relationship, wherein the preset benchmark mapping relationship is configured with type detection prompt words corresponding to different target benchmark words; wherein, the document types include only ID card type, handheld ID card type, household registration book type, birth certificate type, marriage certificate type, relationship certificate type, and other types.

[0167] Furthermore, the device also includes:

[0168] If the classification confidence level is less than a preset threshold, the recognition module performs optical character recognition on the document image information to obtain an optical recognition result.

[0169] Furthermore,

[0170] The extraction module is specifically used to construct a multimodal recognition input based on the visual representation, the structured prompts, and the document image information; to retrieve the multimodal visual language model, and to use the multimodal recognition input as the input parameters of the multimodal visual language model for recognition processing, thereby obtaining the structured text representation.

[0171] Furthermore,

[0172] The extraction module is further configured to retrieve a relation extraction strategy that matches the document type and create relation extraction prompts; according to the relation extraction strategy and the relation extraction prompts, a large language model is used to extract relations from the structured text representation to obtain the structured image information recognition result.

[0173] Furthermore,

[0174] The recognition module is further configured to output the entity information represented by the structured text if the recognition result of the structured image information matches the extraction failure condition; or, if the structured text representation matches the recognition failure condition, perform optical character recognition on the document image information to obtain an optical recognition result.

[0175] This application provides an image information recognition device based on a multimodal large model. Compared with the prior art, this application acquires document image information; extracts the document type from the document image information based on a multimodal visual language model, and performs visual transformation based on the structured prompts of the document type to obtain a visual representation; fuses the visual representation and the document image information to obtain a structured text representation, and extracts structured relationships from the structured text representation according to the document type to obtain a structured image information recognition result. This achieves the goal of accurately recognizing the relationship content in the document image, enhances the understanding ability of image and text recognition, demonstrates the high performance of relationship recognition in document images, and greatly improves the system's data processing performance, thereby improving the accuracy of image information recognition.

[0176] According to one embodiment of this application, a computer-readable storage medium is provided, the storage medium storing at least one executable instruction that can execute the image information recognition method based on a multimodal large model in any of the above method embodiments.

[0177] Figure 6 The diagram shows a structural schematic of a computer device according to one embodiment of the present application. The specific embodiments of the present application do not limit the specific implementation of the computer device.

[0178] like Figure 6 As shown, the computer device may include: a processor 402, a communications interface 404, a memory 406, and a communications bus 408.

[0179] The processor 402, communication interface 404, and memory 406 communicate with each other via communication bus 408.

[0180] Communication interface 404 is used to communicate with other network elements such as clients or other servers.

[0181] The processor 402 is used to execute program 410, specifically to execute the relevant steps in the above embodiment of the image information recognition method based on a multimodal large model.

[0182] Specifically, program 410 may include program code that includes computer operation instructions.

[0183] Processor 402 may be a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of this application. The terminal includes one or more processors, which may be processors of the same type, such as one or more CPUs; or they may be processors of different types, such as one or more CPUs and one or more ASICs.

[0184] Memory 406 is used to store program 410. Memory 406 may include high-speed RAM memory, and may also include non-volatile memory, such as at least one disk storage device.

[0185] Specifically, program 410 can be used to cause processor 402 to perform the following operations:

[0186] Obtain document image information;

[0187] The document type is extracted from the document image information based on a multimodal visual language model, and a visual representation is obtained based on the structured prompts of the document type.

[0188] The visual representation and the document image information are fused to obtain a structured text representation. The structured relationship of the structured text representation is then extracted according to the document type to obtain the structured image information recognition result.

[0189] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For system embodiments, since they largely correspond to method embodiments, the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

[0190] The methods and systems of the present invention may be implemented in many ways. For example, they may be implemented by software, hardware, firmware, or any combination of software, hardware, and firmware. The above-described order of steps for the methods is for illustrative purposes only, and the steps of the methods of the present invention are not limited to the order specifically described above unless otherwise specifically stated. Furthermore, in some embodiments, the present invention may also be implemented as a program recorded on a recording medium, the program comprising machine-readable instructions for implementing the methods according to the present invention. Thus, the present invention also covers recording media storing programs for performing the methods according to the present invention.

[0191] The description of this invention is given for illustrative and descriptive purposes only and is not intended to be exhaustive or to limit the invention to the forms disclosed. Many modifications and variations will be apparent to those skilled in the art. The embodiments were chosen and described in order to better illustrate the principles and practical application of the invention and to enable those skilled in the art to understand the invention and to design various embodiments with various modifications suitable for a particular purpose.

Claims

1. An image information recognition method based on a multimodal large model, characterized in that, include: Obtain document image information; The document type is extracted from the document image information based on a multimodal visual language model, and a visual representation is obtained based on the structured prompts of the document type. The visual representation and the document image information are fused to obtain a structured text representation. The structured relationship of the structured text representation is then extracted according to the document type to obtain the structured image information recognition result. Before performing a visual conversion on the structured prompts based on the document type to obtain a visual representation, the method further includes: Generate structured prompt words that match the document type, the structured prompt words including field definition words, format requirement words, and relation definition words; The visual transformation of the structured prompts based on the document type to obtain a visual representation includes: Based on a preset visual transformation component, the field definition words, the format requirement words, and the relation definition words are visualized and transformed to obtain labeled image representations, chart representations, and relation graph representations. The process of fusing the visual representation and the document image information to obtain a structured text representation includes: A multimodal recognition input is constructed based on the visual representation, the structured prompts, and the document image information; The multimodal visual language model is invoked, and the multimodal recognition input is used as the input parameter of the multimodal visual language model for recognition processing to obtain the structured text representation.

2. The method according to claim 1, characterized in that, After generating the structured prompt word that matches the document type, the method further includes: Determine the word similarity between any two words among the field definition words, the format requirement words, and the relation definition words; When the word similarity is greater than a preset similarity threshold, the two words are deduplicated so that the deduplicated structured prompt words can be used for visual transformation.

3. The method according to claim 1, characterized in that, The document type extracted from the document image information based on the multimodal visual language model includes: Obtain type detection prompts, which are used to characterize the definition content and judgment criteria of different document types; A multimodal classification input is constructed based on the type detection prompts and the encoded information of the document image. The multimodal classification input is used as the input parameter of the multimodal visual language model for classification processing to obtain the document type and classification confidence. The multimodal visual language model is constructed based on an encoder-decoder network with an attention mechanism.

4. The method according to claim 3, characterized in that, The acquisition type detection prompt words include: In response to the prompt word type configuration instruction, receive uploaded type detection prompt words that match different document types; or, The system identifies target reference words in the input reference document image information and retrieves type detection prompt words that match the target reference words based on a preset reference mapping relationship. The preset reference mapping relationship is configured with type detection prompt words corresponding to different target reference words. The document types include ID card only, ID card in hand, household registration book, birth certificate, marriage certificate, relationship certificate, and other types.

5. The method according to claim 4, characterized in that, After classifying the multimodal classification input as input parameters of the multimodal visual language model to obtain the document type and classification confidence, the method further includes: If the classification confidence level is less than a preset threshold, then optical character recognition is performed on the document image information to obtain an optical recognition result.

6. The method according to claim 1, characterized in that, The step of extracting structured relationships from the structured text representation according to the document type to obtain the structured image information recognition result includes: Retrieve the relationship extraction strategy that matches the document type and create relationship extraction prompts; According to the relation extraction strategy and the relation extraction prompts, the structured text representation is extracted using a large language model to obtain the structured image information recognition result.

7. The method according to any one of claims 1-6, characterized in that, The method further includes: If the structured image information recognition result matches the extraction failure condition, then the entity information represented by the structured text is output; or, If the structured text indicates a failure to match the recognition condition, then optical character recognition is performed on the document image information to obtain the optical recognition result.

8. An image information recognition device based on a multimodal large model, characterized in that, include: The acquisition module is used to acquire image information of the document; The extraction module is used to extract the document type from the document image information based on a multimodal visual language model, and to perform a visual transformation based on the structured prompts of the document type to obtain a visual representation; The extraction module is used to fuse the visual representation and the document image information to obtain a structured text representation, and to extract the structured relationship of the structured text representation according to the document type to obtain the structured image information recognition result. The device further includes: a generation module, The generation module is used to generate structured prompt words that match the document type. The structured prompt words include field definition words, format requirement words, and relationship definition words. The extraction module is specifically used to perform visual transformation on the field definition words, the format requirement words and the relationship definition words based on a preset visual transformation component, so as to obtain the labeled image representation, the chart representation and the relationship graph representation; The extraction module is specifically used to construct a multimodal recognition input based on the visual representation, the structured prompts, and the document image information; to retrieve the multimodal visual language model, and to use the multimodal recognition input as the input parameters of the multimodal visual language model for recognition processing, thereby obtaining the structured text representation.

Images