Card picture structuring method based on multi-modal large model

Abstract

The invention discloses a card picture structuring method based on a multi-modal large model, and belongs to the technical field related to optical character recognition. The method is specially used for solving the problem of card information extraction in a general scene instead of the problem of card information extraction in a single type; the used image template is not used for template matching, but is used for improving the accuracy of structured extraction of the multi-modal large model; according to the invention, structured information extraction can be automatically carried out on identification card images of any type; according to the method, key node technologies such as a multi-modal large model, prompt word engineering, single-sample prompt and similarity comparison are fused, a proper template is selected as a sample for few-sample prompt, and the structural extraction capacity of the multi-modal large model is stimulated; and the template can be iteratively updated through a template quality score mode, so that the category of the template is enriched, the quality of the template is improved, and the structuring accuracy of the multi-modal large model is continuously improved.

Description

Technical Field

[0001] The present invention belongs to the technical field related to optical character recognition, and in particular relates to a card image structuring method based on a multimodal large model. Background Art

[0002] With the acceleration of the informationization process, the management and processing needs of ID cards in government agencies, enterprises and other fields have increased significantly. The traditional manual input and processing methods are inefficient and prone to errors. In order to improve the management efficiency of ID card information, automated card structured information collection and processing has become an urgent need.

[0003] Traditional card structuring technology relies on rules, such as predefined templates and field rules, to extract structured information from specific types of cards. This type of method is simple and easy to implement, but it has great limitations when dealing with cards with complex backgrounds, different fonts and layouts. Therefore, the industry currently prefers to use convolutional neural networks to extract structured information from card images. However, the training of deep learning models usually requires a large amount of labeled data, and the recognition of different types of cards requires separate research and development and training, and the generalization performance on any type of card is poor.

[0004] There are many types of card images in massive scenes, and the cost of structured data annotation required for deep learning model training is very high. How to automatically extract structured information from any type of card images in massive scenes at a low cost is a key step in card information collection and processing. Summary of the invention

[0005] The technical problem to be solved by the present invention is to provide a card image structuring method based on a multimodal large model in view of the shortcomings of the background technology, which can automatically extract structured information from any type of card image.

[0006] The present invention adopts the following technical solutions to solve the above technical problems:

[0007] A card image structuring method based on a multimodal large model, the specific steps are as follows:

[0008] Step 1: For different types of cards, manually annotate a small number of images I with categories C and structured information L; set the score S of each template to 1; and construct a template library T n =[(I1,C1,L1,S1),(I2,C2,L2),…,(I n ,C n ,L n ,S n )]; the upper limit of the number of templates in each category is K;

[0009] Step 2: Take out an image from the massive image data, denoted as Img;

[0010] Step 3: Use the multimodal large model and the prompt word project to construct the prompt word P for the card category recognition task cls , identify the category C of image Img img and card area loc img =[x1,y1,x2,y2,x3,y3,x4,y4]; for non-card images, just ignore them;

[0011] Step 4: According to the category C of the image img , use vector retrieval technology to retrieve the card template of this category from the template library

[0012] [t1=(I1,C1,L1),…,t i =(I i ,C i ,L i )], if the search is successful, jump to step 6; if the search fails, jump to step 5;

[0013] Step 5: Use the multimodal large model and combine it with the prompt word project to build a prompt word P that automatically generates a structured template template , after manual verification, it is saved in the template library and the template is recorded as t i =(I i ,C i ,L i ,S i );

[0014] Step 6: Use the image similarity algorithm to select a template with the highest similarity score with the image to be identified from the template library, and then build a template containing the image Img and the template t according to the single sample prompting technology. i The single sample hint P one-shot ;

[0015] Step 7: Set the single sample prompt P one-shot Input into a large multimodal model, the model outputs structured labels

[0016] L img =[(K1,V1,Conf1),(K2,V2,Conf2),…(K n ,V n ,Conf n )]; where K n and V n are the field values ​​corresponding to the structured field names, C n is the confidence of each character in the field value;

[0017] Step 8: According to Category Cimg , use the existing special card recognition technology to identify the image Img and obtain the structured label l img =[(k1,v1,conf1),(k2,v2,conf2),…(k m ,v m ,conf m )]; where k m and v m They are the field values ​​corresponding to the structured field names, conf m The confidence level corresponding to each character in the field value;

[0018] Step 9: Structured label L of the card obtained in steps 7 and 8 img and L img Carry out a merger;

[0019] Step 10: Output the final merged card structured label:

[0020] Label img =[(Key1,Value1),…(Key n ,Value n )], and the confidence list Confidence_Result = [Confidence1,…,Confidence n ];

[0021] Step 11: Using the template quality scoring method, retain the K images with the highest quality in each category as templates and store them in the template library.

[0022] As a further preferred solution of the card image structuring method based on a multimodal large model of the present invention, step 9 specifically includes the following steps:

[0023] Step 9-1: If L img If it is empty, no result merging is performed and the structured label is output directly:

[0024] Label img =[(Key1,Value1,Confidence1),…(Key n ,Value n ,Confidence n ];Key n =K n ,Value n =V n ,Confidence n =Conf n ;

[0025] Step 9-2: If l img If it is not empty, first match l according to the structured field name img and L img The structured field names and field values ​​of ; for unmatched field names, they are assumed to be l no-match =[(k1,v1,c1),…,(k i ,v i ,c i )]and

[0026] L no-match =[(K1,V1,Conf1),…,(K j ,V j ,C j )]; for l no-match The structured results in are retained as part of the final output result, recorded as Label img =[(Key1,Value1),…(Key i ,Value i )], where k i =Key i , v i =Value i ; For L no-match The content is directly discarded; the structured field names are matched through the vector matching algorithm, and the structured tags to be merged are

[0027] Label match =[(k p ,K p ,v p ,conf p ,V p ,Conf p ),…,(k q ,K q ,v q ,conf q ,V q ,Conf q )];

[0028] Step 9-3: Loabel the tags to be merged in step 9-2 match Merge; merge the field name and field value separately. When merging the field name, use the field name output by the special card recognition as the field name of the final structured label, that is, [Key i+1 ,…,Key n ], where Key i+1 =k p ,Key n =k q ;

[0029] Step 9-4: When merging field values, take merging one field value as an example. Assume that the field values ​​to be merged are v p =[s1,…,s n ],V p =[S1,…,S m ], and the corresponding confidence levels are

[0030] conf p =[c1,…,c n ], Conf p =[C1,…,C m ], where s n and S m A character in the field value, c n and C m They are the confidences corresponding to the characters (0≤(c n ,C m )≤1).

[0031] As a further preferred solution of the card image structuring method based on a multimodal large model of the present invention, in step 9-4, the following merging strategy is adopted, specifically: first calculate v p and V p The edit distance, then traverse the string v p =[s1,…,s n ], when s n This is a substitution error, and c n ≤C m When using S m Replace s n , C m Replace c n , otherwise keep s n and c n , the combined character is recorded as char n , the confidence after merging is co1; get the merged field value Value p =[char1,…,char n ] and the combined confidence p =[co1,…,co n ]; Perform the same operation on all the field values ​​to be merged to obtain the merged field value [Value i+1 ,…,Value n ] and the combined confidence i+1 ,…,Confidence n ].

[0032] As a further preferred solution of the card image structuring method based on a multimodal large model of the present invention, in step 11, the specific scoring rules are: in, The ratio of the card image area to the entire image, and its value is Among them, Area card Use the card area loc from step 3 img Find Area img is the size of the image Img; is the average value of the card structured label confidence; the value range is

[0033] Compared with the prior art, the present invention adopts the above technical solution and has the following technical effects:

[0034] 1. The present invention proposes a complete set of card image data structuring solutions, which can automatically extract structured information from any type of card image;

[0035] 2. The present invention integrates key node technologies such as multimodal large model, prompt word engineering, single sample prompt, similarity comparison, etc., selects appropriate templates as examples of few sample prompts, and stimulates the ability of multimodal large model structured extraction; and can iteratively update templates through template quality scores, enrich template categories and improve template quality, and continuously improve the accuracy of multimodal large model structuring;

[0036] 3. Different from other possible card information extraction methods, the present invention is specifically used to solve the card information extraction problem in general scenarios, rather than the card information extraction problem of a single type;

[0037] 4. Different from other possible card information extraction methods, the image template used in the present invention is not used for template matching, but is used to improve the accuracy of multi-modal large model structured extraction. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without paying any creative work.

[0039] Figure 1 It is a flow chart of a card image structuring method based on a multimodal large model of the present invention. DETAILED DESCRIPTION

[0040] The technical solution of the present invention is further described in detail below in conjunction with the accompanying drawings:

[0041] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention. The present invention is described in detail below based on the drawings and preferred embodiments, and the purpose and effect of the present invention will become clearer. It should be understood that the specific embodiments described here are only used to explain the present invention and are not used to limit the present invention.

[0042] Card structured information extraction refers to extracting structured text information from various types of card images. Generally speaking, it is necessary to first classify the image, then perform optical character recognition, then structure the recognition results, and finally output the structured results.

[0043] Currently, there are three main types of card structuring solutions: template matching-based solution, deep learning-based structured solution, and multimodal large model-based structured solution.

[0044] Template matching: Use pre-designed templates to locate the key areas of the card, use optical character recognition technology to recognize the text, and finally organize the recognition results into structured data. This solution relies on pre-defined templates and is difficult to handle cards with complex layouts.

[0045] A three-stage structured solution based on convolutional neural networks: the card image is processed by convolutional neural networks to identify the location of the text area, then optical character recognition technology is used for text recognition, and finally the structured information of the card is output by combining the text area information and character information. This solution usually requires a large number of structured annotated card samples, usually more than 1,000 card samples of each type, and cannot recognize any type of card.

[0046] End-to-end structuring solution based on multimodal large model: Extract the structured information of the card end-to-end based on the multimodal large model, and can be verified in combination with the existing model.

[0047] This patent belongs to an end-to-end structured solution based on a multimodal large model

[0048] Template matching: Relies on the quality of predefined templates and cannot handle complex layouts

[0049] The three-stage structured solution based on convolutional neural network: requires a large amount of data and cannot recognize unknown types of cards.

[0050] a. The present invention does not require the introduction of large amounts of labeling costs and can perform structured extraction on cards of unknown types.

[0051] b. Compared with other structured extraction schemes, the multimodal large model has used a large amount of graphic data on the Internet during pre-training, covering various types of card images, such as ID cards, driver's licenses, invoices, express delivery orders, etc. This scheme selects appropriate samples as single sample templates, and further stimulates the knowledge call mechanism of the large model through single sample prompts, realizing structured information extraction of any type of card data.

[0052] Different from other solutions for extracting structured information from ID cards, this solution uses key technologies such as multimodal large models, prompt word engineering, single sample prompts, vector retrieval and image similarity comparison to extract the structured information of images, and combines optical character recognition and similarity algorithms to verify and fuse the structured results, further improving the accuracy and making it more robust for structured extraction of ID cards in general scenarios.

[0053] The present invention is further described in detail with reference to the accompanying drawings.

[0054] A card image structuring method based on a multimodal large model, such as Figure 1 As shown, the specific steps include the following:

[0055] Step 1: For different types of cards, manually label a small number of images I with categories C and structured information L. The present invention believes that the manually produced templates have the highest credibility, so the score S of each template is set to 1. Constructing the template library T n =[(I1,C1,L1,S1),(I2,C2,L2),…,(I n ,C n ,L n ,S n )]. The upper limit of the number of templates in each category is K.

[0056] As a specific implementation example of the present invention, assume that the upper limit of the template library is 10 templates K of each category, and there are currently 10 templates, and the image I, category C and structured information L of one template are respectively:

[0057] I=ID card picture.jpg

[0058] C = ID card

[0059] L=[9Name, Zhang San ) , ( Nationality, Han ) ,(ID number,123456)]

[0060] S=1

[0061] Step 2: Take out an image from the massive image data, denoted as Img.

[0062] As a specific implementation example of the present invention, it is assumed that an ID card image is taken out, which is recorded as image Img.

[0063] Step 3: Use the multimodal large model and the prompt word project to construct the prompt word P for the card category recognition task cls , identify the category C of image Img img and card area loc img =[x1,y1,x2,y2,x3,y3,x4,y4]. For non-card images, just ignore them.

[0064] As a specific implementation example of the present invention, it is assumed that the prompt word P is constructed cls for:

[0065] Identify the following card images for the category: Image .

[0066] Assume that the image category C output by the multimodal large model img For ID card.

[0067] Step 4: According to the category C of the image img , use vector retrieval technology to retrieve the card template of this category from the template library [t1=(I1,C1,L1),…,t i =(I i ,C i ,L i )], if the search is successful, jump to step 6. If the search fails, jump to step 5.

[0068] As a specific implementation example of the present invention, the category C obtained in step 3 is used img Search in the template library. Assume that two card templates [t1=(I1,C1,L1),[t1=(I2,C2,L2)] are successfully retrieved. Specifically:

[0069] I1=ID card Figure 1 .jpg

[0070] C1=ID card

[0071] L1 = [(name, Zhang San), (ethnicity, Han), (ID number, 123456)]

[0072] S1=1

[0073] I2=ID card picture 2.jpg

[0074] C2=ID Card

[0075] I2 = [(name, Zhang Sanfeng), (nationality, Han), (ID number, 45678)]

[0076] S2=1

[0077] Step 5: Use the multimodal large model and combine it with the prompt word project to build a prompt word P that automatically generates a structured template template , after manual verification, it is saved in the template library and the template is recorded as t i =(I i ,C i ,L i ,S i ).

[0078] As a specific implementation example of the present invention, assuming that there is no ID card template in the library, a multimodal large model is first used to generate a template, and the prompt word P template To extract the structured information of the ID card image, the output format is [(field name 1, field value 1),…, (field name n, field value n)]. Image .

[0079] Assume that the image structured information and confidence output by the multimodal large model are:

[0080] [(name, Li Si, [0.9, 0.5]), (nationality, Han, [0.9, 0.8]], (ID number, 123456, [0.9, 0.8, 0.9, 0.8, 0.5, 0.8] ) ,(age,45,[0.8,0.8])]

[0081] Assuming that no correction is needed after manual verification, it is saved in the template library, that is, t1 = (I1, C1, L1, S1).

[0082] in:

[0083] I1=Img

[0084] C1=ID card

[0085] L1 = [(name, Li Si), (ethnicity, Han), (ID number, 123456), (age, 45)]

[0086] S1=1

[0087] Step 6: Use the image similarity algorithm to select a template with the highest similarity score with the image to be identified from the template library, and then build a template containing the image Img and the template t according to the single sample prompting technology. i The single sample hint P one-shot .

[0088] As a specific implementation example of the present invention, firstly, an image similarity algorithm is used to find the template image with the highest similarity from the template library as a template. According to the search result of step 4, there are two ID card image templates in the library.

[0089] [t1=(I1,C1,L1),[t1=(I2,C2,L2)], calculate the similarity between the template and the image respectively [sim1,sim2]. Assuming sim1>sim2, the selected template is t1=(I1,C1,L1). The constructed single sample prompt P one-shot To: According to the sample image and the corresponding structured template, extract the structured information of the ID card image, the output format is [(field name 1, field value 1), ..., (field name n, field value n)]. Example image: <ID card picture.jpg>, structured template: [(name, Zhang San), (ethnicity, Han), (ID card number, 123456)].

[0090] Step 7: Set the single sample prompt P one-shot Input into a large multimodal model, the model outputs structured labels

[0091] L img =[(K1,V1,Conf1),(K2,V2,Conf2),…(K n ,V n ,Conf n )]. Where K n and V n are the field values ​​corresponding to the structured field names, C n The confidence level for each character in the field value.

[0092] As a specific implementation example of the present invention, assume that the structured label L output by the multimodal large model is img is [(name, Li Si, [0.9, 0.5]), (nationality, Han, [0.9, 0.8]), (ID number, 123456, [0.9, 0.8, 0.9, 0.8, 0.5, 0.8]), (age, 45, [0.8, 0.8])]

[0093] Step 8: According to Category C img , use the existing special card recognition technology to identify the image Img and obtain the structured label l img =[(k1,v1,conf1),(k2,v2,conf2),…(k m ,v m ,conf m )]. Where k m and v mThey are the field values ​​corresponding to the structured field names, conf m The confidence level corresponding to each character in the field value.

[0094] As a specific implementation example of the present invention, assume that there is a special card recognition technology S = [ID card recognition, driver's license recognition, household registration book recognition, ...], use the special ability to recognize the image Img, and obtain the structured label l img for:

[0095] [(Name, Li Si, [0.8, 0.8]), (Nationality, Han, [0.9]], (ID Number, 323458, [1.0, 0.8, 0.9, 0.8, 0.9, 0.2]), (Gender, Male, [0.9])]

[0096] Step 9: Structured label L of the card obtained in steps 7 and 8 img and l img To merge, the specific steps are:

[0097] Step 9-1: If l img If it is empty, no result merging is performed and the structured tags are directly output.

[0098] Label img =[(Key1,Value1,Confidence1),…(Key n ,Value n ,Confidence n )]. Key n =K n ,Value n =V n ,Confidence n =Conf n .

[0099] As a specific implementation case of the present invention, due to l img Not empty, so a merge is required.

[0100] Step 9-2: If l img If it is not empty, first match l according to the structured field name img and L img The structured field names and field values ​​of . For unmatched field names, they are assumed to be l no-match =[(k1,v1,c1),…,(k i ,v i ,c i )] and L no-match =[(K1,V1,Conf1),…,(K j ,Vj ,C j )]. no-match The structured results in are retained as part of the final output result, recorded as Label img =[(Key1,Value1),…(Key i ,Value i )], where k i =KLey i , v i =Value i For L no-match The structured field names are matched by the vector matching algorithm, and the structured label to be merged is Label. match =[(k p ,K p ,v p ,conf p ,V p ,Conf p ),…,(k q ,K q ,v q ,conf q ,V q ,Conf q )].

[0101] As a specific implementation example of the present invention, the unmatched field names are respectively no-match =[(sex, male, [0.9])], L no-match =[(age,45,[0.8,0.8])], no-match Add the result to Label img , and discard L no-match The result is Label img =[(sex, male, [0.9]].

[0102] Through keyword matching, the structured tags to be merged are: [(name, name, Li Si, [0.8, 0.8], Li Si, [0.9, 0.5]), (ethnicity, ethnicity, Han, [0.9], Han, [0.9, 0.8]), (ID number, ID number, 323458, [1.0, 0.8, 0.9, 0.8, 0.9, 0.2], 123456, [0.9, 0.8, 0.9, 0.8, 0.5, 0.8])].

[0103] Step 9-3: For the label to be merged in step 9-2 matchMerge. Merge the field names and field values ​​separately. When merging the field names, use the field names output by the special card recognition as the field names of the final structured labels, that is, [Key i+1 ,…,Key n ], where Key i+1 =k p ,Key n =k q .

[0104] As a specific implementation example of the present invention, the field names are merged, and the field name of the final structured label is: [name, ethnicity, ID number].

[0105] Step 9-4: When merging field values, take merging one field value as an example. Assume that the field values ​​to be merged are v p =[s1,…,s n ],V p =[S1,…,S m ], and the corresponding confidence levels are conf p =[c1,…,c n ],

[0106] Conf p =[C1,…,C m ], where s n and S m A character in the field value, c n and C m They are the confidences corresponding to the characters (0≤(c n ,C m )≤1). The following merging strategy is adopted, specifically: first calculate v p and V p The edit distance, then traverse the string v p =[s1,…,s n ], when s n This is a substitution error, and c n ≤C m When using S m Replace s n , C m Replace c n , otherwise keep s n and c n , the combined character is recorded as char n , the confidence after merging is co1. Get the merged field value:

[0107] Valeu p =[char1,…,char n ] and the combined confidencep =[co1,…,co n Perform the same operation on all the field values ​​to be merged to obtain the merged field value [Value i+1 ,…,Value n ] and the combined confidence i+1 ,…,Confidence n ].

[0108] As a specific implementation case of the present invention, when merging the name field, v1=Li Si and V1=Li Si have the same edit distance, after merging, Value1=Li Si, and confidence1=[0.8,0.8]; when merging the ethnic field, v2=Han and V2=Han, there is a deletion error ("", ethnicity), so after merging, Value2=Han, and confidence2=[0.9]; when merging the ID number field, v3=123456, V3=3 23458, there are two substitution errors (3,1), (8,6), for the first substitution error, the confidence corresponding to character 3 is 1.0, which is greater than the confidence corresponding to character 1 0.9, so character 3 is retained. For the second substitution error, the confidence of character 8 is 0.2, which is less than the confidence of character 6 0.8, so character 6 is retained. After merging, Value3 = 323456, and confidence3 = [1.0, 0.8, 0.9, 0.8, 0.9, 0.8]. Finally, all the merged field values ​​are [Li Si, Han, 323456], and the confidence is [0.8, 0.8, 0.9, 1.0, 0.8, 0.9, 0.8, 0.9, 0.8].

[0109] Step 10: Output the final structured label of the merged card img =[(Key1,Value1),…(Key n ,Value n )], and the confidence list

[0110] Confidence Result =[Confidence1,…,Confidence n ]

[0111] As a specific implementation example of the present invention, the structured label Label of the final card img =[(gender, male),(name, Li Si),(ethnicity, Han),(ID number, 323456)], confidence list

[0112] Confidence Result=[0.9,0.8,0.8,0.9,1.0,0.8,0.9,0.8,0.9,0.8,0.9,0.8]

[0113] Step 11: Using the template quality scoring method, retain the K highest quality images in each category as templates and store them in the template library. The specific scoring rules are: in The ratio of the card image area to the entire image, and its value is Area card You can use the card area loc in step 3 img Find,Area img is the size of the image Img. It is the average value of the card structured label confidence. The value range is

[0114] As a specific implementation example of the present invention, assuming that K is 10, the card coordinates are

[0115] loc img =[10,10,110,10,110,110,10,110], then use the quadrilateral area calculation formula to calculate Area card =10000, the image area is the product of the length and width of the image, assuming it is 15000, then =10000 / 15000=0.67. Based on the confidence list in step 10, the average confidence value is 0.86, so the template quality score is Within the K range, it is stored in the template library.

[0116] Those skilled in the art can understand that the above are only preferred examples of the invention and are not intended to limit the invention. Although the invention is described in detail with reference to the above examples, those skilled in the art can still modify the technical solutions recorded in the above examples or replace some of the technical features with equivalents. Any modification, equivalent replacement, etc. made within the spirit and principle of the invention should be included in the protection scope of the invention. All technical features in this embodiment can be freely combined according to actual needs.

[0117] Finally, it should be noted that the above is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, it is still possible for those skilled in the art to modify the technical solutions described in the aforementioned embodiments or to make equivalent substitutions for some of the technical features therein. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A card image structuring method based on a multimodal large model, characterized by: The specific steps are as follows: Step 1: For different types of cards, manually annotate a small number of images I with categories C and structured information L; set the score S of each template to 1; and construct a template library T n =[(I1,C1,L1,S1),(I2,C2,L2),...,(I n , C n , L n , S n )]; the upper limit of the number of templates in each category is K; Step 2: Take out an image from the massive image data, denoted as Img; Step 3: Use the multimodal large model and the prompt word project to construct the prompt word P for the card category recognition task cls , identify the category C of image Img img and card area loc img =[x1, y1, x2, y2, x3, y3, x4, y4]; for non-card images, just ignore them; Step 4: According to the category C of the image img , use vector retrieval technology to retrieve the card template of this category from the template library [t1=(I1, C1, L1),...,t i =(I i , C i , L i )], if the search is successful, jump to step 6; if the search fails, jump to step 5; Step 5: Use the multimodal large model and combine it with the prompt word project to build a prompt word P that automatically generates a structured template template , after manual verification, it is saved in the template library and the template is recorded as t i =(I i , C i , L i , S i ); Step 6: Use the image similarity algorithm to select a template with the highest similarity score with the image to be identified from the template library, and then build a template containing the image Img and the template t according to the single sample prompting technology. i The single sample hint P one-shot ; Step 7: Set the single sample prompt P one-shot Input into a multimodal large model, the model outputs structured labels L img =[(K1,y1,Conf1),(K2,V2,Conf2),...(K n , V n , Conf n )]; where K n and V n are the field values ​​corresponding to the structured field names, C n is the confidence of each character in the field value; Step 8: According to Category C img , use the existing special card recognition technology to identify the image Img and obtain the structured label l img =[(k1,v1,conf1),(k2,v2,conf2),...(k m , v m ,conf m )]; where k m and v m They are the field values ​​corresponding to the structured field names, conf m The confidence level corresponding to each character in the field value; Step 9: Structured label L of the card obtained in steps 7 and 8 img and l img Carry out a merger; Step 10: Output the final merged card structured label: Label img =[(Key1, Value1), ...(Key n ,Value n )], and the confidence list Confidence_Result=[Confidence1,…,Confidence n ]; Step 11: Using the template quality scoring method, retain the K images with the highest quality in each category as templates and store them in the template library.

2. The card image structuring method based on a multimodal large model according to claim 1, characterized in that: The step 9 specifically comprises the following steps: Step 9-1: If l img If it is empty, no result merging is performed and the structured tags are directly output. Label img =[(Key1,Value1,Confidence1),...(Key n ,Value n ,Confidence n )];Key n =K n ,Value n =V n ,Confidence n =Conf n ; Step 9-2: If l img If it is not empty, first match l according to the structured field name img and L img The structured field names and field values ​​of ; for unmatched field names, they are assumed to be l no-match =[(k1, v1, c1), ..., (k i , v i , c i )]and L no-match =[(K1,V1,Conf1),...,(K j , V j , C j )]; for l no-match The structured results in are retained as part of the final output result, recorded as Label img =[(Key1, Value1), ...(Key i ,Value i )], where k i =Key i , v i =Value i ; For L no-match The content is directly discarded; the structured field name is matched through the vector matching algorithm, and the structured label to be merged is Label match =[(k p , K p , v p ,conf p , V p , Conf p ),...,(k q , K q , v q ,conf q , V q , Conf q )]; Step 9-3: For the label to be merged in step 9-2 match Merge; merge the field name and field value separately. When merging the field name, use the field name output by the special card recognition as the field name of the final structured label, that is, [Key i+1 ,...,Key n ], where Key i+1 =k p ,Key n =k q ; Step 9-4: When merging field values, take merging one field value as an example. Assume that the field values ​​to be merged are v p =[s1, ..., s n ],V p =[S1, ..., S m ], and the corresponding confidence levels are conf p =[c1, ..., c n ], Conf p =[C1, ..., C m ], where s n and S m A character in the field value, c n and C m They are the confidences corresponding to the characters (0≤(c n , C m )≤1).

3. The card image structuring method based on a multimodal large model according to claim 2, characterized in that: In step 9-4, the following merging strategy is adopted, specifically: first calculate v p and V p The edit distance, then traverse the string v p =[s1, ..., s n ], when s n This is a substitution error, and c n ≤C m When using S m Replace s n , C m Replace c n , otherwise keep s n and c n , the combined character is recorded as char n , the combined confidence is co1; Get the merged field value Value p =[char1, ..., char n ] and the combined confidence p =[co1, ..., co n ]; Perform the same operation on all the field values ​​to be merged to obtain the merged field value [Value i+1 , ..., Value n ] and the combined confidence i+1 ,...,Confidence n ].

4. The card image structuring method based on a multimodal large model according to claim 3 is characterized by: In step 11, the specific scoring rules are: in, The ratio of the card image area to the entire image, and its value is Among them, Area card Use the card area loc from step 3 img Find Area img is the size of the image Img; is the average value of the card structured label confidence; the value range is

Images