Form highlighting method and device based on large model, computer equipment and storage medium

Abstract

The invention provides a form highlighting method and device based on a large model, computer equipment and a storage medium. According to the method, a to-be-recognized form screenshot is input into an image recognition model to obtain an image recognition result, then a unique identifier is added, a first mapping relation is determined, and a text character string and an identifier character string are further constructed. And then instructing the large language model to generate result information required by a highlighting function based on the information, and finally, responding to a highlighting request of a user based on each piece of result information to carry out rapid positioning and highlighting. According to the scheme, a complete process from the form image to structural information extraction and visual display is realized, and the efficiency, accuracy and convenience of form highlighting processing are improved by effectively depending on a large language model.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a form highlighting method, device, computer equipment and storage medium based on a large model. Background Art

[0002] Traditional paper record books are difficult to form good data management due to the large amount of paper form data. With the development of information technology, using electronic forms to record data has become the mainstream. However, due to the large amount of data in the form, it is often difficult to locate specific data, and the highlighting function can solve this problem. However, the highlighting function requires the construction of data with a specific data structure as a reference. These data contain information such as the key, value, and location of each field in the form. With the development of artificial intelligence. In traditional technology, there is a method of inputting raw data into a large language model, hoping to rely on the semantic understanding ability of the large language model to build such a solution with specific data, but the processing results obtained by the traditional solution are not ideal. Summary of the invention

[0003] The purpose of this application is to solve at least one of the above-mentioned technical defects, and in particular to provide a solution that can stably realize field value highlighting based on a large language model.

[0004] In a first aspect, the present application provides a form highlighting method based on a large model, comprising:

[0005] For any form to be recognized, the image of the form to be recognized is input into the image recognition model to obtain an image recognition result; the image recognition result includes recognition information corresponding to each text block, and the recognition information includes text content and text box coordinates;

[0006] Adding a corresponding unique identifier to each piece of identification information, and determining a first mapping relationship; the first mapping relationship reflects the corresponding relationship between the unique identifier and the identification information;

[0007] According to the coordinates of the text box, the corresponding text content and the unique identifier are sequentially connected to obtain a text string and an identifier string, so that the text string and the identifier string are the same as the form structure to be identified;

[0008] Generate a prompt word according to the first mapping relationship, the text string and the identification string, and input the prompt word into the large language model; the prompt word is used to instruct the large language model to find the field value corresponding to each preset field from the text string, and find all unique identifiers corresponding to the field value from the identification string according to the first mapping relationship, and then fill the found field value and its corresponding unique identifier according to the set output format to obtain result information corresponding to each preset field, and the preset field includes all fields in the form to be identified;

[0009] In response to a highlight request for taking a value of any field of a target form, the field targeted by the highlight request is determined as a target field, target result information matching the target field is found from the result information, and target identification information matching the unique identifier carried by the target result information is found from the identification information;

[0010] Highlight the target form according to the text box coordinates carried by the target identification information.

[0011] In one embodiment, according to the coordinates of the text box, the corresponding text content and the unique identifier are sequentially connected to obtain a text string and an identifier string, including:

[0012] According to the coordinates of the text box, the corresponding identification information is divided into lines;

[0013] The text content and the unique identifier in the identification information belonging to the same row are sequentially connected to obtain a plurality of text row character strings and identification row character strings;

[0014] The text line character strings and the identification line character strings are connected in sequence using line feed characters to obtain text character strings and identification character strings.

[0015] In one embodiment, the text content and the unique identifier in the identification information belonging to the same row are sequentially connected, including:

[0016] For the identification information in the same row, the left-right relationship is determined based on the corresponding text box coordinates;

[0017] According to the left-right relationship, the text content and the unique identifier in the identification information of the same row are connected respectively in order from left to right to obtain a plurality of text row character strings and identification row character strings.

[0018] In one embodiment, the text content and the unique identifier in the identification information belonging to the same row are sequentially connected, including:

[0019] Determine the upper and lower relationships of each text line string and each logo line string respectively;

[0020] According to the up-down relationship, each text line character string and each identification line character string are connected using line break characters in order from top to bottom to obtain a text character string and an identification character string.

[0021] In one embodiment, the form highlighting method based on the large model further includes:

[0022] The page number identifier of the form to be identified is determined, and the page number identifier is added to each identification information corresponding to the form to be identified.

[0023] In one embodiment, finding the target identification information that matches the unique identifier carried by the target result information from the identification information includes:

[0024] Determine a unique page number identifier according to the mode of the page number identifiers carried by each target identification information;

[0025] Remove the page number identifiers in the target identification information that do not match the unique page number identifier.

[0026] In one embodiment, highlighting the target form according to the text box coordinates carried by the target identification information includes:

[0027] Determine whether the page number corresponding to the current canvas matches the unique page number identifier;

[0028] If so, the target form is highlighted according to the text box coordinates carried by the target identification information; otherwise, the page is cut according to the unique page number identifier, and then the target form is highlighted according to the text box coordinates carried by the target identification information.

[0029] In one embodiment, the form highlighting method based on the large model further includes:

[0030] Display the target form in the first area;

[0031] Determine in the second area a one-to-one corresponding sub-area for each preset field of the target form;

[0032] For any preset field, the name of the preset field is displayed in the corresponding sub-area, and the result information matching the preset field is found from the result information. Then, based on the field value carried in the found result information, the value selection control of the preset field is displayed; the value selection control is used to accept highlight requests.

[0033] In a second aspect, the present application provides a form highlighting device based on a large model, comprising:

[0034] The recognition module is used for inputting the image of any form to be recognized into the image recognition model to obtain an image recognition result; the image recognition result includes recognition information corresponding to each text block, and the recognition information includes text content and text box coordinates;

[0035] A mapping module, used to add a corresponding unique identifier to each piece of identification information and determine a first mapping relationship; the first mapping relationship reflects the corresponding relationship between the unique identifier and the identification information;

[0036] A combination module, used for sequentially connecting the corresponding text content and the unique identifier according to the coordinates of the text box to obtain a text string and an identifier string, so that the text string and the identifier string are the same as the form structure to be identified;

[0037] A conversion module is used to generate a prompt word according to the first mapping relationship, the text string and the identification string, and input the prompt word into the large language model; the prompt word is used to instruct the large language model to find the field value corresponding to each preset field from the text string, and find all unique identifiers corresponding to the field value from the identification string according to the first mapping relationship, and then fill the found field value and its corresponding unique identifier according to the set output format to obtain result information corresponding to each preset field, and the preset field includes all fields in the form to be identified;

[0038] A response module, used to respond to a highlight request for taking a value of any field of a target form, determine the field targeted by the highlight request as a target field, find target result information matching the target field from the result information, and find target identification information matching the unique identifier carried by the target result information from the identification information;

[0039] The highlight module is used to highlight the image of the target form according to the text box coordinates carried by the target recognition information.

[0040] In a third aspect, the present application provides a computer device comprising one or more processors and a memory, wherein the memory stores computer-readable instructions, and when the computer-readable instructions are executed by the one or more processors, the steps of the large model-based form highlighting method in any of the above-mentioned embodiments are executed.

[0041] In a fourth aspect, the present application provides a storage medium storing computer-readable instructions. When the computer-readable instructions are executed by one or more processors, the one or more processors execute the steps of the large model-based form highlighting method in any of the above-mentioned embodiments.

[0042] It can be seen from the above technical solutions that the embodiments of the present application have the following advantages:

[0043] The form highlighting method based on the large model in this application obtains the image recognition result by inputting the screenshot of the form to be identified into the image recognition model, which can efficiently and accurately extract the text content and key information such as the coordinates of the text box in the form, laying the foundation for subsequent processing, and getting rid of the tediousness and error-proneness of manual information entry. Then, a unique identifier is added and the first mapping relationship is determined, and a text string and an identification string are constructed, so that many complex identification information have a clear association and order, which is convenient for the large language model to generate the result information required for the highlighting function according to the requirements, so that the large language model can more accurately find the field values ​​corresponding to each preset field from the text string, further structure the form information, and improve the efficiency and accuracy of information extraction. And the link of responding to the highlight request and performing the corresponding operation can be quickly located and highlighted based on the output results of the large language model. In general, the scheme realizes the complete process from form image to structured information extraction and visual display, effectively improving the efficiency, accuracy and convenience of form processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0045] Figure 1 A schematic diagram of a flow chart of a form highlighting method based on a large model provided in one embodiment of the present application;

[0046] Figure 2 A schematic diagram of a process of concatenating a text string and an identification string in one embodiment of the present application;

[0047] Figure 3 A schematic diagram showing a functional flow in one embodiment of the present application;

[0048] Figure 4 An internal structure diagram of a computer device provided for one embodiment of the present application. DETAILED DESCRIPTION

[0049] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0050] This application provides a form highlighting method based on a large model, please refer to Figure 1 , including steps S102 to S112.

[0051] S102: for any form to be recognized, input the image of the form to be recognized into the image recognition model to obtain an image recognition result, which includes recognition information corresponding to each text block, and the recognition information includes text content and text box coordinates.

[0052] It can be understood that steps S102 to S108 in the present application are the data preparation stage before highlighting is performed. The form to be identified refers to various form files from which information needs to be extracted, such as the company's financial reimbursement form, various business application forms, shipping bills, etc., which are presented in the form of a form and contain a lot of text information. The form to be identified can be a paper form or an electronic form. The image recognition model is a model built based on machine learning or deep learning algorithms. After being trained with a large amount of annotated image data, it can analyze and process the input image. The image recognition model in this embodiment can automatically divide the image into blocks, generate a rectangular box that selects the entire text block, obtain the coordinates of the four vertices of the rectangular box (i.e., the text box coordinates), and can also identify the content (text content) in each text block.

[0053] The image recognition model can choose an open source model with the above functions, such as Tesseract and other models specifically for text recognition, or build a model based on an open source image recognition framework (such as TensorFlow, PyTorch, etc.), and then use the form with the same structure as the form to be recognized as a data set for secondary training. If there are multiple structures of the form to be recognized, add different types of forms to the data set to enhance the recognition ability of the model. For the image of the form to be recognized, the clarity and integrity of the image should be improved as much as possible. If necessary, image preprocessing can be performed, such as resizing, enhancing contrast, etc., so that the model can better recognize the content. Then input the processed image into the selected image recognition model. After internal calculations, the model will output recognition information corresponding to each text block.

[0054] S104, adding a corresponding unique identifier to each piece of identification information, and determining a first mapping relationship. The first mapping relationship reflects the corresponding relationship between the unique identifier and the identification information.

[0055] It can be understood that the unique identifier is a unique number or mark given to each identification information to distinguish different identification information so that a specific one can be accurately located and tracked when processing a large number of identification information later. It can be in the form of a digital number, a combination of letters and numbers, etc. The identification information before adding the unique identifier can be:

[0056]

[0057] The identification information after adding the unique identifier can be:

[0058]

[0059] Obtaining the first mapping relationship is to record the correspondence between the unique identifier and the identification information. In the implementation process, the first thing to do is to determine the generation rule of the unique identifier. You can use numbers in sequence, for example, starting from 1, each new identification information is assigned a number that is 1 larger than the previous one as a unique identifier; you can also use information such as the form type and the area where the text block is located to generate a more meaningful unique identifier. For example, for a financial reimbursement form, the text block in the expense details area can be in the format of "FB_FMXM_001" ("FB" means financial reimbursement form, "FMXM" means expense details, and "001" means the first text block in the area). After determining the generation rule, for each piece of identification information obtained from image recognition, assign it a unique identifier according to the rule. Then create a data structure to store the first mapping relationship. Common data structures such as dictionaries and hash tables can be used to store the text content as the key and the corresponding unique identifier as the value, so that a clear first mapping relationship is established.

[0060] S106, according to the coordinates of the text box, the corresponding text content and the unique identifier are sequentially connected to obtain a text string and an identifier string, so that the text string and the identifier string are the same as the form structure to be identified.

[0061] It can be understood that the text string is a string formed by splicing all the text content in the form according to its original structure. It reflects the arrangement of the text information on the form and is converted into a continuous form of pure text. Because the form itself has a certain layout and order, different text blocks have different functions in different positions. The order and position relationship of them can be known through the text box coordinates. Connecting the text content into a text string in this order is equivalent to integrating the text information of the form together, but still carrying the structural information of the expression to be recognized, providing more information for the subsequent large language model to understand the content of the form to be recognized. From a formal point of view, the identification string is to replace the text content in the text string with the corresponding unique identifier. Part of the text string can be: "Name: Zhang San / n Department: R&D Department / n", where the unique identifier corresponding to "Name" is 1, the unique identifier corresponding to the first ":" is 2, and the unique identifier corresponding to "Zhang San" is 3, / n represents a line break, the unique identifier corresponding to "Department" is 4, the unique identifier corresponding to the second ":" is 5, and the unique identifier corresponding to "R&D Department" is 6. The corresponding identification string for this example can be: "123 / n456 / n".

[0062] S108, generating a prompt word according to the first mapping relationship, the text string and the identification string, and inputting the prompt word into the large language model. The prompt word is used to instruct the large language model to find the field value corresponding to each preset field from the text string, and find all unique identifiers corresponding to the field value from the identification string according to the first mapping relationship, and then fill the found field value and its corresponding unique identifier according to the set output format to obtain result information corresponding to each preset field, and the preset field includes all fields in the form to be identified.

[0063] It can be understood that the first mapping relationship, text string and identification string obtained in steps S102 to S106 are the original data extracted from the form to be identified, and the result information is the data relied on to realize the highlighting function. The large language model in this embodiment is an artificial intelligence model with powerful language understanding and generation capabilities trained based on massive text data. It can understand the task requirements according to the input prompt words, and then use its learned language knowledge and logical reasoning ability to generate corresponding results. It can be an open source large language model or a self-trained private model. The preset fields include all fields in the form to be identified, and each preset field will generate one-to-one corresponding result information in this embodiment.

[0064] The prompt word in this embodiment is an instruction or guiding information described in natural language, which is used to inform the large language model what kind of tasks need to be performed and what kind of requirements are followed to process the input data. Here, it is constructed based on the mapping relationship, text string and other contents obtained in the previous steps. In other words, the preset fields need to be listed in the prompt word, and the task requirement is to accurately extract the field values ​​corresponding to the preset fields from the text string. Since the text string retains both the structural information of the form to be identified and the semantic information, the semantic understanding ability of the large language model can determine the position of each preset field in the text string, and combined with the conventional understanding of the form structure, search for the corresponding field value near the position. For example, the text string will orderly cover content such as "Product Name: Smartphone", "Inventory Quantity: 5", "Customer Name: Li Si", etc. This allows the large language model to use its semantic understanding and analysis capabilities to locate and extract the field values ​​corresponding to the preset fields we need. Just like when you want to obtain the field value of the preset field "Inventory Quantity", you only need to search for related expressions in this complete text string based on language logic, and you can accurately find "Product Quantity: 5 units", thereby extracting the key field value "5", providing the most basic text material support for the generation of the final result information.

[0065] In addition, it is also necessary to clarify that the task requirement is to further find all the unique identifiers corresponding to the field values ​​from the identification string according to the first mapping relationship (some field values ​​are composed of the contents of two or more text blocks), and then fill the found field values ​​and their corresponding unique identifiers according to the set output format. The identification string connects the unique identifiers corresponding to the text content in the order that is completely consistent with the text content. It establishes a strict one-to-one correspondence with the text string. Although the first mapping relationship can link the field value with the unique identifier, the field values ​​of different fields in the form may be the same or similar, which leads to errors in the final output result information, and ultimately affects the effect of the highlighting function. The identification string can be accurately distinguished based on the structural information it reflects because it corresponds to the order of the text string and fits the form structure. For example, the above example also includes the field "sales quantity: 5", which has the same field value as "inventory quantity" but in a different position. The large language model can only determine the position of the field value of the inventory quantity in the text string based on the structural information of each field value reflected in the identification string, and the position of the field value in the identification string must match the position of the text string, so that the two identical field values ​​can be distinguished.

[0066] In some embodiments, in order to enhance the ability of the large language model to understand forms in specific fields, the role played by the large language model can also be indicated in the prompt word, and the role can be a senior person in the field corresponding to the form to be identified. In some embodiments, the structure of some forms to be identified is different from that of regular forms, or the requirements for extracting data by some users are different from the regular ones. These special requirements or precautions also need to be made clear in the prompt word. In some embodiments, the output format can be set to data with a data structure of json, which can indicate which elements need to be included, what the key of each element is, and what the value that needs to be filled in for each key is, for example:

[0067]

[0068] Among them, "field name" is filled with the name of the corresponding preset field, "value" is filled with the field value corresponding to the preset field, and "id" is filled with the unique identifier set corresponding to all text blocks that make up the field value.

[0069] S110, in response to a highlight request for taking a value of any field of the target form, the field targeted by the highlight request is determined as the target field, target result information matching the target field is found from the result information, and target identification information matching the unique identifier carried by the target result information is found from the identification information.

[0070] It can be understood that step S110 and step S112 are the execution stages of the highlight function. The target form is the form that the user needs to highlight. Each form that can be selected as the target form has been processed from step S102 to step S108, and multiple result information and identification information corresponding to it have been obtained. The user triggers the highlight function through a highlight request. Usually, when the user views the form or displays the form information, he hopes to highlight the content corresponding to a specific field for easy intuitive viewing and distinction. When the highlight request is received, the target field is first determined, and then the result information obtained by the previous large language model processing is used to locate the result information matching the target field, that is, the target result information is obtained. Specifically, the mapping relationship between each result information and the corresponding preset field can be established based on the field name, that is, each result information can be matched according to the name of the target field. Each unique identifier is matched with one of the many identification information. The target result information carries all the unique identifiers of the text box corresponding to the field value of the target field, and matches in the identification information according to these unique identifiers. These matched identification information are the target identification information.

[0071] The whole process starts from the user request and gradually locates the specific location information of the content to be highlighted through data association, so as to highlight the key information of the form.

[0072] S112, highlighting the target form according to the text box coordinates carried by the target identification information.

[0073] It can be understood that the form, as a whole image or electronic document presentation form, has corresponding position coordinates for each element, and the text box coordinates specify the specific range of the content we want to highlight. By obtaining these coordinates, the relevant software or system can know the specific position of the form to implement the highlight operation. Whether it is by modifying the color attributes of the area, adding a border of a specific style, etc., it is based on the range defined by the coordinates. The purpose is to change the visual presentation effect of the area so that it can quickly attract the user's attention in the entire form, and facilitate users to view and focus on the field values ​​that need to be paid attention to. The text box coordinates carried by the target identification information correspond to the field value of the target field. After finding the matching target identification information, the area defined by the text box coordinates carried by the target identification information can be connected to determine the area that needs to be highlighted. Highlighting these areas can highlight the field value of the target field.

[0074] In this embodiment, the form highlighting method based on the large model obtains the image recognition result by inputting the screenshot of the form to be identified into the image recognition model, which can efficiently and accurately extract the text content and key information such as the coordinates of the text box in the form, laying the foundation for subsequent processing, and getting rid of the tediousness and error-proneness of manual information entry. Next, a unique identifier is added and the first mapping relationship is determined, and a text string and an identifier string are constructed, so that a large number of complex identification information have a clear association and order, which is convenient for the large language model to generate the result information required for the highlighting function according to the requirements, so that the large language model can more accurately find the field values ​​corresponding to each preset field from the text string, further structure the form information, and improve the efficiency and accuracy of information extraction. And the link of responding to the highlight request and performing the corresponding operation can be quickly located and highlighted based on the output results of the large language model. In general, the scheme realizes the complete process from form image to structured information extraction and visual display, effectively improving the efficiency, accuracy and convenience of form processing.

[0075] In one embodiment, the corresponding text content and the unique identifier are sequentially connected according to the text box coordinates to obtain a text string and an identifier string. Figure 2 , including steps S202 to S206.

[0076] S202, dividing the corresponding identification information into lines according to the text box coordinates.

[0077] It can be understood that as a form of information carrier with a structured layout, its content has certain rules in presentation, and is often arranged in rows, with different rows carrying different information categories or different parts of the same type of information. Using text box coordinates to perform line separation operations is to follow the inherent layout logic of the form, and use the position relationship reflected by the coordinates to group the identification information corresponding to text boxes in the same horizontal position (that is, in the same horizontal row visually) together. The purpose of this is to restore the original structural order of the form, so that subsequent operations such as integration and connection of text content can conform to the logic of the form itself, facilitate further extraction and processing of information, avoid information confusion and mismatch, and prepare for the construction of text strings and identification strings that accurately reflect the content of the form.

[0078] Specifically, these identification information are sorted according to the ordinate of the coordinate (that is, the coordinate value in the vertical direction). The smaller the ordinate value, the higher the text box is in the image, so as to simulate the reading order of the form from top to bottom. Then, the sorted identification information is traversed, and a reasonable ordinate difference threshold is set, such as 5 pixels (which can be adjusted according to the actual situation of the form and the size of the text box). When the ordinate difference between two adjacent identification information is less than this threshold, they are classified as identification information in the same row. In this way, all identification information is traversed in turn, and they are divided into different rows according to the above rules to complete the row division operation.

[0079] S204, sequentially connect the text content and the unique identifier in the identification information belonging to the same row to obtain a plurality of text row character strings and identifier row character strings.

[0080] It can be understood that after completing the line division operation of the identification information, each line of identification information represents the content combination in the same horizontal line on the form. The text content in the identification information in the same line can be connected in sequence to obtain the text line string corresponding to the line, and the unique identifier in the identification information in the same line can be connected in sequence to obtain the identification line string corresponding to the line. The requirement for sequential connection here is to maintain the structure of the form to be identified in the line, specifically, the order of appearance of each field and field value should be consistent. For example, in a row of the employee information table, the first is the "name" text box content, followed by the "department" text box content, etc., and the text content is connected in sequence to form a text line string, which can accurately reflect the text expression order of this row; similarly, the unique identifier is connected in sequence to form an identification line string, so that a one-to-one correspondence is established between the text content and the identifier, which is convenient for subsequent information search, matching and association operations whether in the same row or at the entire form level.

[0081] The sequential connection in step S204 can be for the identification information of the same row, and the left-right relationship is first determined according to the corresponding text box coordinates. Then, according to the left-right relationship, the text content and the unique identifier in the identification information of the same row are connected respectively in order from left to right to obtain multiple text row character strings and identification row character strings. Specifically, the identification information that has been divided into rows is processed line by line. For each row, two empty string variables are first created, one for storing the text row character string to be constructed, and the other for storing the identification row character string. Then, based on the horizontal coordinates of the text box coordinates of the identification information in this row, the left-right relationship is determined in order from small to large. The smaller the horizontal coordinate, the closer to the left. Since the text box coordinates include the coordinates of four vertices, the horizontal coordinate here can be to select the vertices of the same orientation for comparison, or it can be to calculate the center coordinates based on these four coordinates and then compare them. This embodiment does not limit this. After determining the left-right relationship, the text content in each identification information can be taken out in turn in order from left to right, and added to the text row character string variable. In addition, in the same order, the unique identifier corresponding to each identification information is taken out and added to the identification row character string variable.

[0082] S206, using line feed characters to sequentially connect the text line character strings and the identification line character strings to obtain a text character string and an identification character string.

[0083] It can be understood that the form itself is an information carrier with a multi-line structure. Previously, the information has been integrated at the level of each line by dividing the lines and constructing the text line string and the identification line string separately. Using the line feed character to connect them in sequence is to restore the complete multi-line structure of the form, from the combination of information at the line level to the information integration at the level of the entire form. By inserting the line feed character, the text line strings can be connected and presented according to the original multi-line layout of the form when displayed or processed later. Similarly, the identification string is also connected in the same order and manner, maintaining a strict correspondence with the text string. In this way, the entire text string and identification string completely replicate the text content and structural logic of the form.

[0084] The sequential connection in step S206 can be to determine the upper and lower relationships of each text line string and each identification line string respectively. According to the upper and lower relationships, each text line string and each identification line string are connected using line feed characters in a top-to-bottom order to obtain a text string and an identification string. Specifically, the line feed character format to be used can be determined first, and a suitable line feed character can be selected according to the specific operating system, programming language and subsequent processing requirements. For example, if the text is processed in the Python language and is mainly oriented to Linux system-related applications, "\n" is usually selected as the line feed character. Next, two empty string variables are created to store the complete text string and identification string to be constructed. Then, the ordinate of the text box coordinate carried by any identification information in the line during line division represents the ordinate of the line, and then the upper and lower relationships are determined in the order of the ordinate from large to small. Among them, the larger the ordinate, the closer it is to the top. Then, each text line string is taken out in turn, a line feed character is added thereafter, and then it is added to the text string variable until all text line strings are added, and finally a complete text string is formed. Similarly, for the identification line string, take out each identification line string in the same order, add a newline character and add it to the identification string variable until all identification line strings are processed. In this way, you will get a text string and identification string that are completely consistent with the form structure.

[0085] In one of the embodiments, the form highlighting method based on the large model further includes: determining the page number identifier of the form to be identified, and adding the page number identifier to each identification information corresponding to the form to be identified.

[0086] It is understandable that in actual form application scenarios, there are often situations where the form content is large and spans multiple pages. In order to accurately locate, track, and subsequently integrate the information on each page of the form, it is necessary to introduce page number identification. By determining the page number identification of each form to be identified and adding it to the corresponding identification information, it is equivalent to establishing a corresponding relationship between each piece of identification information and the page number. For example, when it is necessary to find the specific location of a specific field value in the form later, or to perform information association, aggregation, and other operations based on the identification information, if a multi-page form is encountered, it may not be possible to accurately determine which page it belongs to by relying solely on text box coordinates, unique identifiers, etc., and with the page number identification integrated into the identification information, the page range of the information can be clearly known, so that the entire form information processing process can still be carried out in an orderly and accurate manner in the complex situation of multiple pages, ensuring the integrity and traceability of the information, and better conforming to the actual use logic of multi-page forms.

[0087] In one embodiment, finding the target identification information matching the unique identifier carried by the target result information from the identification information includes: determining the unique page identifier according to the mode of the page identifiers carried by each target identification information, and removing the page identifiers in the target identification information that do not match the unique page identifier.

[0088] It is understandable that when processing multi-page form related operations, since the page number identifier has been added to each identification information before, when it comes to screening target identification information, the unique identifier of some identification information may be incorrectly added to the result information. The unique page number identifier is determined by calculating the mode of the page number identifier carried by the target identification information, based on the idea of ​​probability statistics, that is, the page number identifier with the most occurrences often represents the page number where the field value of the target field is located, that is, the identifier of the page that we are most likely to focus on. For example, in a multi-page questionnaire form, multiple target identification information screened out in response to a field highlight request may have page number identifiers such as "2", "2", "3", "2", "4", etc. "2" appears the most times, so it can be determined that "2" is the unique page number identifier, which means that most of the target identification information is concentrated on page 2, and those with page number identifiers of "3" and "4" are abnormal information.

[0089] Then, the page number identifiers in the target identification information that do not match the unique page number identifier are removed. This is done to further accurately screen out the target identification information that truly meets our expected page, avoid interfering with subsequent operations, and cause unnecessary highlighting on other pages, affecting the final presentation effect and accurate visual display of the target field value. By removing unmatched information, it is ensured that subsequent operations are carried out around the accurate information of the page where the subject is located, thereby improving the accuracy and effectiveness of the entire process.

[0090] In one embodiment, highlighting the target form according to the text box coordinates carried by the target identification information includes: determining whether the page number corresponding to the current canvas matches the unique page number identifier. If so, highlighting the target form according to the text box coordinates carried by the target identification information; otherwise, after cutting the page according to the unique page number identifier, highlighting the target form according to the text box coordinates carried by the target identification information.

[0091] It can be understood that in the processing scenario of multi-page forms, in order to accurately highlight specific target identification information on the target form, it is necessary to ensure that the page where the operation is located (that is, the page number corresponding to the current canvas) is consistent with the page where the target identification information is located (determined by the unique page number identifier). The judgment of whether the two match is based on such a logic: if the page number corresponding to the current canvas and the unique page number identifier are the same, it means that the current page is exactly the page where the target identification information is located, and the highlighting operation can be performed directly according to the text box coordinates carried by the target identification information. In this way, the corresponding text box can be accurately highlighted at the correct position, meeting the expected visual display requirements.

[0092] However, when the two do not match, it means that the current page is not the page where the target information is located. At this time, if the highlighting operation is rashly performed based on the coordinates of the target identification information, it will inevitably cause the highlighting to be displayed on the wrong page, and the purpose of accurately displaying the target field value cannot be achieved. Therefore, it is necessary to perform a page switching operation based on the unique page number identifier, switch the current operation page to the correct page corresponding to the unique page number identifier, so that the subsequent highlighting operation based on the text box coordinates of the target identification information can be performed on the correct page and the correct position, ensuring the accuracy and effectiveness of the entire highlighting process.

[0093] In one embodiment, a form highlighting method based on a large model is shown in FIG. Figure 3 , also includes steps S302 to S306.

[0094] S302, displaying a target form in a first area.

[0095] It can be understood that the display area of ​​the display device in this step includes a first area and a second area. Among them, the first area is the display area of ​​the target form, and the user can clearly and intuitively see the full picture or key parts of the target form, and the highlighting of the target form will also be performed at a specific position in the first area. If the target form is in an electronic document format, such as PDF, Word document, etc., then the corresponding document parsing and display technology needs to be used. Taking PDF documents as an example, some open source PDF rendering libraries (such as PDF.js, etc.) can be used to load and parse the PDF document, and then the first area is delineated in the interface of the web page or software (usually a div element area with a specific size and position can be defined as the first area through HTML and CSS), and the parsed PDF page content is displayed in this area according to the original ratio or the appropriate scaling ratio. If the target form is in the form of an image, such as a scanned paper form image, the image display component can also be used (img tags can be used on the web page side, etc.), the image is loaded and the display style (such as image size, alignment, etc.) is set to adapt it to the first area for display. In addition, some interactive functions can be added according to actual needs, such as setting zoom, pan and other operation buttons in the first area to help users better view different parts of the target form and ensure that the target form can be clearly and completely presented in the first area, thus paving the way for subsequent operations.

[0096] S304: Determine, in the second area, corresponding sub-areas for each preset field of the target form.

[0097] It can be understood that the second area is similar to the first area. It is also a specific space range divided in the display interface. However, its main function is to display the relevant information of each preset field in the target form in a subdivided manner. It is an exclusive area for further operation and presentation of content around each preset field. Determining a one-to-one corresponding sub-area for each preset field in the second area is mainly based on the consideration of information classification display and user convenience. The target form often contains multiple preset fields. Placing the corresponding display content of each of them in a scattered manner will appear messy, and it is difficult for users to quickly find and pay attention to the specific field information they want to know. By dividing the sub-area, it is like creating an exclusive "small space" for each preset field. According to the original field logic of the form or the order of focus of the user, the relevant display content of each preset field can be neatly and orderly placed in the corresponding sub-area, so as to realize the classification management and clear presentation of information. In this way, when viewing the form information, the user can intuitively find the sub-area where the corresponding preset field is located in the second area, and then view the name of the field, select the value, etc., which improves the efficiency and accuracy of information acquisition, and makes the entire form information display interface more standardized and reasonable, which conforms to the design principle of facilitating users to quickly locate and operate in human-computer interaction.

[0098] Specifically, first obtain a list of all preset fields of the target form, which can be read from a pre-configured form template definition. Then, according to the number of preset fields and the overall layout design of the display interface, reasonably divide the sub-areas in the second area. If a web interface is used for display, HTML and CSS technology can also be used to create sub-areas through div elements. For example, in the case of 5 preset fields, 5 parallel div sub-areas can be created (which can be set to a suitable layout mode such as horizontal arrangement or vertical arrangement, which is determined according to the aesthetics of the interface and the convenience of operation), and each sub-area is set with appropriate style attributes such as width, height, spacing, etc., so that it is evenly distributed and clearly identifiable in the second area. The sub-areas corresponding to each preset field can be associated in sequence according to the logic of the importance of the preset field, alphabetical order, or the order of appearance in the form.

[0099] S306: For any preset field, the name of the preset field is displayed in the corresponding sub-area, and result information matching the preset field is found from the result information, and then a value selection control of the preset field is displayed according to the field value carried in the found result information. The value selection control is used to accept a highlight request.

[0100] It can be understood that this step is to organically integrate the basic information (name), actual value, and interactive function (value selection control accepts highlight request) of the preset field and display them in the corresponding sub-area. First, the name of the preset field is displayed, which allows users to quickly understand the specific content category corresponding to the sub-area, which is in line with people's habit of recognizing things from names. Then find the matching result information from the numerous result information, that is, find the matching result information from the result information according to the name of the preset field, and then get the corresponding field value from it, and display the value selection control based on the field value. This can intuitively present the specific value of the preset field, and when the user needs to highlight, he can directly initiate a highlight request in the interactive form of clicking the value selection control.

[0101] The present application provides a form highlighting device based on a large model, including a recognition module, a mapping module, a combination module, a conversion module, a response module and a highlighting module.

[0102] The recognition module is used to input the image of the form to be recognized into the image recognition model for any form to be recognized, and obtain the image recognition result. The image recognition result includes recognition information corresponding to each text block, and the recognition information includes text content and text box coordinates. The mapping module is used to add the corresponding unique identifier to each identification information and determine the first mapping relationship. The first mapping relationship reflects the correspondence between the unique identifier and the recognition information. The combination module is used to connect the corresponding text content and the unique identifier in sequence according to the text box coordinates to obtain a text string and an identification string, so that the text string and the identification string are the same as the structure of the form to be recognized. The conversion module is used to generate a prompt word according to the first mapping relationship, the text string and the identification string, and input the prompt word into the large language model. The prompt word is used to instruct the large language model to find the field value corresponding to each preset field from the text string, and find all the unique identifiers corresponding to the field value from the identification string according to the first mapping relationship, and then fill the found field value and its corresponding unique identifier according to the set output format to obtain the result information corresponding to each preset field, and the preset field includes all fields in the form to be recognized. The response module is used to respond to a highlight request for a value of any field in the target form, determine the field targeted by the highlight request as the target field, find the target result information matching the target field from the result information, and find the target identification information matching the unique identifier carried by the target result information from the identification information. The highlight module is used to highlight on the image of the target form according to the text box coordinates carried by the target identification information.

[0103] For the specific limitations of the form highlighting device based on the large model, please refer to the limitations of the form highlighting method based on the large model mentioned above, which will not be repeated here. The various modules in the above-mentioned viewpoint sorting device can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules. It should be noted that the division of modules in the embodiment of the present application is schematic and is only a logical function division. There may be other division methods in actual implementation.

[0104] The present application provides a computer device, including one or more processors and a memory, wherein the memory stores computer-readable instructions. When the computer-readable instructions are executed by the one or more processors, the steps of the large model-based form highlighting method in any of the above-mentioned embodiments are executed.

[0105] Indicatively, Figure 4 As shown, Figure 4 A schematic diagram of the internal structure of a computer device provided in an embodiment of the present application. Figure 4 The computer device 400 includes a processing component 402, which further includes one or more processors, and a memory resource represented by a memory 401, for storing instructions executable by the processing component 402, such as an application. The application stored in the memory 401 may include one or more modules, each corresponding to a set of instructions. In addition, the processing component 402 is configured to execute instructions to perform the steps of the large model-based form highlighting method of any of the above embodiments.

[0106] The computer device 400 may further include a power supply component 403 configured to perform power management of the computer device 400 , a wired or wireless model interface 404 configured to connect the computer device 400 to the model, and an input / output (I / O) interface 405 .

[0107] The present application provides a storage medium storing computer-readable instructions. When the computer-readable instructions are executed by one or more processors, the one or more processors execute the steps of the large model-based form highlighting method in any of the above-mentioned embodiments.

[0108] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0109] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The various embodiments can be combined as needed, and the same or similar parts can refer to each other.

[0110] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A form highlighting method based on a large model, characterized in that: include: For any form to be identified, the image of the form to be identified is input into the image recognition model to obtain an image recognition result; the image recognition result includes identification information corresponding to each text block, and the identification information includes text content and text box coordinates; Adding a corresponding unique identifier to each piece of identification information, and determining a first mapping relationship; The first mapping relationship reflects the corresponding relationship between the unique identifier and the identification information; According to the text box coordinates, the corresponding text content and the unique identifier are sequentially connected to obtain a text string and an identifier string, so that the text string and the identifier string are the same as the form structure to be identified; Generate a prompt word according to the first mapping relationship, the text string and the identification string, and input the prompt word into the large language model; the prompt word is used to instruct the large language model to find the field value corresponding to each preset field from the text string, and find all the unique identifiers corresponding to the field value from the identification string according to the first mapping relationship, and then fill the found field value and the corresponding unique identifier according to the set output format to obtain result information corresponding to each preset field, and the preset field includes all fields in the form to be identified; In response to a highlight request for taking a value of any field of a target form, determining the field targeted by the highlight request as a target field, finding target result information matching the target field from the result information, and finding target identification information matching the unique identifier carried by the target result information from the identification information; The target form is highlighted according to the text box coordinates carried by the target identification information.

2. The form highlighting method based on a large model according to claim 1, characterized in that: The step of sequentially connecting the corresponding text content and the unique identifier according to the text box coordinates to obtain a text string and an identifier string includes: According to the text box coordinates, the corresponding identification information is divided into lines; The text content and the unique identifier in the identification information belonging to the same row are sequentially connected to obtain a plurality of text row character strings and identifier row character strings; The text line character strings and the identification line character strings are connected in sequence respectively using line feed characters to obtain the text character string and the identification character string.

3. The form highlighting method based on a large model according to claim 2 is characterized in that: The step of sequentially connecting the text content and the unique identifier in the identification information belonging to the same row includes: For the identification information in the same row, the left-right relationship is determined according to the corresponding text box coordinates; According to the left-right relationship, the text content and the unique identifier in the identification information of the same row are connected respectively in order from left to right to obtain a plurality of the text row character strings and the identifier row character strings.

4. The form highlighting method based on a large model according to claim 2, characterized in that: The step of sequentially connecting the text content and the unique identifier in the identification information belonging to the same row includes: Determine the upper and lower relationships of the text line character strings and the logo line character strings respectively; According to the up-down relationship, the text line character strings and the identification line character strings are connected in order from top to bottom using line break characters to obtain the text character string and the identification character string.

5. The form highlighting method based on a large model according to claim 1, characterized in that: Also includes: The page number identifier of the form to be identified is determined, and the page number identifier is added to each piece of identification information corresponding to the form to be identified.

6. The form highlighting method based on a large model according to claim 5 is characterized in that: Finding target identification information matching the unique identifier carried by the target result information from the identification information includes: Determine a unique page number identifier according to the mode of the page number identifiers carried by each target identification information; The page number identifiers in the target identification information that do not match the unique page number identifier are removed.

7. The form highlighting method based on a large model according to claim 6, characterized in that: The highlighting of the text box coordinates carried by the target identification information on the target form includes: Determine whether the page number corresponding to the current canvas matches the unique page number identifier; If so, the target form is highlighted according to the text box coordinates carried by the target identification information; otherwise, the page is cut according to the unique page number identifier, and then the target form is highlighted according to the text box coordinates carried by the target identification information.

8. The form highlighting method based on a large model according to claim 1, characterized in that: Also includes: Display the target form in the first area; Determine in the second area a one-to-one corresponding sub-area for each of the preset fields of the target form; For any one of the preset fields, the name of the preset field is displayed in the corresponding sub-area, and the result information matching the preset field is found from the result information, and then the value selection control of the preset field is displayed based on the field value carried in the found result information; the value selection control is used to accept the highlight request.

9. A form highlighting device based on a large model, characterized in that: include: A recognition module, for any form to be recognized, inputting the image of the form to be recognized into an image recognition model to obtain an image recognition result; the image recognition result includes recognition information corresponding to each text block, and the recognition information includes text content and text box coordinates; A mapping module, used to add a corresponding unique identifier to each piece of identification information and determine a first mapping relationship; The first mapping relationship reflects the corresponding relationship between the unique identifier and the identification information; A combining module, used for sequentially connecting the corresponding text content and the unique identifier according to the text box coordinates to obtain a text string and an identifier string, so that the text string and the identifier string are the same as the form structure to be identified; a conversion module, configured to generate a prompt word according to the first mapping relationship, the text string and the identification string, and input the prompt word into a large language model; the prompt word is used to instruct the large language model to find field values ​​corresponding to each preset field from the text string, and find all the unique identifiers corresponding to the field value from the identification string according to the first mapping relationship, and then fill the found field value and its corresponding unique identifier according to a set output format to obtain result information corresponding to each preset field, wherein the preset field includes all fields in the form to be identified; A response module, configured to respond to a highlight request for taking a value of any field of a target form, determine the field targeted by the highlight request as a target field, find target result information matching the target field from the result information, and find target identification information matching the unique identifier carried by the target result information from the identification information; A highlighting module is used to highlight the image of the target form according to the text box coordinates carried by the target identification information.

10. A computer device, characterized in that: It includes one or more processors and a memory, wherein the memory stores computer-readable instructions, and when the computer-readable instructions are executed by the one or more processors, the steps of the large model-based form highlighting method as described in any one of claims 1-8 are performed.

11. A storage medium, characterized in that: The storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by one or more processors, the one or more processors execute the steps of the large model-based form highlighting method as described in any one of claims 1-8.

Images