A method, apparatus, computer device, and storage medium for form highlighting based on a large model.

Abstract

This application provides a form highlighting method, apparatus, computer device, and storage medium based on a large language model. The method involves inputting a screenshot of the form to be recognized into an image recognition model to obtain the image recognition result. Then, a unique identifier is added, a first mapping relationship is determined, and a text string and an identifier string are further constructed. The large language model is then instructed to generate the result information required for the highlighting function based on this information. Finally, based on the result information, the method can quickly locate and highlight the form in response to the user's highlighting request. This solution realizes a complete process from form image to structured information extraction and visualization, effectively relying on a large language model to improve the efficiency, accuracy, and convenience of form highlighting processing.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a form highlighting method, apparatus, computer device, and storage medium based on a large model. Background Technology

[0002] Traditional paper-based record books, due to the large amount of paper forms, struggle with effective data management. With the development of information technology, using electronic forms for data recording has become the mainstream. However, because forms contain a lot of data, locating specific data can be difficult. Highlighting features can solve this problem. However, highlighting requires first constructing a data structure with a specific reference, containing information such as the keys, values, and positions of each field in the form. With the development of artificial intelligence, traditional techniques involve inputting raw data into a large language model, hoping to rely on the model's semantic understanding capabilities to construct such specific data. However, the processing results obtained by traditional methods are not ideal. Summary of the Invention

[0003] The purpose of this application is to at least solve one of the aforementioned technical defects, and in particular, to provide a solution that can stably achieve field value highlighting based on a large language model.

[0004] Firstly, this application provides a form highlighting method based on a large model, including:

[0005] For any form to be recognized, the image of the form is input into the image recognition model to obtain the image recognition result; the image recognition result includes recognition information corresponding to each text block, including text content and text box coordinates;

[0006] Each piece of identification information is assigned a corresponding unique identifier, and a first mapping relationship is determined; the first mapping relationship reflects the correspondence between the unique identifier and the identification information.

[0007] Based on the coordinates of the text boxes, the corresponding text content and unique identifier are concatenated in sequence to obtain the text string and the identifier string, so that the text string and the identifier string are the same as the structure of the form to be recognized;

[0008] Based on the first mapping relationship, the text string, and the identifier string, a prompt word is generated and input into the large language model. The prompt word is used to instruct the large language model to find the field value corresponding to each preset field from the text string, and to find all unique identifiers corresponding to the field value from the identifier string according to the first mapping relationship. Then, the found field value and its corresponding unique identifier are filled in according to the set output format to obtain the result information corresponding to each preset field. The preset fields include all fields in the form to be recognized.

[0009] In response to a highlight request for any field value in the target form, the field targeted by the highlight request is identified as the target field. The target result information that matches the target field is found from the result information, and the target identification information that matches the unique identifier carried by the target result information is found from the identification information.

[0010] Highlight the target form based on the coordinates of the text box carried by the target recognition information.

[0011] In one embodiment, based on the text box coordinates, the corresponding text content and unique identifier are concatenated sequentially to obtain a text string and an identifier string, including:

[0012] Based on the coordinates of the text box, the corresponding recognition information is divided into lines;

[0013] The text content and unique identifier in the identification information belonging to the same line are concatenated in sequence to obtain multiple text line strings and identification line strings;

[0014] By using newline characters, the text lines and identifier lines are concatenated in sequence to obtain the text string and the identifier string.

[0015] In one embodiment, the text content and unique identifier in the identification information belonging to the same row are sequentially concatenated, including:

[0016] For the recognition information in the same row, the left and right relationships are determined based on the coordinates of the corresponding text boxes;

[0017] Based on the left-right relationship, the text content and unique identifier in the identification information of the same line are concatenated in order from left to right to obtain multiple text line strings and identifier line strings.

[0018] In one embodiment, the text content and unique identifier in the identification information belonging to the same row are sequentially concatenated, including:

[0019] Determine the relationship between the text lines and the identifier lines respectively;

[0020] Based on the top-to-bottom relationship, the text lines and identifier lines are concatenated using newline characters to obtain the text string and identifier string.

[0021] In one embodiment, the form highlighting method based on a large model further includes:

[0022] Determine the page number identifier of the form to be recognized, and add the page number identifier to each recognition information corresponding to the form.

[0023] In one embodiment, finding target identification information that matches the unique identifier carried by the target result information from the identification information includes:

[0024] A unique page number identifier is determined based on the mode of the page number identifier carried by each target identification information;

[0025] Remove pages in the target identification information that do not match the unique page number identifier.

[0026] In one embodiment, highlighting the target form based on the text box coordinates carried by the target recognition information includes:

[0027] Determine if the page number corresponding to the current canvas matches the unique page number identifier;

[0028] If so, the text box coordinates carried by the target identification information are highlighted on the target form; otherwise, the page is paginated according to the unique page number identifier, and then the text box coordinates carried by the target identification information are highlighted on the target form.

[0029] In one embodiment, the form highlighting method based on a large model further includes:

[0030] Display the target form in the first area;

[0031] In the second area, a sub-area is defined that corresponds one-to-one with each preset field of the target form;

[0032] For any given preset field, the name of the preset field is displayed in the corresponding sub-area. The result information that matches the preset field is found from the result information. Then, based on the field value carried in the found result information, the value selection control of the preset field is displayed. The value selection control is used to accept highlighting requests.

[0033] Secondly, this application provides a form highlighting device based on a large model, comprising:

[0034] The recognition module is used to input the image of any form to be recognized into the image recognition model and obtain the image recognition result. The image recognition result includes recognition information corresponding to each text block, including text content and text box coordinates.

[0035] The mapping module is used to add a corresponding unique identifier to each piece of identification information and determine the first mapping relationship; the first mapping relationship reflects the correspondence between the unique identifier and the identification information.

[0036] The combination module is used to connect the corresponding text content and unique identifier in sequence according to the coordinates of the text box to obtain the text string and the identifier string, so that the text string and the identifier string are the same as the form structure to be recognized;

[0037] The conversion module is used to generate prompt words based on the first mapping relationship, text string, and identifier string, and input the prompt words into the large language model. The prompt words are used to instruct the large language model to find the field values ​​corresponding to each preset field from the text string, and to find all unique identifiers corresponding to the field values ​​from the identifier string according to the first mapping relationship. Then, the found field values ​​and their corresponding unique identifiers are filled in according to the set output format to obtain the result information corresponding to each preset field. The preset fields include all fields in the form to be recognized.

[0038] The response module is used to respond to a highlight request for any field value of the target form, identify the field targeted by the highlight request as the target field, find the target result information that matches the target field from the result information, and find the target identification information that matches the unique identifier carried by the target result information from the identification information.

[0039] The highlighting module is used to highlight the text box coordinates carried by the target recognition information on the image of the target form.

[0040] Thirdly, this application provides a computer device including one or more processors and a memory storing computer-readable instructions. When executed by one or more processors, the computer-readable instructions perform the steps of the large-model-based form highlighting method in any of the above embodiments.

[0041] Fourthly, this application provides a storage medium storing computer-readable instructions, which, when executed by one or more processors, cause the one or more processors to perform the steps of the large-model-based form highlighting method in any of the above embodiments.

[0042] As can be seen from the above technical solutions, the embodiments of this application have the following advantages:

[0043] This application's large-scale model-based form highlighting method obtains image recognition results by inputting a screenshot of the form to be recognized into an image recognition model. This efficiently and accurately extracts key information such as text content and text box coordinates from the form, laying the foundation for subsequent processing and eliminating the tediousness and error-proneness of manual data entry. Next, a unique identifier is added and the first mapping relationship is determined. Furthermore, text strings and identifier strings are constructed, giving clear connections and organization to numerous complex recognition information. This facilitates the large-scale language model in generating the result information required for the highlighting function, enabling it to more accurately find the field values ​​corresponding to each preset field from the text string. This further structures the form information, improving the efficiency and accuracy of information extraction. The step of responding to highlighting requests and performing corresponding operations can be quickly located and highlighted based on the output results of the large-scale language model. Overall, this solution realizes a complete process from form image to structured information extraction and visualization, effectively improving the efficiency, accuracy, and convenience of form processing. Attached Figure Description

[0044] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0045] Figure 1 A flowchart illustrating a form highlighting method based on a large model provided in one embodiment of this application;

[0046] Figure 2 This is a flowchart illustrating the process of concatenating a text string and an identifier string in one embodiment of this application;

[0047] Figure 3 This is a schematic diagram illustrating the functional flow in one embodiment of this application;

[0048] Figure 4 This is an internal structural diagram of a computer device provided in one embodiment of this application. Detailed Implementation

[0049] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0050] This application provides a form highlighting method based on a large model; please refer to [link / reference]. Figure 1 This includes steps S102 to S112.

[0051] S102, for any form to be recognized, input the image of the form into the image recognition model to obtain the image recognition result. The image recognition result includes recognition information corresponding to each text block, including text content and text box coordinates.

[0052] It is understood that steps S102 to S108 in this application are the data preparation stage before highlighting. The form to be recognized refers to various form files from which information needs to be extracted, such as corporate financial expense reports, various business application forms, shipping documents, etc., which are documents presented in form form and contain a lot of text information. The form to be recognized can be a paper form or an electronic form. An image recognition model is a model built based on machine learning or deep learning algorithms. After being trained on a large amount of labeled image data, it can analyze and process the input image. The image recognition model in this embodiment can automatically divide the image into blocks and generate a rectangle that selects the entire text block, obtain the coordinates of the four vertices of the rectangle (i.e., the text box coordinates), and also recognize the content (text content) in each text block.

[0053] Image recognition models can be selected from open-source models with the aforementioned functions, such as Tesseract, a model specifically designed for text recognition. Alternatively, models can be built based on open-source image recognition frameworks (such as TensorFlow and PyTorch), and then trained again using a dataset containing forms with the same structure as the form to be recognized. If the form to be recognized has multiple structures, all types of forms should be included in the dataset to enhance the model's recognition ability. For the image of the form to be recognized, the image clarity and integrity should be improved as much as possible. If necessary, image preprocessing can be performed, such as resizing and enhancing contrast, to enable the model to better recognize the content. The processed image is then input into the selected image recognition model. After internal calculations, the model outputs recognition information containing one-to-one correspondences between each text block.

[0054] S104, add a corresponding unique identifier to each piece of identification information and determine the first mapping relationship. The first mapping relationship reflects the correspondence between the unique identifier and the identification information.

[0055] As we can understand, a unique identifier is a unique number or mark assigned to each piece of identification information to distinguish different pieces of information, enabling accurate location and tracking of a specific one when processing numerous pieces of identification information. It can be in the form of a numerical code, an alphanumeric combination, etc. Identification information before adding a unique identifier can be:

[0056]

[0057] The identification information after adding a unique identifier can be:

[0058]

[0059] Obtaining the first mapping relationship involves recording the correspondence between unique identifiers and identification information. In implementation, the first step is to determine the rules for generating unique identifiers. This can be done by sequentially numbering each identifier, for example, starting from 1, assigning each new identifier a number one greater than the previous one. Alternatively, a more meaningful unique identifier can be generated by combining information such as form type and the area where the text block is located. For example, for a financial reimbursement form, the text block in the expense details area could use a format like "FB_FMXM_001" ("FB" represents financial reimbursement form, "FMXM" represents expense details, and "001" represents the first text block in that area). After determining the generation rules, each piece of identification information obtained from image recognition is assigned a unique identifier according to the rules. Then, a data structure is created to store the first mapping relationship. Common data structures include dictionaries and hash tables, storing the text content as the key and the corresponding unique identifier as the value. This establishes a clear first mapping relationship.

[0060] S106. Based on the coordinates of the text boxes, concatenate the corresponding text content and unique identifier in sequence to obtain a text string and an identifier string, so that the text string and the identifier string are the same as the structure of the form to be recognized.

[0061] It can be understood that a text string is a string formed by concatenating all the text content in a form according to its original structure. It reflects the arrangement of text information on the form and transforms it into a continuous form of plain text. Because the form itself has a certain layout and order, different text blocks in different positions have different functions. The coordinates of the text boxes indicate their order and positional relationship. Concatenating the text content into a text string in this order is equivalent to integrating the text information of the form together, but still carrying the structural information to be recognized, providing more information for the subsequent large language model to understand the content of the form to be recognized. Formally, an identifier string is simply replacing the text content in the text string with the corresponding unique identifier. Part of the text string can be: "Name: Zhang San / nDepartment: R&D Department / n", where the unique identifier for "Name" is 1, the unique identifier for the first ":" is 2, the unique identifier for "Zhang San" is 3, / n represents a newline, the unique identifier for "Department" is 4, the unique identifier for the second ":" is 5, and the unique identifier for "R&D Department" is 6. Then the identifier string corresponding to this example can be: "123 / n456 / n".

[0062] S108: Generate prompt words based on the first mapping relationship, the text string, and the identifier string, and input the prompt words into the large language model. The prompt words are used to instruct the large language model to find the field values ​​corresponding to each preset field from the text string, and to find all unique identifiers corresponding to the field values ​​from the identifier string according to the first mapping relationship. Then, fill the found field values ​​and their corresponding unique identifiers according to the set output format to obtain the result information corresponding to each preset field. The preset fields include all fields in the form to be recognized.

[0063] It can be understood that the first mapping relationship, text string, and identifier string obtained in steps S102 to S106 are the raw data extracted from the form to be recognized, and the result information is the data upon which the highlighting function is based. The large language model in this embodiment is an artificial intelligence model with powerful language understanding and generation capabilities, trained on massive amounts of text data. It can understand the task requirements based on the input prompts and then use its learned language knowledge and logical reasoning abilities to generate the corresponding results. It can be an open-source large language model or a self-trained private model. The preset fields include all fields in the form to be recognized, and each preset field in this embodiment will generate corresponding result information.

[0064] In this embodiment, the prompt words are instructions or guidance information described in natural language, used to tell the large language model what task to perform and what requirements to follow when processing the input data. Here, they are constructed based on the mapping relationships and text strings obtained in the previous steps. That is, the prompt words need to list the preset fields and clearly state that the task requirement is to accurately extract the field values ​​corresponding to the preset fields from the text string. Since the text string retains both the structural and semantic information of the form to be recognized, the large language model's semantic understanding ability can determine the position of each preset field in the text string and, combined with its general understanding of the form structure, search for the corresponding field value from the vicinity of that position. For example, the text string might contain content such as "Product Name: Smartphone", "Inventory Quantity: 5", and "Customer Name: Li Si" in an orderly manner. This allows the large language model to use its semantic understanding and analysis capabilities to locate and extract the field values ​​corresponding to the preset fields we need. For example, when you want to get the value of the preset field "inventory quantity", you only need to search for relevant expressions in this complete text string according to the language logic to accurately find "product quantity: 5 units", thereby extracting the key field value "5", which provides the most basic text material support for the generation of the final result information.

[0065] Furthermore, the task requirement needs to be clarified: to further find all unique identifiers corresponding to the field values ​​in the identifier string based on the first mapping relationship (some field values ​​are composed of the content of two or more text blocks), and then fill the found field values ​​and their corresponding unique identifiers according to the set output format. The identifier string connects the unique identifiers corresponding to the text content in a completely consistent order with the text content. It establishes a strict one-to-one correspondence with the text string. Although the first mapping relationship can link field values ​​with unique identifiers, the field values ​​of different fields in the form may be the same or similar, leading to errors in the final output information and ultimately affecting the highlighting function. However, because the identifier string corresponds to the text string in order and fits the form structure, it can accurately distinguish them based on the structural information it reflects. For example, the above example also includes the field "Sales Quantity: 5", which has the same field value as "Inventory Quantity" but in a different position. The large language model can only determine the position of the "Inventory Quantity" field value in the text string based on the structural information of each field value reflected in the identifier string. The position of this field value in the identifier string must match the position in the text string, thus distinguishing these two identical field values.

[0066] In some embodiments, to enhance the large language model's understanding of domain-specific forms, the prompt can indicate the role played by the large language model, which could be a senior expert in the domain corresponding to the form to be identified. In some embodiments, the structure of some forms to be identified differs from regular forms, or some users' data extraction requirements differ from the norm. These special requirements or considerations also need to be clearly stated in the prompt. In some embodiments, the output format can be set to JSON data, which can specify which elements need to be included, what the key of each element is, and what value needs to be filled for each key, for example:

[0067]

[0068] Among them, "Field Name" is filled with the name of the corresponding preset field, "Value" is filled with the field value corresponding to the preset field, and "id" is filled with the unique identifier set corresponding to all text blocks that make up the field value.

[0069] S110, in response to a highlight request for any field value of the target form, the field targeted by the highlight request is determined as the target field, the target result information matching the target field is found from the result information, and the target identification information matching the unique identifier carried by the target result information is found from the identification information.

[0070] It is understandable that steps S110 and S112 are the execution phase of the highlighting function. The target form is the form that the user needs to highlight. Each form that can be selected as a target form has been processed through steps S102 to S108, resulting in multiple corresponding result information and identification information. The user triggers the highlighting function through a highlighting request. Typically, when the user views a form or displays form information, they want to highlight the content corresponding to a specific field for easy viewing and differentiation. Upon receiving a highlighting request, the target field is first determined, and then the result information obtained from the previous large language model processing is used to locate the result information that matches the target field, thus obtaining the target result information. Specifically, the mapping relationship between each result information and the corresponding preset field can be established based on the field name, meaning that each result information can be matched based on the name of the target field. Each unique identifier is matched with one of the many identification information. The target result information carries all the unique identifiers of the text boxes corresponding to the field value of the target field. Matching is performed on these unique identifiers in the identification information, and these matched identification information are the target identification information.

[0071] The entire process starts with a user request and gradually locates the specific position of the content to be highlighted through data association, thereby highlighting the key information in the form.

[0072] S112, Highlight the target form based on the text box coordinates carried by the target recognition information.

[0073] It's understandable that a form, as a whole, is presented as an image or electronic document, with each element having corresponding coordinates. The coordinates of the text boxes clearly define the specific area containing the content we want to highlight. By obtaining these coordinates, relevant software or systems can determine the specific location within the form to perform the highlighting operation. Whether it's modifying the area's color attributes or adding a specific style of border, it's all based on the range defined by the coordinates. The goal is to change the visual presentation of that area, making it quickly attract the user's attention within the entire form, allowing the user to easily view and focus on the field values ​​they need to focus on. The text box coordinates carried by the target identification information correspond to the field values ​​of the target field. After finding the matching target identification information, connecting the areas defined by the text box coordinates carried by the target identification information determines the areas that need to be highlighted. Highlighting these areas will highlight the field values ​​of the target field.

[0074] This embodiment's form highlighting method based on a large model obtains image recognition results by inputting a screenshot of the form to be recognized into an image recognition model. This efficiently and accurately extracts key information such as text content and text box coordinates from the form, laying the foundation for subsequent processing and eliminating the tediousness and error-proneness of manual data entry. Next, a unique identifier is added and the first mapping relationship is determined. Furthermore, text strings and identifier strings are constructed, giving clear connections and organization to numerous complex recognition information. This facilitates the large language model in generating the result information required for the highlighting function, enabling it to more accurately find the field values ​​corresponding to each preset field from the text string. This further structures the form information, improving the efficiency and accuracy of information extraction. The step of responding to highlighting requests and performing corresponding operations can be quickly located and highlighted based on the output results of the large language model. Overall, this solution realizes a complete process from form image to structured information extraction and visualization, effectively improving the efficiency, accuracy, and convenience of form processing.

[0075] In one embodiment, based on the text box coordinates, the corresponding text content and unique identifier are concatenated in sequence to obtain a text string and an identifier string. Please refer to [link to relevant documentation]. Figure 2 This includes steps S202 to S206.

[0076] S202, Based on the coordinates of the text box, divide the corresponding recognition information into lines.

[0077] It's understandable that forms, as structured information carriers, present their content according to certain rules, often arranged in rows, with different rows carrying different categories of information or different parts of the same type of information. Using text box coordinates for line breaks follows this inherent layout logic of forms, utilizing the positional relationships indicated by coordinates to group the corresponding identification information of text boxes at the same horizontal level (i.e., visually in the same row). The purpose of this is to restore the original structural order of the form, ensuring that subsequent operations such as integrating and connecting text content conform to the form's own logic, facilitating further information extraction and processing, avoiding information confusion and mismatch, and preparing for the construction of accurate text strings and identifier strings reflecting the form's content.

[0078] Specifically, these identified information entries are sorted according to their ordinates (vertical coordinates). The smaller the ordinate value, the higher the text box is in the image, simulating the top-to-bottom reading order of the form. Then, the sorted information is iterated through, and a reasonable threshold for the difference in ordinates is set, such as 5 pixels (this can be adjusted based on the actual form and text box size). When the difference in ordinates between two adjacent pieces of information is less than this threshold, they are grouped into the same row. This process is repeated for all identified information, dividing them into different rows according to the above rules, thus completing the row splitting operation.

[0079] S204, the text content and unique identifier in the identification information belonging to the same line are concatenated in sequence to obtain multiple text line strings and identification line strings.

[0080] It's understandable that after splitting the identification information into rows, each row represents a combination of content within the same horizontal line of the form. Concatenating the text content within the identification information in sequence within the same row yields the corresponding text line string, and concatenating the unique identifiers within the identification information in sequence yields the corresponding identifier line string. The requirement for sequential concatenation is to maintain the structure of the form within that row; specifically, the order of appearance of fields and field values ​​should be consistent. For example, in a row of an employee information table, if the text box for "Name" appears first, followed by the text box for "Department," etc., concatenating the text content in sequence to form the text line string accurately reflects the order of text descriptions in that row. Similarly, concatenating unique identifiers in sequence to form the identifier line string establishes a one-to-one correspondence between text content and identifiers, facilitating subsequent information searching, matching, and association operations, whether within the same row or at the entire form level.

[0081] The sequential connection in step S204 can be performed on the same row of identification information. First, the left-right relationship is determined based on the coordinates of the corresponding text boxes. Then, based on the left-right relationship, the text content and unique identifier in the identification information of the same row are connected in order from left to right, resulting in multiple text line strings and identifier line strings. Specifically, the identification information that has already been divided into rows is processed line by line. For each row, two empty string variables are created, one to store the text line string to be constructed and the other to store the identifier line string. Then, based on the horizontal coordinates of the text box coordinates of the identification information in this row, the left-right relationship is determined in ascending order. The smaller the horizontal coordinate, the more to the left. Since the text box coordinates include the coordinates of four vertices, the horizontal coordinates here can be selected from vertices in the same direction for comparison, or the center coordinates can be calculated based on these four coordinates and then compared. This embodiment does not limit this. After determining the left-right relationship, the text content of each identification information can be extracted in order from left to right and added to the text line string variable. In addition, the unique identifier corresponding to each identification information is extracted in the same order and added to the identifier line string variable.

[0082] S206: Use newline characters to concatenate the text lines and identifier lines in sequence to obtain the text string and identifier string.

[0083] As we can understand, a form is a multi-line information carrier. Previously, by using line breaks and separately constructing text and identifier strings, information was integrated at the line level. Connecting them sequentially using line breaks restores the complete multi-line structure of the form, elevating information integration from the line-level combination to the entire form level. Inserting line breaks ensures that the connected text strings are displayed and processed according to the form's original multi-line layout. Similarly, the identifier strings are connected in the same order and manner, maintaining a strict correspondence with the text strings. In this way, the entire text and identifier strings completely replicate the form's text content and structural logic.

[0084] The sequential concatenation in step S206 can be achieved by separately determining the top-to-bottom relationship between each text line string and each identifier line string. Based on this relationship, the text line strings and identifier line strings are concatenated using newline characters in a top-to-bottom order to obtain the text string and identifier string. Specifically, the newline character format to be used can be determined first. A suitable newline character can be selected based on the specific operating system, programming language, and subsequent processing requirements. For example, if processing text in Python and primarily targeting Linux systems, "\n" is typically chosen as the newline character. Next, two empty string variables are created to store the complete text string and the identifier string to be constructed, respectively. Then, the vertical coordinate of the line is represented by the y-coordinate of the text box carried by any identifier information within that line, and the top-to-bottom relationship is determined according to the descending order of the y-coordinate. The larger the y-coordinate, the higher the position. Each text line string is then extracted sequentially, a newline character is added to its end, and then it is added to the text string variable until all text line strings have been added, ultimately forming the complete text string. Similarly, for the identifier string, each identifier string is extracted in the same order, a newline character is added, and then added to the identifier string variable, until all identifier string strings have been processed. This gives us a text string and identifier string that perfectly match the form structure.

[0085] In one embodiment, the form highlighting method based on a large model further includes: determining the page number identifier of the form to be identified, and adding the page number identifier to each identification information corresponding to the form to be identified.

[0086] It's understandable that in real-world form applications, there are often situations where form content is extensive and spans multiple pages. To accurately locate, track, and integrate information on each page of the form, introducing page number identifiers is essential. By assigning a page number identifier to each form to be identified and adding it to the corresponding identification information, a correspondence is established between each piece of identification information and the page number. For example, when it's necessary to find the specific location of a particular field value in the form, or to perform information association or summarization based on the identification information, if there are multiple pages, relying solely on text box coordinates or unique identifiers may not accurately determine which page it belongs to. However, with page number identifiers integrated into the identification information, the page range where the information is located becomes clear. This ensures that the entire form information processing flow remains orderly and accurate even in complex multi-page scenarios, guaranteeing information integrity and traceability, and better aligning with the usage logic of multi-page forms in practice.

[0087] In one embodiment, finding target identification information that matches the unique identifier carried by the target result information from the identification information includes: determining a unique page number identifier based on the mode of the page number identifiers carried by each target identification information; and removing page number identifiers in the target identification information that do not match the unique page number identifier.

[0088] It's understandable that when processing multi-page forms, since page numbers have already been added to each piece of identification information, some unique identifiers might be incorrectly included in the results when filtering target identification information. Determining the unique page number identifier by calculating the mode of the page number identifiers carried by the target identification information is based on probabilistic statistics. The page number identifier that appears most frequently often represents the page number where the target field's value is located—that is, the identifier of the page we are most likely to focus on. For example, in a multi-page questionnaire, after responding to a field highlighting request, multiple target identification information entries might have page number identifiers like "2", "2", "3", "2", and "4". Since "2" appears most frequently, it can be determined that "2" is the unique page number identifier. This means that most target identification information is concentrated on page 2, and those with page number identifiers "3" and "4" are abnormal information.

[0089] Then, pages in the target identification information that do not match the unique page number identifier are removed. This is done to further accurately filter out the target identification information that truly matches the page we expect it to be on, avoiding interference with subsequent operations and preventing unnecessary highlighting on other pages, which would affect the final presentation and the accurate visualization of the target field values. By removing mismatched information, we ensure that subsequent operations are all carried out around the accurate information of the page where the main content is located, thus improving the accuracy and effectiveness of the entire process.

[0090] In one embodiment, highlighting the target form based on the text box coordinates carried by the target identification information includes: determining whether the page number corresponding to the current canvas matches the unique page number identifier. If yes, the target form is highlighted based on the text box coordinates carried by the target identification information; otherwise, the page is paginated based on the unique page number identifier, and then the target form is highlighted based on the text box coordinates carried by the target identification information.

[0091] It's understandable that in multi-page form processing scenarios, to accurately highlight specific target identification information on the target form, it's necessary to ensure that the page where the operation is taking place (i.e., the page number corresponding to the current canvas) matches the page containing the target identification information (determined by a unique page number). The determination of whether they match is based on the following logic: if the page number corresponding to the current canvas and the unique page number identifier are the same, it means that the current page is indeed the page containing the target identification information. Highlighting can then be performed directly based on the coordinates of the text boxes carried by the target identification information. This ensures accurate highlighting of the corresponding text boxes in the correct positions, meeting the expected visualization requirements.

[0092] However, when the two do not match, it indicates that the current page is not the page containing the target information. In this case, if highlighting is performed based solely on the coordinates of the target information, it will inevitably result in highlighting on the wrong page, failing to accurately display the target field value. Therefore, it is necessary to perform a page switching operation based on the unique page number identifier, switching the current operation page to the correct page corresponding to the unique page number identifier. This ensures that subsequent highlighting operations based on the text box coordinates of the target information are performed on the accurate page and in the accurate location, guaranteeing the accuracy and effectiveness of the entire highlighting process.

[0093] In one embodiment, a form highlighting method based on a large model is described in [reference needed]. Figure 3 It also includes steps S302 to S306.

[0094] S302, display the target form in the first area.

[0095] It is understood that the display area of ​​the display device in this step includes a first area and a second area. The first area is the display area of ​​the target form, where the user can clearly and intuitively see the entirety or key parts of the target form. Highlighting of the target form will also occur at a specific location within the first area. If the target form is an electronic document format, such as PDF or Word, then appropriate document parsing and display technologies are required. Taking PDF documents as an example, some open-source PDF rendering libraries (such as PDF.js) can be used to load and parse the PDF document. Then, a first area can be defined in the webpage or software interface (usually defined using HTML and CSS as a div element area with specific size and position). The parsed PDF page content can then be displayed within this area at its original scale or a suitable scaling ratio. If the target form is an image, such as a scanned image of a paper form, image display components (such as the img tag on a webpage) can also be used to load the image and set its display style (such as image size and alignment) to fit the first area for display. In addition, some interactive functions can be added according to actual needs, such as setting zoom, pan and other operation buttons in the first area to make it easier for users to view different parts of the target form and ensure that the target form can be clearly and completely presented in the first area, laying the groundwork for subsequent operations.

[0096] S304, in the second area, a sub-area corresponding to each preset field of the target form is determined.

[0097] As is understandable, the second area, similar to the first, is also a specific space defined within the display interface. However, its primary function is to subdivide and display information related to each preset field in the target form. It is a dedicated area for further operations and content presentation around each preset field. Defining one-to-one sub-areas for each preset field within the second area is mainly based on considerations of information categorization and user convenience. Target forms often contain multiple preset fields; scattering their corresponding content would appear cluttered, making it difficult for users to quickly find and focus on the specific field information they want. By dividing the space into sub-areas, it's like creating a dedicated "small space" for each preset field. The relevant content for each preset field can be neatly and orderly placed within its corresponding sub-area according to the original field logic of the form or the user's priority order, achieving categorized management and clear presentation of information. In this way, when viewing form information, users can intuitively find the sub-area containing the corresponding preset field within the second area, and then view the field name, select values, and perform other operations, improving the efficiency and accuracy of information retrieval. It also makes the entire form information display interface more standardized and logical, conforming to the design principle of facilitating quick user location and operation in human-computer interaction.

[0098] Specifically, the first step is to obtain a list of all preset fields in the target form, which can be read from a pre-configured form template definition. Then, based on the number of preset fields and the overall layout design of the display interface, the second area is rationally divided into sub-areas. If a web interface is used, HTML and CSS techniques can also be used to create sub-areas using div elements. For example, for a case with 5 preset fields, 5 parallel div sub-areas can be created (they can be set to a suitable layout, such as horizontal or vertical arrangement, depending on the aesthetics and ease of use of the interface). Each sub-area can be given appropriate width, height, spacing, and other style attributes to ensure even distribution and clear visibility within the second area. The sub-areas corresponding to each preset field can be associated sequentially according to their importance, alphabetical order, or order of appearance in the form.

[0099] S306: For any preset field, display the name of the preset field in the corresponding sub-area, find the result information that matches the preset field from the result information, and then display the value selection control of the preset field based on the field value carried in the found result information. The value selection control is used to accept highlighting requests.

[0100] This step, as you can understand it, organically integrates and displays the basic information (name), actual value, and interactive functionality (value selection control accepting highlight requests) of a preset field within its corresponding sub-area. First, displaying the name of the preset field allows users to quickly understand the specific content scope of that sub-area, aligning with people's habit of recognizing things from their names. Then, it searches for matching results from numerous results, specifically finding the matching information based on the preset field's name, retrieving the corresponding field value, and displaying the value selection control based on that value. This visually presents the specific value of the preset field, and allows users to directly initiate a highlight request by clicking the value selection control when needed.

[0101] This application provides a form highlighting device based on a large model, including a recognition module, a mapping module, a combination module, a conversion module, a response module, and a highlighting module.

[0102] The recognition module takes any form to be recognized, inputs its image into the image recognition model, and obtains the image recognition result. The image recognition result includes recognition information corresponding to each text block, including text content and text box coordinates. The mapping module adds a corresponding unique identifier to each piece of recognition information and determines a first mapping relationship. The first mapping relationship reflects the correspondence between the unique identifier and the recognition information. The combination module concatenates the corresponding text content and unique identifier in sequence according to the text box coordinates to obtain a text string and an identifier string, ensuring that the text string and identifier string have the same structure as the form to be recognized. The conversion module generates prompt words based on the first mapping relationship, the text string, and the identifier string, and inputs the prompt words into the large language model. The prompt words instruct the large language model to find the field values ​​corresponding to each preset field in the text string, and to find all unique identifiers corresponding to the field values ​​in the identifier string according to the first mapping relationship. The found field values ​​and their corresponding unique identifiers are then filled according to a set output format to obtain the result information corresponding to each preset field, which includes all fields within the form to be recognized. The response module is used to respond to highlighting requests for any field value in the target form. It identifies the field targeted by the highlighting request as the target field, finds the target result information matching the target field from the result information, and finds the target identification information matching the unique identifier carried by the target result information from the identification information. The highlighting module is used to highlight the text box coordinates carried by the target identification information on the image of the target form.

[0103] Specific limitations regarding the form highlighting device based on a large model can be found in the limitations of the form highlighting method based on a large model mentioned above, and will not be repeated here. Each module in the above-mentioned sorting device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module. It should be noted that the module division in this embodiment is illustrative and only represents a logical functional division; other division methods may be used in actual implementation.

[0104] This application provides a computer device including one or more processors and a memory storing computer-readable instructions. When executed by one or more processors, the computer-readable instructions perform the steps of the large-model-based form highlighting method in any of the above embodiments.

[0105] Indicatively, such as Figure 4 As shown, Figure 4 This is a schematic diagram of the internal structure of a computer device provided in an embodiment of this application. (Refer to...) Figure 4 The computer device 400 includes a processing component 402, which further includes one or more processors, and memory resources represented by memory 401 for storing instructions, such as application programs, that can be executed by the processing component 402. The application programs stored in memory 401 may include one or more modules, each corresponding to a set of instructions. Furthermore, the processing component 402 is configured to execute instructions to perform the steps of the large-model-based form highlighting method of any of the above embodiments.

[0106] The computer device 400 may also include a power supply component 403 configured to perform power management of the computer device 400, a wired or wireless model interface 404 configured to connect the computer device 400 to a model, and an input / output (I / O) interface 405.

[0107] This application provides a storage medium storing computer-readable instructions that, when executed by one or more processors, cause the one or more processors to perform the steps of the large-model-based form highlighting method in any of the above embodiments.

[0108] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0109] The various embodiments in this specification are described in a progressive manner. Each embodiment focuses on the differences from other embodiments. The various embodiments can be combined as needed, and the same or similar parts can be referred to each other.

[0110] The above description of the disclosed embodiments enables those skilled in the art to make or use this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A form highlighting method based on a large model, characterized in that, include: For any form to be recognized, the image of the form to be recognized is input into the image recognition model to obtain the image recognition result; the image recognition result includes recognition information corresponding to each text block, and the recognition information includes text content and text box coordinates; A corresponding unique identifier is added to each of the identification information, and a first mapping relationship is determined; The first mapping relationship reflects the correspondence between the unique identifier and the identification information; Based on the coordinates of the text box, the corresponding text content and the unique identifier are concatenated in sequence to obtain a text string and an identifier string, so that the text string and the identifier string are the same as the structure of the form to be identified; Based on the first mapping relationship, the text string, and the identifier string, a prompt word is generated and input into the large language model. The prompt word is used to instruct the large language model to find the field value corresponding to each preset field from the text string, and to find all the unique identifiers corresponding to the field value from the identifier string according to the first mapping relationship. Then, the found field value and its corresponding unique identifier are filled according to the set output format to obtain the result information corresponding to each preset field. The preset fields include all fields in the form to be identified. In response to a highlight request for any field value of a target form, the field targeted by the highlight request is determined as the target field. Target result information matching the target field is found from the result information, and target identification information matching the unique identifier carried by the target result information is found from the identification information. The text box coordinates carried by the target recognition information are highlighted on the target form.

2. The form highlighting method based on a large model according to claim 1, characterized in that, The step of concatenating the corresponding text content and the unique identifier in sequence according to the text box coordinates to obtain a text string and an identifier string includes: Based on the coordinates of the text box, the corresponding recognition information is divided into lines; The text content and the unique identifier in the identification information belonging to the same line are concatenated in sequence to obtain multiple text line strings and identifier line strings; The text lines and the identifier lines are concatenated sequentially using newline characters to obtain the text string and the identifier string.

3. The form highlighting method based on a large model according to claim 2, characterized in that, The step of sequentially concatenating the text content and the unique identifier in the identification information belonging to the same row includes: For the recognition information in the same row, the left and right relationships are determined based on the coordinates of the corresponding text boxes; Based on the left-right relationship, the text content and the unique identifier in the identification information of the same row are connected in order from left to right to obtain multiple text line strings and identifier line strings.

4. The form highlighting method based on a large model according to claim 2, characterized in that, The step of sequentially concatenating the text content and the unique identifier in the identification information belonging to the same row includes: Determine the vertical relationship between each of the text line strings and each of the identifier line strings; Based on the above-bottom relationship, the text lines and the identifier lines are concatenated using newline characters in a top-to-bottom order to obtain the text string and the identifier string.

5. The form highlighting method based on a large model according to claim 1, characterized in that, Also includes: Determine the page number identifier of the form to be identified, and add the page number identifier to each of the identification information corresponding to the form to be identified.

6. The form highlighting method based on a large model according to claim 5, characterized in that, Finding target identification information that matches the unique identifier carried in the target result information from the identification information includes: A unique page number identifier is determined based on the mode of the page number identifier carried by each of the target identification information; Remove any pages in the target identification information that do not match the unique page number identifier.

7. The form highlighting method based on a large model according to claim 6, characterized in that, The step of highlighting the text box coordinates on the target form based on the target recognition information includes: Determine whether the page number corresponding to the current canvas matches the unique page number identifier; If so, the text box coordinates carried by the target identification information are used to highlight the target form; otherwise, the page is cut according to the unique page number identifier, and then the text box coordinates carried by the target identification information are used to highlight the target form.

8. The form highlighting method based on a large model according to claim 1, characterized in that, Also includes: The target form is displayed in the first area; In the second region, a sub-region is determined that corresponds one-to-one with each of the preset fields of the target form; For any of the preset fields, the name of the preset field is displayed in the corresponding sub-area, and the result information that matches the preset field is found from the result information. Then, based on the field value carried in the found result information, the value selection control of the preset field is displayed; the value selection control is used to accept the highlight request.

9. A form highlighting device based on a large model, characterized in that, include: The recognition module is used to input the image of any form to be recognized into the image recognition model to obtain the image recognition result; the image recognition result includes recognition information corresponding to each text block, and the recognition information includes text content and text box coordinates; A mapping module is used to add a corresponding unique identifier to each of the identification information and determine a first mapping relationship; The first mapping relationship reflects the correspondence between the unique identifier and the identification information; The combination module is used to connect the corresponding text content and the unique identifier in sequence according to the coordinates of the text box to obtain a text string and an identifier string, so that the text string and the identifier string are the same as the structure of the form to be identified; The conversion module is used to generate prompt words based on the first mapping relationship, the text string, and the identifier string, and input the prompt words into the large language model; the prompt words are used to instruct the large language model to find the field values ​​corresponding to each preset field from the text string, and to find all the unique identifiers corresponding to the field values ​​from the identifier string according to the first mapping relationship, and then fill the found field values ​​and their corresponding unique identifiers according to a set output format to obtain the result information corresponding to each preset field, wherein the preset fields include all fields in the form to be recognized; The response module is used to respond to a highlighting request for any field value of the target form, determine the field targeted by the highlighting request as the target field, find the target result information that matches the target field from the result information, and find the target identification information that matches the unique identifier carried by the target result information from the identification information. The highlighting module is used to highlight the text box coordinates carried by the target recognition information on the image of the target form.

10. A computer device, characterized in that, The method includes one or more processors and a memory storing computer-readable instructions that, when executed by the one or more processors, perform the steps of the large-model-based form highlighting method as described in any one of claims 1-8.

11. A storage medium, characterized in that, The storage medium stores computer-readable instructions that, when executed by one or more processors, cause the one or more processors to perform the steps of the form highlighting method based on a large model as described in any one of claims 1-8.

Images