Form filling method and device and electronic equipment

Abstract

The invention discloses a form filling method and device and electronic equipment, and belongs to the technical field of artificial intelligence. The form filling method comprises the steps that a form image of a to-be-filled form is input into a field information extraction model, field attribute information of a to-be-filled field contained in the to-be-filled form is output, and the field attribute information comprises a field name, a filling type and a filling position of the to-be-filled field; and according to the filling type, filling the filling position corresponding to the field name with the field information corresponding to the user identity information.

Description

Technical Field

[0001] This application belongs to the field of artificial intelligence technology, specifically relating to a form filling method, device, and electronic device. Background Technology

[0002] Currently, when filling out online forms, users need to spend a lot of time filling in personal information such as name, ID number, and home address. The filling efficiency is low and errors are easy to make, especially in scenarios where a large number of forms need to be filled out, which requires a lot of effort. Summary of the Invention

[0003] The purpose of this application is to provide a form filling method, device, and electronic device. By using a field information extraction model, the field name, fill type, and fill position of the fields to be filled in the form are automatically extracted. Based on the fill type, the field information corresponding to the user's identity information is automatically filled into the position corresponding to the field name. This achieves automatic form filling, saves users a lot of time and effort, improves the efficiency of form filling, and avoids filling errors caused by human factors, thus ensuring the accuracy of the filling results.

[0004] In a first aspect, embodiments of this application provide a form filling method, including:

[0005] Extract the form image input field information model from the form image of the form to be filled, and output the field attribute information of the fields to be filled in the form. The field attribute information includes the field name, fill type and fill position of the fields to be filled.

[0006] Based on the fill type, fill the field information corresponding to the user's identity information into the fill position corresponding to the field name.

[0007] In some possible implementations of the embodiments of this application, the fill type includes at least one of the following: input type, drop-down list type, upload type, click type, and submit type.

[0008] In some possible implementations of the embodiments of this application, the form filling method further includes:

[0009] The form image samples, prompt words, and size information of the form image samples in the training samples are input into the pre-trained visual language model, and the output is the predicted field attribute information of the fields to be filled contained in the form image samples. The predicted field attribute information is the information predicted by the pre-trained visual language model based on the prompt words and size information. The predicted field attribute information is the field attribute information of the fields to be filled contained in the form image samples, including the predicted field name, predicted filling type, and predicted filling position of the fields to be filled contained in the form image samples.

[0010] Based on the predicted field attribute information and the actual field attribute information of the fields to be filled contained in the form image sample, the visual language model is trained to obtain the field information extraction model; wherein, the actual field attribute information is the actual field attribute information of the fields to be filled contained in the form image sample, including the actual field name, actual filling type and actual filling position of the fields to be filled contained in the form image sample.

[0011] In some possible implementations of this application's embodiments, before inputting the form image samples, prompt words, and size information of the form image samples from the training samples into a pre-trained visual language model and outputting the predicted field attribute information of the fields to be filled contained in the form image samples, the form filling method further includes:

[0012] Collect multiple form image samples, which include at least one of the following: form images in a reference filling scenario, and form images suitable for the reference device type;

[0013] The fields to be filled in each form image sample are labeled to obtain structured labeled data for the fields to be filled. The structured labeled data includes the field name and the filling type of the fields to be filled.

[0014] Mark the fill position of the field to be filled to obtain the position coordinate information of the fill position;

[0015] The structured annotation data and location coordinate information are used to determine the actual field attribute information of the fields to be filled.

[0016] In some possible implementations of the embodiments of this application, the reference filling scenario includes at least one of the following: license plate number filling scenario, pass filling scenario, passport filling scenario, driver's license filling scenario;

[0017] The reference device type includes at least one of the following: candybar device type, foldable device type, and tablet device type.

[0018] In some possible implementations of the embodiments of this application, the structured annotation data further includes at least one of the following: the already filled content of the field to be filled, and the fill prompt words of the field to be filled.

[0019] In some possible implementations of the embodiments of this application, the fill type includes the input type;

[0020] The fill position of the field to be filled is marked to obtain the position coordinate information of the fill position, including:

[0021] Determine the annotation method for the fill position based on the fill type of the field to be filled; different fill types correspond to different annotation methods.

[0022] According to the annotation method of the fill position, the fill position is annotated to obtain the position coordinate information of the fill position;

[0023] The location coordinate information includes the coordinates of the center point of the filling location.

[0024] In some possible implementations of the embodiments of this application, the fill position is marked according to the marking method of the fill position to obtain the position coordinate information of the fill position, including:

[0025] When the input type is an input box, a marker box is displayed at the fill position according to the size information of the input box, and the position coordinate information of the fill position is determined according to the display position of the marker box;

[0026] When the input type is underline, a marker box is displayed at the fill position, and the position coordinates of the fill position are determined based on the display position of the marker box; the reference line of the marker box is the end of the underline or the boundary of the field to be filled.

[0027] In some possible implementations of the embodiments of this application, a visual language model is trained based on the predicted field attribute information and the actual field attribute information of the fields to be filled contained in the form image sample to obtain a field information extraction model, including:

[0028] Based on the predicted field attribute information and the actual field attribute information of the fields to be filled contained in the form image samples, the visual language model is trained for N rounds to obtain N candidate models.

[0029] Based on the validation set, the performance of each candidate model among the N candidate models is validated to obtain the accuracy of the field attribute information output by each candidate model;

[0030] The candidate model with the highest accuracy was selected as the field information extraction model.

[0031] In some possible implementations of the embodiments of this application, before filling the field information corresponding to the user identity information into the fill position corresponding to the field name according to the fill type, the form filling method further includes:

[0032] Based on the user identity information database, determine the field information corresponding to the field name. The user identity information database includes user identity information.

[0033] In some possible implementations of the embodiments of this application, the field information corresponding to the field name is determined based on the user identity information database, including:

[0034] Extract field keywords from field names;

[0035] Based on the field mapping table, the standard field names corresponding to the field keywords are determined. The field mapping table is used to store the mapping relationship between different standard field names and field keywords.

[0036] Based on the user identity information database, determine the field information corresponding to the standard field names.

[0037] In some possible implementations of the embodiments of this application, the form filling method further includes:

[0038] If the user identity information database does not contain the field name, output the first prompt message. The first prompt message is used to indicate that the user identity information database does not contain the field name.

[0039] Displays the user identity information editing interface, which includes field names not included in the user identity information database;

[0040] Accepts edit input for field names not included in the user identity information database;

[0041] In response to edit input, the corresponding field information and field name are stored in the user identity information database.

[0042] In some possible implementations of the embodiments of this application, before extracting the form image input field information of the form to be filled from the model and outputting the field attribute information of the fields to be filled contained in the form to be filled, the form filling method further includes:

[0043] Input the form image into the form judgment model and output the judgment result information of the form image. The judgment result information is used to characterize whether the form corresponding to the form image is a form to be filled.

[0044] Extract the form image input field information from the model of the form to be filled, and output the field attribute information of the fields to be filled in the form, including:

[0045] If the form corresponding to the form image in the result information is a form to be filled, the form image input field information extraction model is used to output the field attribute information of the fields to be filled contained in the form to be filled.

[0046] In some possible implementations of the embodiments of this application, after inputting the form image into the form judgment model and outputting the judgment result information of the form image, the form filling method further includes:

[0047] If the form corresponding to the form image is not a form to be filled, a second prompt message is output. The second prompt message is used to indicate that the form corresponding to the form image is not a form to be filled.

[0048] In some possible implementations of the embodiments of this application, according to the fill type, the field information corresponding to the user identity information is filled into the fill position corresponding to the field name, including:

[0049] Displays a form fill preview window, which includes the field names of all fields to be filled in the form image and the field information corresponding to each field name; the form fill preview window is used to preview the fill result of each field to be filled.

[0050] Receive confirmation input for the filled content in the form fill preview window;

[0051] In response to the confirmation of the input, the field information corresponding to each field name is filled into the corresponding fill position of the form to be filled, according to the fill type corresponding to each field name.

[0052] Secondly, embodiments of this application provide a form filling device, including:

[0053] The extraction module is used to extract the form image of the form to be filled from the input field information model and output the field attribute information of the fields to be filled in the form. The field attribute information includes the field name, fill type and fill position of the fields to be filled.

[0054] The fill module is used to fill the fields corresponding to the user's identity information into the fill positions corresponding to the field names, based on the fill type.

[0055] In some possible implementations of the embodiments of this application, the fill type includes at least one of the following: input type, drop-down list type, upload type, click type, and submit type.

[0056] In some possible implementations of embodiments of this application, the form filling device further includes:

[0057] The training module is specifically used for:

[0058] The form image samples, prompt words, and size information of the form image samples in the training samples are input into the pre-trained visual language model, and the output is the predicted field attribute information of the fields to be filled contained in the form image samples. The predicted field attribute information is the information predicted by the pre-trained visual language model based on the prompt words and size information. The predicted field attribute information is the field attribute information of the fields to be filled contained in the form image samples, including the predicted field name, predicted filling type, and predicted filling position of the fields to be filled contained in the form image samples.

[0059] Based on the predicted field attribute information and the actual field attribute information of the fields to be filled contained in the form image sample, the visual language model is trained to obtain the field information extraction model; wherein, the actual field attribute information is the actual field attribute information of the fields to be filled contained in the form image sample, including the actual field name, actual filling type and actual filling position of the fields to be filled contained in the form image sample.

[0060] In some possible implementations of embodiments of this application, the form filling device further includes:

[0061] The acquisition module is used to acquire multiple form image samples before the training module inputs the form image samples, prompt words and size information of the form image samples in the training samples into the pre-trained visual language model and outputs the predicted field attribute information of the fields to be filled contained in the form image samples. The form image samples include at least one of the following: form images in the reference filling scenario, and form images suitable for the reference device type.

[0062] The annotation module is used to annotate the fields to be filled in each form image sample, and obtain the structured annotation data of the fields to be filled. The structured annotation data includes the field name and the fill type of the fields to be filled; and to annotate the fill position of the fields to be filled, and obtain the position coordinate information of the fill position.

[0063] The determination module is used to determine the actual field attribute information of the field to be filled from the structured annotation data and location coordinate information.

[0064] In some possible implementations of the embodiments of this application, the reference filling scenario includes at least one of the following: license plate number filling scenario, pass filling scenario, passport filling scenario, driver's license filling scenario;

[0065] The reference device type includes at least one of the following: candybar device type, foldable device type, and tablet device type.

[0066] In some possible implementations of the embodiments of this application, the structured annotation data further includes at least one of the following: the already filled content of the field to be filled, and the fill prompt words of the field to be filled.

[0067] In some possible implementations of the embodiments of this application, the fill type includes the input type;

[0068] The annotation module is specifically used for:

[0069] Determine the annotation method for the fill position based on the fill type of the field to be filled; different fill types correspond to different annotation methods.

[0070] According to the annotation method of the fill position, the fill position is annotated to obtain the position coordinate information of the fill position;

[0071] The location coordinate information includes the coordinates of the center point of the filling location.

[0072] In some possible implementations of the embodiments of this application, the annotation module is specifically used for:

[0073] When the input type is an input box, a marker box is displayed at the fill position according to the size information of the input box, and the position coordinate information of the fill position is determined according to the display position of the marker box;

[0074] When the input type is underline, a marker box is displayed at the fill position, and the position coordinates of the fill position are determined based on the display position of the marker box; the reference line of the marker box is the end of the underline or the boundary of the field to be filled.

[0075] In some possible implementations of the embodiments of this application, the training module is specifically used for:

[0076] Based on the predicted field attribute information and the actual field attribute information of the fields to be filled contained in the form image samples, the visual language model is trained for N rounds to obtain N candidate models.

[0077] Based on the validation set, the performance of each candidate model among the N candidate models is validated to obtain the accuracy of the field attribute information output by each candidate model;

[0078] The candidate model with the highest accuracy was selected as the field information extraction model.

[0079] In some possible implementations of the embodiments of this application, the determining module is specifically used for:

[0080] Before the fill module fills the field information corresponding to the user identity information into the fill position corresponding to the field name according to the fill type, the field information corresponding to the field name is determined based on the user identity information database, which includes user identity information.

[0081] In some possible implementations of the embodiments of this application, the extraction module is also used to extract field keywords of field names;

[0082] The module is specifically used for:

[0083] Based on the field mapping table, the standard field names corresponding to the field keywords are determined. The field mapping table is used to store the mapping relationship between different standard field names and field keywords.

[0084] Based on the user identity information database, determine the field information corresponding to the standard field names.

[0085] In some possible implementations of embodiments of this application, the form filling device further includes:

[0086] The output module is used to output a first prompt message when the user identity information database does not contain the field name. The first prompt message is used to prompt the user that the user identity information database does not contain the field name.

[0087] The display module is used to display the user identity information editing interface, which includes field names not included in the user identity information database.

[0088] The receiving module is used to receive edit input for field names not included in the user identity information database;

[0089] The storage module is used to respond to edit input by storing the field information and field name corresponding to the edit input into the user identity information database.

[0090] In some possible implementations of embodiments of this application, the form filling device further includes:

[0091] The judgment module is used to input the form image into the form judgment model before the extraction module extracts the form image of the form to be filled into the form image and outputs the field attribute information of the fields to be filled contained in the form to be filled. The judgment result information is used to characterize whether the form image is a form to be filled.

[0092] The extraction module is specifically used for:

[0093] If the form corresponding to the form image in the result information is a form to be filled, the form image input field information extraction model is used to output the field attribute information of the fields to be filled contained in the form to be filled.

[0094] In some possible implementations of the embodiments of this application, the output module is further configured to, after the judgment module inputs the form image into the form judgment model and outputs the judgment result information of the form image, output a second prompt information when the judgment result information indicates that the form corresponding to the form image is not a form to be filled. The second prompt information is used to prompt that the form corresponding to the form image is not a form to be filled.

[0095] In some possible implementations of the embodiments of this application, the display module is specifically used to: display a form fill preview window, the form fill preview window including the field names of all fields to be filled in the form image and the field information corresponding to each field name; the form fill preview window is used to preview the filling result of each field to be filled;

[0096] The receiving module is specifically used for:

[0097] Receive confirmation input for the filled content in the form fill preview window;

[0098] The fill module is specifically used for:

[0099] In response to the confirmation of the input, the field information corresponding to each field name is filled into the corresponding fill position of the form to be filled, according to the fill type corresponding to each field name.

[0100] Thirdly, embodiments of this application provide an electronic device including a processor and a memory, the memory storing a program or instructions executable on the processor, the program or instructions, when executed by the processor, implementing the steps of the method as described in the first aspect.

[0101] Fourthly, embodiments of this application provide a readable storage medium on which a program or instructions are stored, which, when executed by a processor, implement the steps of the method as described in the first aspect.

[0102] Fifthly, embodiments of this application provide a chip, which includes a processor and a communication interface, the communication interface and the processor being coupled together, the processor being used to run programs or instructions to implement the steps of the method as described in the first aspect.

[0103] In a sixth aspect, embodiments of this application provide a computer program product stored in a storage medium, which is executed by at least one processor to implement the steps of the method as described in the first aspect.

[0104] This application embodiment utilizes a field information extraction model to automatically extract the field name, fill type, and fill position of the fields to be filled in the form to be filled. Based on the fill type, it automatically fills the field information corresponding to the user's identity information into the position corresponding to the field name, thereby realizing automatic form filling, saving users a lot of time and effort, improving the form filling efficiency, and avoiding filling errors caused by human factors, thus improving the accuracy of the filling results. Attached Figure Description

[0105] Figure 1 Flowcharts of form filling methods provided for some embodiments of this application;

[0106] Figure 2 A schematic diagram of a form to be filled in, provided for some embodiments of this application;

[0107] Figure 3 Flowcharts of form filling methods provided for some embodiments of this application;

[0108] Figure 4 A schematic diagram of form image samples provided for some embodiments of this application;

[0109] Figure 5 Schematic diagrams of structured data provided for some embodiments of this application;

[0110] Figure 6a A schematic diagram of the marker box corresponding to the input box provided in some embodiments of this application;

[0111] Figure 6b A schematic diagram of the marker boxes and reference lines provided for some embodiments of this application;

[0112] Figure 6c A schematic diagram of the marker boxes and reference lines provided for some embodiments of this application;

[0113] Figure 6d A schematic diagram of the marker boxes and reference lines provided for some embodiments of this application;

[0114] Figure 7 Flowcharts of form filling methods provided for some embodiments of this application;

[0115] Figure 8 A schematic diagram of a user identity information editing interface provided for some embodiments of this application;

[0116] Figure 9 A schematic diagram of a form fill preview window provided for some embodiments of this application;

[0117] Figure 10 A schematic diagram of a patient information interface provided for some embodiments of this application;

[0118] Figure 11 A schematic diagram of a form filling preview window corresponding to patient information provided in some embodiments of this application;

[0119] Figure 12 Schematic diagrams of the form filling device provided for some embodiments of this application;

[0120] Figure 13 Schematic diagrams of the structure of electronic devices provided for some embodiments of this application;

[0121] Figure 14 A schematic diagram of the hardware structure of an electronic device provided for some embodiments of this application. Detailed Implementation

[0122] The technical solutions of the embodiments of this application will be clearly described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application are within the scope of protection of this application.

[0123] The terms "first," "second," etc., used in this application's specification are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class, without limiting the number of objects; for example, a first object can be one or more. Furthermore, in the specification, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects have an "or" relationship.

[0124] The terminology used in the implementation section of this application is only for explaining specific embodiments of this application and is not intended to limit this application. The terminology involved in the embodiments of this application is explained below.

[0125] Field information extraction model: This model is used to extract the field attribute information of the field to be filled. It is obtained by fine-tuning the visual language model and can be stored in the cloud.

[0126] Visual language model: A large, pre-trained model with language understanding, sequence processing, and structure recognition capabilities. This model takes images as input and text as output.

[0127] User identity information database: Used to store user identity information, such as name, ID number, mobile phone number, home address, etc. In actual storage, it stores different field names such as name, ID number, mobile phone number, home address and corresponding field information.

[0128] Prompt: A prompt word is text or instructions provided by the user to an artificial intelligence (AI) assistant, designed to guide the AI ​​assistant program to generate specific, desired output.

[0129] AI assistants are software tools or platforms developed based on artificial intelligence technology. They enable human-computer interaction through technologies such as natural language processing and machine learning, helping users complete various tasks and improving efficiency and convenience.

[0130] The form filling method provided in this application can be applied to online form filling scenarios. For example, it can be applied when a user goes to a hospital for treatment and needs to register patient information online, such as name, ID number, home address, and mobile phone number. It can also be applied when a user registers for an application online and needs to register their name, email address, ID number, and other information. Furthermore, it can be applied when a user sends a package and needs to add recipient and delivery address information. Of course, it can also be applied to other scenarios that require online entry of user identity information.

[0131] The form filling method, apparatus, and electronic device provided in this application will be described in detail below with reference to the accompanying drawings and through specific embodiments and application scenarios.

[0132] It should be noted that the form filling method provided in this application can be executed by electronic devices with data processing and display functions, such as mobile phones, tablets, laptops, PDAs, and in-vehicle electronic devices. Some embodiments of this application use electronic devices as the executing entity to illustrate the form filling method provided in this application.

[0133] Figure 1 A flowchart of a form filling method provided for some embodiments of this application, the form filling method may include the following steps 110-120.

[0134] Step 110: Extract the form image input field information model of the form to be filled, and output the field attribute information of the fields to be filled in the form. The field attribute information includes the field name, fill type and fill position of the fields to be filled.

[0135] The form to be filled here can be an online form. For example, the form to be filled can be a form for user information that a user needs to fill in when applying for a driver's license, a form for registering patient information when a user goes to the hospital, a form for user information that a user needs to fill in when registering for an application, or an online form that can be applied to other scenarios, such as a form for filling in user information when applying for a pass, license plate number, or passport.

[0136] For example, the form to be filled can be a form displayed on different types of devices, such as a form displayed on a foldable device, a form displayed on a candybar device, or a form displayed on a tablet device. That is, this embodiment can automatically fill online forms displayed on any device, and can also automatically fill forms in any scenario, thereby improving the universality of the form filling solution.

[0137] The form image can be a screenshot obtained by an electronic device of the form to be filled. For example, when an electronic device receives a screenshot input from a user, it can take a screenshot of the form to be filled to obtain the form image. The screenshot input is used to trigger or control the electronic device to take a screenshot of the form to be filled. For example, the screenshot input includes, but is not limited to: touch input by the user using a finger or stylus to the screen of the electronic device, or voice commands input by the user, or specific gestures input by the user, or other feasible inputs. The specific input can be determined according to actual usage needs, and this application embodiment does not limit it. Specific gestures in this application embodiment can be any one of a single-click gesture, a swipe gesture, a drag gesture, a pressure-recognition gesture, a long-press gesture, an area-change gesture, a double-press gesture, or a double-tap gesture; the click input in this application embodiment can be a single-click input, a double-tap input, or any number of clicks, and can also be a long-press input or a short-press input. For example, the screenshot input can be a long-press input by the user using two fingers at any position on the screen.

[0138] For example, such as Figure 2 As shown, the electronic device 200 displays a form to be filled in for the new address on the interface 201. The screenshot input is a long press input performed by the user using the thumb and index finger of the right hand 202 on the interface 201. After receiving the user's long press input on the interface 201, the electronic device 200 can take a screenshot of the interface 201 to obtain the form image of the form to be filled.

[0139] The field information extraction model is used to extract the field attribute information of the fields to be filled in the form to be populated. This model takes the form image as input and outputs the field attribute information of the fields to be filled in the form. For example, this field information extraction model can be obtained by training a visual language model. The specific training process of the visual language model can be found in the following embodiment.

[0140] The field names to be filled can include, but are not limited to, name, mobile phone number, ID card number, address, etc. In practical applications, the field names can include only first-level fields such as name, mobile phone number, ID card number, etc., or they can include two or more levels of fields. Taking two-level fields as an example, the field names can include "Household Registration #Province", "Household Registration #City", "Applicant 1 # Name", etc.

[0141] The fill type is used to characterize how the field to be filled is filled, and may include, but is not limited to, input type, drop-down list type, upload type, click type, submit type, etc.

[0142] The input type refers to the field information corresponding to the field name that is directly entered. For example, the fill type for names, mobile phone numbers, and ID card numbers is usually an input type, meaning that the user directly enters the information such as name, mobile phone number, and ID card number. For example, the input type can include an input box, that is, the corresponding field information is entered in the input box. As another example, the input type can also include an underscore, that is, the corresponding field information is entered on the underscore.

[0143] The dropdown list type indicates that field information is selected by field name via a dropdown list. For example, for user gender, an electronic device can display "Male" and "Female" as dropdown lists, and when filling in the user's gender, the corresponding gender can be selected from the dropdown list.

[0144] Upload types can be applied to image uploads, table uploads, etc., and usually correspond to the "Upload" control. For example, when it is necessary to upload a photo of a user's ID card, the "Upload" control can be triggered to upload the photo and complete the information entry.

[0145] Click-type indicates that field information is selected by clicking on a field name. For this type of field name, multiple field information options are usually provided, and selection needs to be made based on the actual situation. For example, to determine whether a user is Han Chinese, two controls are displayed: a "Yes" control and a "No" control. Electronic devices can select "Yes" or "No" by clicking.

[0146] Submission types typically correspond to submission controls. For example, by triggering a submission control, a filled form can be submitted, enabling automatic form filling and submission, reducing user operations.

[0147] The fill position is where the field information is filled in. By specifying the fill position, the field information can be filled in accurately.

[0148] For example, when an electronic device receives a screenshot of a form to be filled from a user, it can take a screenshot of the form and input the screenshot image into a field information extraction model. The field information extraction model then outputs the field name, fill type, and fill position of the fields to be filled in the form, providing a basis for the automatic filling of subsequent forms.

[0149] Step 120: According to the fill type, fill the field information corresponding to the user identity information into the fill position corresponding to the field name.

[0150] The fields here can be information associated with user identity information, such as, but not limited to, the user's name, ID number, bank card number, email address, address, passport number, mobile phone number, password, etc. User identity information can be stored in a designated location for easy retrieval later; for example, it can be stored in a user identity information database.

[0151] In some embodiments, prior to step 120, the form filling method may further include the following steps:

[0152] Based on the user identity information database, determine the field information corresponding to the field name. The user identity information database includes user identity information.

[0153] For example, the user identity information database can be stored in a local Trusted Execution Environment (TEE) to ensure the security of user identity information. For example, the user identity information database can associate and store different field names with their corresponding field information for easy retrieval later.

[0154] For example, an electronic device can match the field name of the field to be filled output by the field information extraction model with the user identity information database one by one to obtain the field information that matches the field name. For example, if the field name of the field to be filled is "user name", and the user identity information database also contains the field name "user name", then the field information corresponding to "user name" in the user identity information database can be used as the search result.

[0155] Considering that there are multiple fields to be filled, for example, when an electronic device searches for user identity information based on field names, it can use multiple search threads to execute in parallel. Each search thread corresponds to one field name, thereby allowing multiple field names to be searched simultaneously, which improves the efficiency of field information retrieval.

[0156] For example, electronic devices can fill the field information corresponding to the user's identity information into the fill position corresponding to the field name according to the fill type of each field to be filled, without requiring the user to manually enter each one, saving the user's time and effort in filling out the form, and improving the form filling efficiency and the accuracy of the filling results.

[0157] For example, when the fill type is input type and the input type is an input box, the electronic device can directly input the found field information into the corresponding input box. For instance, for the field name "User Name", the electronic device can directly fill the found "Zhang San" into the corresponding input box.

[0158] For example, when the fill type is click-based, the electronic device can fill in the content by clicking the option that matches the search result from multiple candidate contents based on the found field information. For instance, if the user's gender is "female", the electronic device can click the "female" control to fill in the user's gender.

[0159] This application embodiment utilizes a field information extraction model to automatically extract the field name, fill type, and fill position of the fields to be filled in the form. Based on the field name, it directly retrieves the field information corresponding to the field name from the user identity information database. Based on the fill type, it automatically fills the field information corresponding to the user identity information into the position corresponding to the field name, realizing automatic form filling. This saves users a lot of time and effort, improves the form filling efficiency, and avoids filling errors caused by human factors, ensuring the accuracy of the filling results.

[0160] Figure 3 Flowcharts of form filling methods provided for some embodiments of this application Figure 3 and Figure 1 The difference is that, Figure 3 It also includes steps 310-320.

[0161] Step 310: Input the form image samples, prompt words, and size information of the form image samples in the training samples into the pre-trained visual language model, and output the predicted field attribute information of the fields to be filled contained in the form image samples.

[0162] The form image samples here can be image samples obtained by cropping multiple forms to be filled, used to train the visual language model. This visual language model can be a pre-trained large model. The pre-trained visual language model already has language understanding, sequence processing, and structure recognition capabilities. On this basis, further training the visual language model using form image samples can improve the model's perception and understanding capabilities in form parsing, and can also greatly save the model's training time, optimizing the model's training efficiency and training effect.

[0163] A prompt is used to guide a visual language model to generate a specific and desired output. For example, in this embodiment, a prompt is used to guide the visual language model to output the fields to be filled and the field attribute information of the fields to be filled in a form image sample in a specific format.

[0164] For example, the prompt may include instruction information, specification information for the model output format, and example information for the output format. The instruction information tells the model the task to be performed, such as "List the fields to be filled in this image." The specification information for the model output format instructs the model to identify and output the fields to be filled and their attribute information in the image according to the given format. The example information for the output format provides an example of the output format, enabling the model to output the prediction result according to the given output format.

[0165] In some embodiments, the output format may be as follows:

[0166] Field name: Use <ref> {Field Name}< / ref>

[0167] Fill position: Use <box> {Location coordinates}< / box> The coordinate format is defined as {(x0,y0),(x1,y1)}, representing the coordinates of the top-left and bottom-right corners of the fill position.

[0168] Fill type: Use <type> {Fill type}< / type>

[0169] For example, data: <ref> ID card< / ref> <box> (68,205),(931,249)< / box> <type> input< / type> This refers to the output result corresponding to the ID card field. <ref> and< / ref> The spaces between the fields represent field names, which can be names, phone numbers, addresses, etc. <type> and< / type> The spaces between them represent the fill type, such as: input type, click type, dropdown type, upload type, etc. <box> and< / box> The coordinates between (68, 205) represent the coordinates of the top-left corner of the fill position, and (931, 249) represent the coordinates of the bottom-right corner. Based on the coordinates of the top-left and bottom-right corners, the coordinates of the center point of the fill position can be determined, allowing for accurate filling of the field information.

[0170] Of course, other output formats can be used in practical applications. This embodiment does not limit the specific form of the output format, as long as it contains the field attribute information of the field to be filled.

[0171] The size information of a form image sample may include, for example, its width and height. The size information of a form image sample differs for devices of different sizes. This size information can be pre-obtained using annotation tools; for instance, annotation tools can be used to annotate the form image sample to obtain its size information.

[0172] The predicted field attribute information is the information predicted by the pre-trained visual language model based on prompt words and size information. The predicted field attribute information is the field attribute information of the fields to be filled contained in the form image sample. The predicted field attribute information includes the predicted field name, predicted fill type and predicted fill position of the fields to be filled contained in the form image sample.

[0173] During the model training phase, electronic devices can input form image samples, prompts, and the size information of the form image samples into a pre-trained visual language model, which then outputs the predicted field attribute information of the fields to be filled contained in the form image samples.

[0174] Step 320: Based on the predicted field attribute information and the actual field attribute information of the fields to be filled contained in the form image samples, train the visual language model to obtain the field information extraction model.

[0175] The actual field attribute information refers to the actual field attribute information of the field to be filled. For example, the actual field attribute information may include the actual field name, actual fill type, and actual fill position of the field to be filled contained in the form image sample. The actual field attribute information can be obtained through pre-annotation, and the specific annotation process can be found in the following embodiment.

[0176] Electronic devices can train a visual language model based on the predicted field attribute information and the actual field attribute information of each field to be filled, thus obtaining a field information extraction model.

[0177] For example, a loss function value can be calculated based on the predicted field attribute information and the actual field attribute information, and the visual language model can be trained based on the loss function value. For instance, training can end when the loss function value shows a decreasing trend and tends to stabilize. Alternatively, training can end when the preset number of iterations is reached.

[0178] For example, after training, the trained field information extraction model can be tested using a test set to assess its performance. If the performance metrics meet the requirements, the model can be directly used for subsequent field attribute information prediction. If the performance metrics do not meet the requirements, training needs to continue until the performance metrics meet the requirements. The performance metrics here can include the recall and precision of the field information extraction model. For instance, the number of fields to be filled in the test set, the number of fields to be filled extracted by the field information extraction model, and the field attribute information of each field to be filled extracted by the model can be statistically analyzed. Combining the actual number of fields to be filled in the test set and the actual field attribute information of each field to be filled, the recall and precision of the field information extraction model can be determined. Based on the recall and precision, the performance of the field information extraction model can be tested.

[0179] This embodiment, based on the large visual language model, uses form image samples, prompt words, and size information of the form image samples to fine-tune the large visual language model, thereby improving training efficiency while ensuring the accuracy of the results.

[0180] In some embodiments, prior to step 310, the form filling method may further include steps 3101-3104.

[0181] Step 3101: Collect multiple form image samples.

[0182] The form image samples include at least one of the following: form images for reference filling scenarios, and form images applicable to reference device types.

[0183] For example, the reference fill-in scenarios may include at least one of the following: license plate number fill-in scenario, pass fill-in scenario, passport fill-in scenario, and driver's license fill-in scenario. In addition to basic user information such as user name, ID number, mobile phone number, and address, different fill-in scenarios also include specific information corresponding to the fill-in scenario. For example, the license plate number fill-in scenario may also include the license plate number; the pass fill-in scenario may also include uploading a user image; the passport fill-in scenario may also include uploading the user's personal page from their household registration book; and the driver's license fill-in scenario may also include uploading a driver's license application form.

[0184] For example, the reference device type may include at least one of the following: candybar device type, foldable device type, and tablet device type. Different types of devices have different sizes, and the corresponding form image sample sizes are also different.

[0185] This embodiment can collect form images applicable to different scenarios and different device sizes as training samples, increasing the diversity of training samples. This allows the trained field information extraction model to be applicable to forms in various scenarios and on various device types, thereby achieving automatic filling of various forms.

[0186] Step 3102: Annotate the fields to be filled in each form image sample to obtain structured annotation data for the fields to be filled.

[0187] The structured annotation data includes the field name and fill type of the field to be filled. In some embodiments, the structured annotation data may further include at least one of the following: the already filled content of the field to be filled, and the fill prompt text for the field to be filled. The already filled content refers to the content of the field to be filled that has already been filled. The already filled content can be automatically filled by the electronic device or pre-filled by the user. In practical applications, if it has not been pre-filled, the already filled content is "None". The fill prompt text for the field to be filled is used to suggest the content to be entered. For example, for the field to be filled "Contact Information", the fill prompt text may include "Please enter your mobile phone number". If there is no prompt, the fill prompt text is "None".

[0188] For example, annotation tools can be used to annotate the fields to be filled in each form image sample to obtain structured annotated data. Annotation tools such as Labelme can be used.

[0189] In some embodiments, structured annotation data can be in the following format: A|B|C|D, where A represents the field name. If the field name includes a special symbol such as "*" before or after it, the structured annotation data must also include the special symbol "*", for example, "*ID number". B represents the content already filled in the field to be filled; if it has not been pre-filled, B is "none". C represents the fill prompt for the field to be filled; if there is no fill prompt, C is "none". D represents the fill type, which may include, but is not limited to, "input", "click", "dropdown", etc.

[0190] Figure 4 An exemplary diagram of a form image sample is provided, which includes four fields to be filled: username (4011), phone number (4012), detailed address (4013), and verification code (4014), as well as a "submit" button (4015). By labeling each field, we can obtain... Figure 5 The structured data shown is 4016. For example, for username 4011, the structured data is: username|none|none|input; for phone number 4012, the structured data is: phone number|none|none|input; for detailed address 4013, the structured data is: detailed address|none|none|input; and for verification code, the structured data is: verification code|none|none|input.

[0191] Specifically, for cases containing multi-level fields, structured data can take the following form: key1#key2#......#keyN|filled content|fill hint|fill type. Here, key represents each level of field. For example, for "household registration#province", key1 is the household registration number, and key2 is the province. For image CAPTCHAs, key1 can be the CAPTCHA itself, and key2 can be the CAPTCHA content.

[0192] In some embodiments, if the form image sample does not directly display the field to be filled, semantic parsing can be performed on the form image sample, and the field to be filled can be determined based on the semantic parsing results. Alternatively, if the form image sample contains the prompt "Please enter...", all fields after "Please enter..." can be used as the key. For example, if the form image sample includes the prompt "Please enter a detailed address", then "detailed address" can be used as the key.

[0193] In some embodiments, the same key may need to be labeled with multiple boxes. For example, in the case of license plate numbers, structured data can be in the following form: "field#number|filled content|fill prompt|fill type". For numbers, one number can correspond to one box. For example, "license plate number#□□□□□□□|none|none|input".

[0194] In practical applications, in addition to labeling fields to be filled, buttons can also be labeled, for example, in the same way as the fields to be filled described above. Figure 4 The "Submit" button 4015 in the image yields structured data: Submit | None | None | Submit.

[0195] In some embodiments, for SMS verification codes, the send input field can be labeled as a button, and the labeling method is the same as... Figure 4 The "Submit" button in 4015 is similar.

[0196] In some embodiments, for an agreement checkbox, if it's a "I agree to xx" checkbox and the key is short, it can be directly labeled based on the key. If the key is long, a portion of the statement can be extracted as the key. For example, if the agreement only contains "User Service Agreement" and has limited content, the key could be "I agree to User Service Agreement," and the complete structured data would be "I agree to User Service Agreement | None | None | Click." If the agreement includes "Privacy Policy," "User Service Agreement," "Credit Card Authorization Agreement," etc., and has more content, the key can be a portion of it, for example, "I agree to Privacy Policy," and the complete structured data would be "I agree to Privacy Policy | None | None | Click."

[0197] In some embodiments, for the camera button, if there are no other field descriptions, the structured data can be in the form of "Photo|None|None|Upload". If there are other field descriptions, it can be in the form of "Field#Photo|None|None|Upload". For example, if the camera button contains other description information "Upload Business License", then the structured data would be "Upload Business License#Photo|None|None|Upload".

[0198] Step 3103: Mark the filling position of the field to be filled to obtain the position coordinate information of the field to be filled.

[0199] Marking the fill positions of the fields to be filled makes it easier to determine the fill positions of the field information later, so that the field information can be filled accurately.

[0200] Taking the example that the fill type includes the input type, step 3103 above may include the following steps:

[0201] Determine the annotation method for the fill position based on the fill type of the field to be filled; different fill types correspond to different annotation methods.

[0202] According to the annotation method of the fill position, the fill position is annotated to obtain the position coordinate information of the fill position;

[0203] The location coordinate information includes the coordinates of the center point of the filling location.

[0204] Labeling the fill location is to more accurately determine its coordinates, thereby enabling more accurate filling of field information and improving the accuracy of the fill result.

[0205] The location coordinate information here can include the coordinates of the center point of the fill position; that is, when filling field information, the field information is filled to the center of the fill position. For example, the location coordinate information can also include the coordinates of the endpoints of the fill position, such as the coordinates of the top-left endpoint and the bottom-right endpoint. The electronic device can then determine the specific fill position based on the coordinates of the top-left and bottom-right endpoints. For example, the field information can be filled to the center of the line connecting the top-left and bottom-right endpoints.

[0206] In this embodiment, different fill types can correspond to different annotation methods, thereby improving the flexibility of the annotation method and making it applicable to various scenarios. For example, the annotation methods for the corresponding fill positions are different for input boxes and underlines.

[0207] For example, when the input type is an input box, a marker box is displayed at the fill position according to the size information of the input box, and the position coordinate information of the fill position is determined according to the display position of the marker box.

[0208] The input box described above can be a rectangle or any other shape. Taking a rectangle as an example, the input box's dimensions can include its width and height. The width is the horizontal length of the input box, and the height is the vertical length. When the input type is an input box, a marker box can be displayed at the fill position according to the input box's dimensions. The marker box can be a box of other colors or with other line styles, used to mark the fill position.

[0209] like Figure 6a As shown, Figure 6a An exemplary schematic diagram of a form image sample 601 is provided, which includes three fields to be filled: username 602, phone number 603, and detailed address 604. In actual applications, the form image sample 601 may also contain other information. Each field to be filled corresponds to an input box, indicating that the corresponding field information is to be filled into the input box. For example, username 602 corresponds to input box 6021, phone number 603 corresponds to input box 6031, and detailed address 604 corresponds to input box 6041. After the electronic device determines the field information of each field to be filled, it can directly input the field information into the corresponding input box. For example, the electronic device can fill the field information corresponding to username 602 into input box 6021, the field information corresponding to phone number 603 into input box 6031, and the field information corresponding to detailed address 604 into input box 6041.

[0210] For easy distinction, Figure 6a The display shows the annotation results for some filled positions. For example, for username 602, before annotating input box 6021, the line style of input box 6021 is the same as that of input boxes 6031 and 6041. After annotation, the line style of input box 6021 is thickened, and the thickened box is the annotation box for input box 6021. The annotation process for input boxes 6031 and 6041 is similar. Thus, each input box is annotated with a thickened annotation box. The size information of the annotation box is exactly the same as the size information of the input box.

[0211] For example, when the input type is underscore, a marker box is displayed at the fill position, and the position coordinates of the fill position are determined based on the display position of the marker box. The reference line for the marker box is either the end of the underscore or the boundary of the field to be filled.

[0212] In some embodiments, the reference line of the marker box can be the end of an underscore. For example... Figure 4 As shown, Figure 4The input type is underscore. To more accurately fill in the field information, a marker box can be displayed at the fill position to indicate that the field information needs to be filled into the marker box. The position of the marker box can be flexibly determined. For example, in some embodiments, such as Figure 6b As shown, the reference line 606 for the marker box 605 is the end of the underscore 607. For example, the straight line containing the right border of the marker box 605 can be used as the reference line; in this case, the right sides of each marker box 605 are aligned. For example, as... Figure 6c As shown, the straight line containing the left border of the marker box 605 can also be used as the reference line 606. In this case, the left sides of each marker box 605 are aligned.

[0213] In some embodiments, the reference line for the marker box can also be the boundary of the filled field. This boundary can be either the left or right boundary of the filled field. Considering the varying lengths of the filled fields, when multiple fields to be filled are arranged sequentially, the field with the longest name can be selected, and then the marker box can be displayed using the right or left boundary of that field as the reference line. For example... Figure 6d As shown, the phone number 4012 and the detailed address 4013 are both 4 characters long, and the username 4011 and the verification code 4014 are both 3 characters long. Therefore, in some embodiments, the marker box 605 can be displayed with the right boundary of the "code" or the right boundary of the "address" as the reference line 606.

[0214] In particular, if the field to be filled contains a prompt word, the prompt word must be included when marking the fill position. For example, if the prompt word is on the underline, the marker box must contain the prompt word, that is, the prompt word must be inside the marker box.

[0215] This embodiment can mark the fill position based on the input type, so that the electronic device can accurately determine the fill position of the field information based on the display position of the mark box, thereby achieving accurate form filling.

[0216] Step 3104: Determine the structured annotation data and location coordinate information as the actual field attribute information of the field to be filled.

[0217] Based on the structured annotation data and location coordinate information, the actual field attribute information of the field to be filled can be obtained: field name, fill type, and fill position. The fill position can be represented in the following format: {(x0,y0),(x1,y1)}, where (x0,y0) represents the coordinates of the top-left corner of the marker box, and (x1,y1) represents the coordinates of the bottom-right corner of the marker box.

[0218] This embodiment automatically labels each form image sample, which can obtain the actual field attribute information of each field to be filled in each form image sample, providing a more accurate basis for subsequent model training.

[0219] It is understandable that visual language models are trained according to a certain data format. In order to enable the pre-trained visual language model to directly and efficiently apply its internal representation to the fine-tuning task and reduce the extra burden on the model to learn new data paradigms during the fine-tuning stage, the structured data and position coordinate information mentioned above can be converted based on the data format corresponding to the pre-trained model, for example.

[0220] For example, the data format corresponding to the visual language model is {"category":"","images":[{"width":"","path":"","height":""}],"conversations":[{"from":"human","value":""},{"from":"gpt","value":""}],"llava":""}, where "category" represents the task category, for example, in this embodiment, the task category is table filling. "images":[{"width":"","path":"","height":""} represents the attribute information of the form image sample, including the width, storage path, and height of the form image sample. "conversations" represents a list containing two dictionaries: {"from":"human","value":""} represents the prompt words, and {"from":"gpt","value":""} represents the conversion result. "llava" represents the type of the large model; typically, "llava" is 1, indicating that a visual language model is used. In this context, {"category":"","images":[{"width":"","path":"","height":""}],{"from":"human","value":"" represents the input format of the model, and {"from":"gpt","value":""} represents the output format of the model.

[0221] For example, for a given form image sample, the form image sample, prompt text, and form image sample size information can be converted into the following input format: {"category":"table","images":[{"width":"747","path":" / data / picture / 1.png","height":"845"}],"conversations":[{"from":"human","id":"1","value":" Please list all the fields to be filled in this image and output the result using... <ref> {Field Name}< / ref> <box> {Fill position}< / box> <type> {Fill type}< / type> This indicates that the {fill position} is represented in the format {(x0,y0),(x1,y1)}.

[0222] To improve the accuracy of the field information extraction model, Figure 7 An exemplary flowchart of a form filling method is provided. Figure 7 and Figure 3 The difference is that, Figure 3 Step 320 in the middle can be refined as follows: Figure 7 Steps 710-730 in the text.

[0223] Step 710: Based on the predicted field attribute information and the actual field attribute information of the fields to be filled contained in the form image samples, perform N rounds of training on the visual language model to obtain N candidate models.

[0224] N is an integer greater than or equal to 1. For example, when N = 1, it represents one round of training for the visual language model. The number of form image samples used in one round of training can be set as needed; for example, one round of training can use all the training samples in the training set or only a portion of the training samples. When N is greater than 1, it represents multiple rounds of training for the visual language model. The number of training samples used in each round can be the same or different. Taking N = 2 as an example, the second round of training is based on the model obtained in the first round. After each round of training, a candidate model can be obtained, resulting in N candidate models, each with different parameters.

[0225] For example, the training step size of the visual language model can be set to 1e-5, and the batch size can be set to 1.

[0226] Step 720: Perform performance verification on each of the N candidate models based on the validation set, and obtain the accuracy of the field attribute information output by each candidate model.

[0227] For example, the labeled dataset can be divided into a training set, a validation set, and a test set. For instance, 80% of the dataset can be used as the training set, 10% as the validation set, and 10% as the test set. The training set, validation set, and test set should cover as many form scenarios as possible to improve the model's robustness.

[0228] For example, a validation set can be used to validate each candidate model. For instance, the validation image corresponding to the validation set can be input into the candidate model, the field attribute information of the field to be filled contained in the validation image can be output, and the field attribute information of the field to be filled output by the candidate model can be compared with the actual field attribute information of the field to be filled contained in the validation image to determine the accuracy of the field attribute information output by the candidate model.

[0229] Step 730: The candidate model with the highest accuracy is determined as the field information extraction model.

[0230] For example, the candidate model with the highest accuracy can be identified as the field information extraction model.

[0231] For example, the recall rate of a candidate model can be calculated based on the number of fields to be filled and the actual number output by the candidate model. Based on the recall rate and accuracy, a field information extraction model can be determined. For instance, a candidate model with a recall rate greater than or equal to a recall threshold and an accuracy rate greater than or equal to an accuracy threshold can be determined as the field information extraction model. For example, this field information extraction model can be deployed in the cloud.

[0232] For example, the electronic device can also validate the visual language model using a validation set after each iteration of a certain number of steps, such as 100 steps. If the accuracy is less than the accuracy threshold or the recall is less than the recall threshold, the hyperparameters of the visual language model, such as the training step size and batch size, can be adjusted. Then, the above process can continue based on the adjusted hyperparameters.

[0233] This embodiment can train the visual language model in multiple rounds to obtain multiple candidate models. The model with the best performance is selected for subsequent field attribute information extraction, which can improve the accuracy of the extraction results and thus achieve accurate form filling.

[0234] Based on the final field information extraction model determined above, inputting a form image containing name, mobile phone number, and address into the field information extraction model will output the results shown in Table 1.

[0235] Table 1

[0236] <ref> Name< / ref> <box> (68,205),(931,249)< / box> <type> input< / type> <ref> Phone number< / ref> <box> (68,293),(931,337)< / box> <type> input< / type> <ref> address< / ref> <box> (68,382),(931,426)< / box> <type> input< / type>

[0237] Given that field names often have multiple synonyms in real-world business scenarios, such as "telephone number," "mobile phone number," and "contact information," although the descriptions differ, the actual content entered often points to the user's mobile phone number. To improve the accuracy of the matching results between field names and the user identity information database, in some embodiments, the aforementioned "determining the field information corresponding to the field name based on the user identity information database, where the user identity information database includes user identity information" may include the following steps:

[0238] Extract field keywords from field names;

[0239] Based on the field mapping table, the standard field names corresponding to the field keywords are determined. The field mapping table is used to store the mapping relationship between different standard field names and field keywords.

[0240] Based on the user identity information database, determine the field information corresponding to the standard field names.

[0241] The field keywords here can include part or all of the field names. For example, keyword extraction tools can be used to extract field keywords from field names, or algorithms such as semantic analysis or models can be used to extract field keywords from each field name.

[0242] Table 2 provides an example of the relationships between some standard field names and field keywords. In practical applications, relationships between more standard field names and field keywords can be included. The extracted field keywords are matched with the field keywords in the mapping relationship to obtain the corresponding standard field names. The relationship between these standard field names and field keywords is established based on regular expressions.

[0243] Table 2

[0244]

[0245]

[0246] For example, when extracting field names containing keywords such as "citizenship," "citizen," or "identity," they can be uniformly mapped to the standard field name "citizen identity number." Similarly, when extracting field names containing keywords such as "birth," "birth," "birthday," or "birth date," they can be uniformly mapped to the standard field name "date of birth." This significantly improves data consistency and lays a solid foundation for subsequent processing and analysis.

[0247] For example, an electronic device can match standard field names with a user identity information database to obtain field information that matches the standard field names. This field information is the field information of the field name corresponding to the standard field name in the form to be filled.

[0248] In this embodiment, before matching the field name with the user identity information database, the field keywords of the field name are extracted first. Then, combined with the field mapping table, the standard field name corresponding to the field keyword is determined. The standard field name is then matched with the user identity information database to determine the field information, which helps to improve the matching efficiency and the accuracy of the matching results.

[0249] It is understandable that the information stored in the user identity information database is limited. In practical applications, there may be cases where some field names fail to match, meaning that the user identity information database does not contain those field names. Based on this, in some embodiments, the form filling method may further include the following steps:

[0250] If the user identity information database does not contain the field name, output the first prompt message. The first prompt message is used to indicate that the user identity information database does not contain the field name.

[0251] Displays the user identity information editing interface, which includes field names not included in the user identity information database;

[0252] Accepts edit input for field names not included in the user identity information database;

[0253] In response to edit input, the corresponding field information and field name are stored in the user identity information database.

[0254] This embodiment does not limit the output method of the first prompt information. For example, it can be prompted through a pop-up window, through voice prompts, or through a combination of pop-up windows and voice prompts. The first prompt information may include field names not included in the user identity information database.

[0255] For example, when an electronic device detects that a certain field name is not contained in the user information database, in addition to outputting the first prompt message, it can also display... Figure 8 The user identity information editing interface 800 shown can include field names not included in the user identity information database. It can also include field names corresponding to other basic user information.

[0256] Taking the example that the user identity information database does not contain the field name "address" 801, the user identity information editing interface 800, in addition to containing "address" 801, also contains other basic information of the user such as name 802, phone number 803, gender 804, and ethnicity 805. The name 802, phone number 803, gender 804, and ethnicity 805 in the user identity information editing interface 800 are all filled with corresponding field information, which is obtained by searching the user identity information database, and the user can confirm this.

[0257] The aforementioned editing input is used to supplement the missing field names in the user identity information database. For example, this editing input can include touch input from the user using a touch device such as a finger or stylus to input the missing field names. This touch input can include, for example, single-click input, double-click input, or any number of clicks, and can also be long-press input or short-press input. For instance, the editing input described above could be a long-press input of the missing field name by the user.

[0258] After receiving the user's edit input for the missing field name, the electronic device puts the missing field name into edit mode, at which point the user can edit the field information of the missing field name.

[0259] For example, such as Figure 8 As shown, the user identity information editing interface 800 may also include an "edit" control 806. By touching the "edit" control 806, the user can edit the field information of all field names in the user identity information editing interface 800. For example, the user can add specific content to the "address" 801, or modify the field information in name 802, phone number 803, gender 804, and ethnicity 805. Through touching the "edit" control 806, the user can edit the entire user identity information editing interface 800.

[0260] For example, such as Figure 8 As shown, the user identity information editing interface 800 may also include a "confirm" control 807. When a user touches the "confirm" control 807, the information on the user identity information editing interface 800 can be saved. For example, when the electronic device receives the user's input to the "confirm" control 807, it can save the information on the user identity information editing interface 800 to the user identity information database for easy retrieval later.

[0261] In this embodiment, when a user's information is not found in the user identity information database, a prompt message can be output and a user identity information editing interface can be displayed to guide the user to add the missing information and save it to the user identity information database for later use, thereby improving the efficiency of form filling.

[0262] In some embodiments, prior to step 110, the form filling method may further include the following steps:

[0263] Input the form image into the form judgment model and output the judgment result information of the form image. The judgment result information is used to characterize whether the form corresponding to the form image is a form to be filled.

[0264] Accordingly, step 110 above may include the following steps:

[0265] If the form corresponding to the form image in the result information is a form to be filled, the form image input field information extraction model is used to output the field attribute information of the fields to be filled contained in the form to be filled.

[0266] The aforementioned form image can be an image of the form currently displayed on the electronic device, which can be obtained by taking a screenshot of the form displayed on the electronic device.

[0267] The aforementioned form judgment model is used to determine whether the form currently displayed on the electronic device is a form to be filled. For example, this form judgment model can be a binary classification model, outputting "yes" or "no," where "yes" indicates that the form currently displayed on the electronic device is a form to be filled, and "no" indicates that the form currently displayed on the electronic device is not a form to be filled. For example, this form judgment model can employ models such as support vector machines or logistic regression.

[0268] For example, if the form corresponding to the form image is a form to be filled, the electronic device can extract the form image input field information model and output the field attribute information of the fields to be filled contained in the form to be filled.

[0269] For example, if the form corresponding to the form image in the determination result information is not the form to be filled, a second prompt message is output.

[0270] The second prompt message is used to indicate that the form corresponding to the form image is not a form to be filled. For example, the second prompt message may include "The form cannot be filled" or "The currently displayed form is not a form to be filled". This embodiment does not limit the output method of the second prompt message. For example, it can be output by voice, by pop-up window, or by a combination of voice and pop-up window.

[0271] In this embodiment, before scheduling the field information extraction model, the form image is first judged using the form judgment model. If it is determined that the currently displayed form is a form to be filled, the form image is then input into the field information extraction model to execute the subsequent process. If it is determined that the currently displayed form is not a form to be filled, the second prompt information is directly output, thereby avoiding invalid operations by the user.

[0272] In some embodiments, step 120 above may include the following steps:

[0273] Displays a form fill preview window, which includes the field names of all fields to be filled in the form image and the field information corresponding to each field name; the form fill preview window is used to preview the fill result of each field to be filled.

[0274] Receive confirmation input for the filled content in the form fill preview window;

[0275] In response to the confirmation of the input, the field information corresponding to each field name is filled into the corresponding fill position of the form to be filled, according to the fill type corresponding to each field name.

[0276] For example, such as Figure 9 As shown, after matching the field names with the user identity information database, the electronic device can display... Figure 9 The form fill preview window 900 shown can float above the interface 201 of the form to be filled. This form fill preview window 900 can include the names of successfully matched fields and the corresponding field information. Figure 9 Taking the successfully matched field names including name 902, mobile phone number 903, and detailed address 904 as an example, the corresponding field information for each field name is displayed.

[0277] Users can check the field information displayed in the form fill preview window 900 and correct any errors. The electronic device can then store the correct information in the user identity information database and delete the previously stored incorrect information, ensuring the accuracy of the form filling.

[0278] The fill content confirmation input is used to confirm the field information displayed on the form fill preview window 900, that is, it represents the user's approval of the field information displayed on the form fill preview window 900. For example, the above-mentioned fill content confirmation input includes, but is not limited to: touch input from the user via a finger or stylus to the form fill preview window 900, or voice commands input by the user, or specific gestures input by the user, or other feasible inputs. The specific input can be determined according to actual usage needs, and this application embodiment does not limit it. Specific gestures in this application embodiment can be any one of a single-click gesture, a swipe gesture, a drag gesture, a pressure-recognition gesture, a long-press gesture, an area-change gesture, a double-press gesture, or a double-tap gesture; the click input in this application embodiment can be a single-click input, a double-tap input, or any number of clicks, and can also be a long-press input or a short-press input. For example, the above-mentioned fill content confirmation input can be a voice command or gesture command input by the user.

[0279] For example, when an electronic device receives a voice or gesture command from a user, it can automatically fill in the corresponding field information into the corresponding fill position of the form based on the fill type corresponding to each field name. For instance, as shown... Figure 9 As shown, the electronic device can fill the field information "Zhang San" into the fill position 9021 corresponding to the name 902. Similarly, it can fill the field information "134***888" into the fill position 9031 corresponding to the mobile phone number 903. Similarly, it can fill the field information "A Community B Unit 1203" into the fill position 9041 corresponding to the address 904.

[0280] For example, such as Figure 9 As shown, the form fill preview window 900 may also include a "Confirm" control 901. Accordingly, the confirmation input for the fill content can also be the user's click on the "Confirm" control 901. Accordingly, upon receiving the user's click on the "Confirm" control 901, the electronic device can fill the corresponding field information into the corresponding fill positions of the form to be filled based on the fill type corresponding to each field name, thereby achieving automatic form filling.

[0281] In this embodiment, after matching the field names with the user identity information database, a form fill preview window can be displayed for user confirmation. After user confirmation, the matched field information can be automatically filled into the corresponding positions according to the fill type, ensuring fill efficiency and accuracy of the fill results.

[0282] The form filling scheme provided in this application embodiment will be described below with reference to a specific application scenario.

[0283] Taking the example of a user's first visit to the hospital where the patient's information needs to be registered, this application provides an example of the form filling scheme.

[0284] Specifically, users can first open the registration interface for adding patient information. Figure 10 As shown, users need to fill in their name (1002), resident ID card number (1003), and mobile phone number (1004) on the patient information interface (1000) to register the patient's information. In actual application, the patient information interface (1000) can also display other information such as home address and emergency contact number.

[0285] For example, a user can take a screenshot of the patient information interface 1000 (input 1001). After receiving the screenshot input 1001 from the user, the electronic device can upload the screenshot image to the cloud. The cloud-based form judgment model then determines whether the form corresponding to the screenshot image is a form to be filled. If the output of the form judgment model is "yes", the electronic device can further schedule the field information extraction model in the cloud, input the screenshot image into the field information extraction model, and output the field name, fill type, and fill position of the field to be filled. The specific results are as follows: <ref> Name< / ref> <box> (82,302),(650,350)< / box> <type>input <type> ； <ref> Resident ID Card< / ref> <box> (82,403),(650,466)< / box> <type>input <type> ； <ref> Phone number< / ref> <box> (82,506),(650,561)< / box> <type>input <type>.

[0286] Afterwards, the electronic device can extract the field keywords of each field name, and based on the field keywords, look up the field mapping table to obtain the standard field name corresponding to the field keyword. For example, for "name" 1002, the extracted field keywords include: name, and after looking up the field mapping table, it can be obtained that the field name is the standard field name. For example, for "resident ID card" 1003, the field keywords that can be extracted include: resident ID, ID card, and resident ID card, and after looking up the field mapping table, it can be obtained that the standard field name of the field name "resident ID card" is "citizen ID number". Similarly, the field keywords of the extracted "mobile phone number" 1004 include: mobile phone number, number, and mobile phone, and after matching with the field mapping table, it can be obtained that the standard field name of the field name "mobile phone number" is "mobile phone number".

[0287] After obtaining the standard field name of each field name, the standard field name can be matched with the user identity information library to obtain the field information corresponding to each standard field name, and the field information is displayed Figure 11 As shown in the form filling preview window 1100, the form filling preview window 1100 displays the field information of each field name that is matched successfully, and the user can confirm the field information on the form filling preview window 1100. After confirming that there is no error, the user can click the "confirm" control 1101, and after the electronic device receives the click input 1102 of the user on the "confirm" control 1101, the field information can be filled into the corresponding filling position based on the filling type corresponding to each field name. For example, the electronic device can fill the field information "Li Si" into the filling position 10021 corresponding to the name 1002, and similarly, fill the field information "42***123" into the filling position 10031 corresponding to the resident ID card 1003, and similarly, fill the field information "132***456" into the filling position 10041 corresponding to the mobile phone number 1004. In this way, the automatic filling of the form is realized, and the user does not need to manually fill in the patient information, which is simple and efficient.

[0288] Especially when the user needs to fill in various online forms, the scheme of the embodiment of the present application can greatly simplify the operation of the user, save the time and effort of the user, and improve the filling efficiency of the form and the accuracy of the filling result.

[0289] It should be noted that the form filling method provided in the embodiments of the present application can be executed by a form filling device or a processing module in the form filling device for executing the form filling method. In the embodiments of the present application, the form filling device executes the form filling method as an example to illustrate the form filling device provided in the embodiments of the present application.

[0290] Figure 12 A structural schematic diagram of a form filling device provided in an embodiment of the present application.

[0291] As shown in Figure 12 The form filling device 1200 can include:

[0292] The extraction module 1201 is configured to input a form image of a form to be filled into a field information extraction model, and output field attribute information of a field to be filled contained in the form to be filled, the field attribute information including a field name, a filling type, and a filling position of the field to be filled.

[0293] The filling module 1202 is configured to fill field information corresponding to user identity information into a filling position corresponding to the field name according to the filling type.

[0294] The embodiment of the present application automatically extracts the field name, the filling type, and the filling position of the field to be filled in the form to be filled by using the field information extraction model, and automatically fills the field information corresponding to the user identity information into the position corresponding to the field name based on the filling type, thereby realizing automatic filling of the form, saving a large amount of time and effort of the user, improving the filling efficiency of the form, and avoiding filling errors caused by human factors, and ensuring the accuracy of the filling result.

[0295] In some possible implementations of the embodiment of the present application, the filling type includes at least one of the following: an input type, a drop-down box type, an upload type, a click type, and a submission type.

[0296] In some possible implementations of the embodiment of the present application, the form filling device 1200 can include:

[0297] The training module is configured to input a form image sample in a training sample, output result prompt information corresponding to the form image sample, and size information of the form image sample into a pre-trained visual language model, and output predicted field attribute information of a field to be filled contained in the form image sample.

[0298] The form filling device 1200 can further include:

[0299] The training module is specifically configured to:

[0300] input the form image sample in the training sample, a prompt word, and the size information of the form image sample into the pre-trained visual language model, and output predicted field attribute information of a field to be filled contained in the form image sample; wherein the predicted field attribute information is information predicted by the pre-trained visual language model based on the prompt word and the size information, the predicted field attribute information is field attribute information of the field to be filled contained in the form image sample, and the predicted field attribute information includes a predicted field name, a predicted filling type, and a predicted filling position of the field to be filled contained in the form image sample.

[0301] The visual language model is trained based on the predicted field attribute information and actual field attribute information of the to-be-filled field contained in the form image sample, to obtain a field information extraction model; the actual field attribute information is actual field attribute information of the to-be-filled field contained in the form image sample, and the actual field attribute information includes actual field name, actual filling type and actual filling position of the to-be-filled field contained in the form image sample.

[0302] In some possible implementations of the embodiments of the present application, the form filling apparatus 1200 can further include:

[0303] The acquisition module is configured to acquire a plurality of form image samples before the training module inputs the form image sample, the prompt word and the size information of the form image sample in the training sample into the pre-trained visual language model and outputs the predicted field attribute information of the to-be-filled field contained in the form image sample, the form image sample including at least one of the following: a form image in a reference filling scenario, a form image suitable for a reference device type.

[0304] The labeling module is configured to label the to-be-filled field of each form image sample to obtain structured labeling data of the to-be-filled field, the structured labeling data including a field name of the to-be-filled field and a filling type of the to-be-filled field; and label a filling position of the to-be-filled field to obtain position coordinate information of the filling position.

[0305] The determination module is configured to determine the structured labeling data and the position coordinate information as actual field attribute information of the to-be-filled field.

[0306] In some possible implementations of the embodiments of the present application, the reference filling scenario includes at least one of the following: a license plate number filling scenario, a pass filling scenario, a passport filling scenario, and a driver's license filling scenario.

[0307] The reference device type includes at least one of the following: a straight device type, a folding device type, and a tablet device type.

[0308] In some possible implementations of the embodiments of the present application, the structured labeling data further includes at least one of the following: filled content of the to-be-filled field and a filling prompt word of the to-be-filled field.

[0309] In some possible implementations of the embodiments of the present application, the filling type includes an input type.

[0310] The labeling module is specifically configured to:

[0311] According to the filling type of the to-be-filled field, determine a labeling manner of the filling position; different filling types correspond to different labeling manners.

[0312] According to the annotation manner of the filling position, the filling position is annotated to obtain position coordinate information of the filling position.

[0313] The position coordinate information includes a coordinate of a center point of the filling position.

[0314] In some possible implementations of the embodiments of the present application, the annotation module is specifically configured to:

[0315] In the case where the input type is an input box, a mark box is displayed on the filling position according to the size information of the input box, and the position coordinate information of the filling position is determined according to the display position of the mark box.

[0316] In the case where the input type is an underline, a mark box is displayed on the filling position, and the position coordinate information of the filling position is determined according to the display position of the mark box; wherein the reference line of the mark box is the end of the underline or the boundary of the field to be filled.

[0317] In some possible implementations of the embodiments of the present application, the training module is specifically configured to:

[0318] Based on the predicted field attribute information and the actual field attribute information of the field to be filled contained in the form image sample, the visual language model is trained for N rounds to obtain N candidate models.

[0319] The form filling device 1200 can further include:

[0320] The verification module is configured to perform performance verification on each candidate model in the N candidate models according to a verification set to obtain the accuracy of the field attribute information output by each candidate model.

[0321] The determination module is specifically configured to:

[0322] The candidate model with the maximum accuracy is determined as the field information extraction model.

[0323] In some possible implementations of the embodiments of the present application, the determination module is specifically configured to:

[0324] Before the filling module 1202 fills the field information corresponding to the user identity information into the filling position corresponding to the field name according to the filling type, the field information corresponding to the field name is determined based on a user identity information library, and the user identity information library includes the user identity information.

[0325] In some possible implementations of the embodiments of the present application, the extraction module 1201 is specifically configured to:

[0326] Extract the field keyword of the field name.

[0327] The determination module is specifically configured to:

[0328] determine the standard field name corresponding to the field keyword based on a field mapping table, the field mapping table being configured to store a mapping relationship between different standard field names and field keywords;

[0329] determine the field information corresponding to the standard field name based on a user identity information library.

[0330] In some possible implementations of the embodiments of the present application, the form filling apparatus 1200 can further include:

[0331] an output module configured to output first prompt information in a case where the user identity information library does not contain the field name, the first prompt information being configured to prompt that the user identity information library does not contain the field name;

[0332] a display module configured to display a user identity information editing interface, the user identity information editing interface including the field name not contained in the user identity information library;

[0333] a receiving module configured to receive an editing input on the field name not contained in the user identity information library;

[0334] a storage module configured to store, in response to the editing input, field information corresponding to the editing input and the field name to the user identity information library.

[0335] In some possible implementations of the embodiments of the present application, the form filling apparatus 1200 can further include:

[0336] a judgment module configured to input the form image into a form judgment model, and output judgment result information of the form image, the judgment result information being configured to represent whether the form corresponding to the form image is a form to be filled in;

[0337] an extraction module 1201, specifically configured to:

[0338] in a case where the judgment result information represents that the form corresponding to the form image is the form to be filled in, input the form image into a field information extraction model, and output field attribute information of a to-be-filled field contained in the form to be filled in.

[0339] In some possible implementations of the embodiments of the present application, the output module is further configured to, in a case where the judgment result information represents that the form corresponding to the form image is not the form to be filled in, output second prompt information, the second prompt information being configured to prompt that the form corresponding to the form image is not the form to be filled in.

[0340] In some possible implementations of the embodiments of the present application, the display module is further configured to display a form filling preview window, the form filling preview window including field names of all to-be-filled fields in the form image and field information corresponding to each field name; the form filling preview window is configured to preview a filling result of each to-be-filled field.

[0341] The receiving module is also used to receive confirmation input for the fill content in the form fill preview window;

[0342] Filling module 1202 is specifically used for:

[0343] In response to the confirmation of the input, the field information corresponding to each field name is filled into the corresponding fill position of the form to be filled, according to the fill type corresponding to each field name.

[0344] The form filling device in this application embodiment can be a device or a component in an electronic device, such as an integrated circuit or a chip. For example, the electronic device can be a mobile phone, tablet computer, laptop computer, PDA, in-vehicle electronic device, mobile internet device (MID), augmented reality (AR) / virtual reality (VR) device, robot, wearable device, ultra-mobile personal computer (UMPC), netbook, or personal digital assistant (PDA), etc. It can also be a server, network attached storage (NAS), personal computer (PC), television (TV), ATM, or self-service machine, etc. This application embodiment does not specifically limit the device.

[0345] The electronic device in this application embodiment can be a terminal with an operating system. This operating system can be Android, iOS, or other possible operating systems; this application embodiment does not specifically limit the specific operating system used.

[0346] The form filling device provided in this application embodiment can achieve... Figures 1 to 11 The various processes in the form filling method embodiment can achieve the same technical effect, and will not be described again here to avoid repetition.

[0347] like Figure 13 As shown, this application embodiment also provides an electronic device 1300, including a processor 1301 and a memory 1302. The memory 1302 stores programs or instructions that can run on the processor 1301. When the program or instructions are executed by the processor 1301, they implement the various steps of the above-described form filling method embodiment and can achieve the same technical effect. To avoid repetition, they will not be described again here.

[0348] It should be noted that the electronic devices in the embodiments of this application include the mobile terminals and non-mobile terminals mentioned above.

[0349] Figure 14 This is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of this application.

[0350] The electronic device 1400 includes, but is not limited to, components such as: radio frequency unit 1401, network module 1402, audio output unit 1403, input unit 1404, sensor 1405, display unit 1406, user input unit 1407, interface unit 1408, memory 1409, and processor 1410.

[0351] Those skilled in the art will understand that the electronic device 1400 may also include a power supply (such as a battery) for supplying power to various components. The power supply may be logically connected to the processor 1410 through a power management system, thereby enabling functions such as managing charging, discharging, and power consumption through the power management system. Figure 14 The structure of the electronic device 1400 shown does not constitute a limitation on the electronic device 1400. The electronic device 1400 may include more or fewer components than shown, or combine certain components, or have different component arrangements, which will not be described in detail here.

[0352] The processor 1410 is used to extract the form image input field information of the form to be filled from the model, and output the field attribute information of the fields to be filled contained in the form to be filled. The field attribute information includes the field name, fill type and fill position of the field to be filled; according to the fill type, the field information corresponding to the user identity information is filled into the fill position corresponding to the field name.

[0353] This application embodiment utilizes a field information extraction model to automatically extract the field name, fill type, and fill position of the fields to be filled in the form to be filled. Based on the fill type, it automatically fills the field information corresponding to the user's identity information into the position corresponding to the field name, realizing automatic form filling, saving users a lot of time and effort, improving the form filling efficiency, and avoiding filling errors caused by human factors, thus ensuring the accuracy of the filling results.

[0354] In some possible implementations of the embodiments of this application, the fill type includes at least one of the following: input type, drop-down list type, upload type, click type, and submit type.

[0355] In some possible implementations of embodiments of this application, the processor 1410 is specifically used for:

[0356] The form image samples, prompt words, and size information of the form image samples in the training samples are input into the pre-trained visual language model, and the output is the predicted field attribute information of the fields to be filled contained in the form image samples. The predicted field attribute information is the information predicted by the pre-trained visual language model based on the prompt words and size information. The predicted field attribute information is the field attribute information of the fields to be filled contained in the form image samples, including the predicted field name, predicted filling type, and predicted filling position of the fields to be filled contained in the form image samples.

[0357] Based on the predicted field attribute information and the actual field attribute information of the fields to be filled contained in the form image sample, the visual language model is trained to obtain the field information extraction model; wherein, the actual field attribute information is the actual field attribute information of the fields to be filled contained in the form image sample, including the actual field name, actual filling type and actual filling position of the fields to be filled contained in the form image sample.

[0358] In some possible implementations of embodiments of this application, the processor 1410 is specifically used for:

[0359] Before inputting the form image samples, prompt words, and size information of the form image samples from the training samples into the pre-trained visual language model and outputting the predicted field attribute information of the fields to be filled contained in the form image samples, multiple form image samples are collected. The form image samples include at least one of the following: form images in reference filling scenarios and form images suitable for reference device types.

[0360] The fields to be filled in each form image sample are labeled to obtain structured labeled data for the fields to be filled. The structured labeled data includes the field name and the filling type of the fields to be filled.

[0361] Mark the fill position of the field to be filled to obtain the position coordinate information of the fill position;

[0362] The structured annotation data and location coordinate information are used to determine the actual field attribute information of the fields to be filled.

[0363] In some possible implementations of the embodiments of this application, the reference filling scenario includes at least one of the following: license plate number filling scenario, pass filling scenario, passport filling scenario, driver's license filling scenario;

[0364] The reference device type includes at least one of the following: candybar device type, foldable device type, and tablet device type.

[0365] In some possible implementations of the embodiments of this application, the structured annotation data further includes at least one of the following: the already filled content of the field to be filled, and the fill prompt words of the field to be filled.

[0366] In some possible implementations of the embodiments of this application, the fill type includes the input type;

[0367] Processor 1410, specifically used for:

[0368] Determine the annotation method for the fill position based on the fill type of the field to be filled; different fill types correspond to different annotation methods.

[0369] According to the annotation method of the fill position, the fill position is annotated to obtain the position coordinate information of the fill position;

[0370] The location coordinate information includes the coordinates of the center point of the filling location.

[0371] In some possible implementations of embodiments of this application, the processor 1410 is specifically used for:

[0372] When the input type is an input box, a marker box is displayed at the fill position according to the size information of the input box, and the position coordinate information of the fill position is determined according to the display position of the marker box;

[0373] When the input type is underline, a marker box is displayed at the fill position, and the position coordinates of the fill position are determined based on the display position of the marker box; the reference line of the marker box is the end of the underline or the boundary of the field to be filled.

[0374] In some possible implementations of embodiments of this application, the processor 1410 is specifically used for:

[0375] Based on the predicted field attribute information and the actual field attribute information of the fields to be filled contained in the form image samples, the visual language model is trained for N rounds to obtain N candidate models.

[0376] Based on the validation set, the performance of each candidate model among the N candidate models is validated to obtain the accuracy of the field attribute information output by each candidate model;

[0377] The candidate model with the highest accuracy was selected as the field information extraction model.

[0378] In some possible implementations of embodiments of this application, the processor 1410 is specifically used for:

[0379] Before filling the field information corresponding to the user identity information into the fill position corresponding to the field name according to the fill type, the field information corresponding to the field name is determined based on the user identity information database, which includes user identity information.

[0380] In some possible implementations of embodiments of this application, the processor 1410 is specifically used for:

[0381] Extract field keywords from field names;

[0382] Based on the field mapping table, the standard field names corresponding to the field keywords are determined. The field mapping table is used to store the mapping relationship between different standard field names and field keywords.

[0383] Based on the user identity information database, determine the field information corresponding to the standard field names.

[0384] In some possible implementations of embodiments of this application, the processor 1410 is specifically used for:

[0385] If the user identity information database does not contain the field name, output the first prompt message. The first prompt message is used to indicate that the user identity information database does not contain the field name.

[0386] Display unit 1406 is used to display the user identity information editing interface, which includes field names not included in the user identity information database;

[0387] User input unit 1407 is used to receive edit input for field names not included in the user identity information database;

[0388] Processor 1410, specifically used for:

[0389] In response to edit input, the corresponding field information and field name are stored in the user identity information database.

[0390] In some possible implementations of embodiments of this application, the processor 1410 is specifically used for:

[0391] Before extracting the field information of the form image of the form to be filled into the model and outputting the field attribute information of the fields to be filled contained in the form, the form image is input into the form judgment model and the judgment result information of the form image is output. The judgment result information is used to characterize whether the form corresponding to the form image is the form to be filled.

[0392] If the form corresponding to the form image in the result information is a form to be filled, the form image input field information extraction model is used to output the field attribute information of the fields to be filled contained in the form to be filled.

[0393] In some possible implementations of embodiments of this application, the processor 1410 is specifically used for:

[0394] After inputting the form image into the form judgment model and outputting the judgment result information of the form image, if the judgment result information indicates that the form corresponding to the form image is not a form to be filled, a second prompt information is output. The second prompt information is used to indicate that the form corresponding to the form image is not a form to be filled.

[0395] In some possible implementations of the embodiments of this application, the display unit 1406 is used to display a form fill preview window, which includes the field names of all fields to be filled in the form image and the field information corresponding to each field name; the form fill preview window is used to preview the filling result of each field to be filled.

[0396] User input unit 1407 is used to receive confirmation input for the fill content of the form fill preview window;

[0397] Processor 1410, specifically used for:

[0398] In response to the confirmation of the input, the field information corresponding to each field name is filled into the corresponding fill position of the form to be filled, according to the fill type corresponding to each field name.

[0399] It should be understood that, in this embodiment, the input unit 1404 may include a graphics processing unit (GPU) 14041 and a microphone 14042. The GPU 14041 processes image data of still images or videos obtained by an image capture device (such as a camera) in video capture mode or image capture mode. The display unit 1406 may include a display panel 14061, which may be configured in the form of a liquid crystal display, an organic light-emitting diode, or the like. The user input unit 1407 includes at least one of a touch panel 14071 and other input devices 14072. The touch panel 14071 is also called a touch screen. The touch panel 14071 may include a touch detection device and a touch controller. Other input devices 14072 may include, but are not limited to, physical keyboards, function keys (such as volume control buttons, power buttons, etc.), trackballs, mice, and joysticks, which will not be described in detail here.

[0400] The memory 1409 can be used to store software programs and various data. The memory 1409 may primarily include a first storage area for storing programs or instructions and a second storage area for storing data. The first storage area may store the operating system, application programs or instructions required for at least one function (such as sound playback, image playback, etc.). Furthermore, the memory 1409 may include volatile memory or non-volatile memory, or both. The non-volatile memory may be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory can be random access memory (RAM), static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct memory bus RAM (DRRAM). The memory 909 in the embodiments of this application includes, but is not limited to, these and any other suitable types of memory.

[0401] Processor 1410 may include one or more processing units; optionally, processor 1410 integrates an application processor and a modem processor, wherein the application processor mainly handles operations involving the operating system, user interface, and applications, and the modem processor mainly handles wireless communication signals, such as a baseband processor. It is understood that the aforementioned modem processor may also not be integrated into processor 1410.

[0402] This application also provides a readable storage medium storing a program or instructions. When the program or instructions are executed by a processor, they implement the various processes of the above-described form filling method embodiments and achieve the same technical effect. To avoid repetition, they will not be described again here.

[0403] The processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes computer-readable storage media, such as computer read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0404] This application also provides a chip, which includes a processor and a communication interface. The communication interface and the processor are coupled. The processor is used to run programs or instructions to implement the various processes of the above-described form filling method embodiments and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0405] It should be understood that the chip mentioned in the embodiments of this application may also be referred to as a system-on-a-chip, system chip, chip system, or system-on-a-chip, etc.

[0406] This application provides a computer program product that is stored in a storage medium and executed by at least one processor to implement the various processes of the form filling method embodiment described above, and achieves the same technical effect. To avoid repetition, it will not be described again here.

[0407] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one…" does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, it should be noted that the scope of the methods and apparatuses in the embodiments of this application is not limited to performing functions in the order shown or discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

[0408] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the related technology, can be embodied in the form of a computer software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0409] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.< / type> < / type> < / type> < / type> < / type> < / type>

Claims

1. A form filling method, characterized in that, include: The form image of the form to be filled is used to extract field information from the model, and the field attribute information of the fields to be filled in the form to be filled is output. The field attribute information includes the field name, fill type and fill position of the fields to be filled. According to the fill type, the field information corresponding to the user identity information is filled into the fill position corresponding to the field name.

2. The method according to claim 1, characterized in that, The fill type includes at least one of the following: input type, drop-down list type, upload type, click type, and submit type.

3. The method according to claim 1, characterized in that, The method further includes: The form image sample, prompt words, and size information of the form image sample in the training samples are input into a pre-trained visual language model, which outputs the predicted field attribute information of the field to be filled contained in the form image sample; wherein, the predicted field attribute information is the information predicted by the pre-trained visual language model based on the prompt words and the size information, and the predicted field attribute information is the field attribute information of the field to be filled contained in the form image sample, including the predicted field name, predicted filling type, and predicted filling position of the field to be filled contained in the form image sample; Based on the predicted field attribute information and the actual field attribute information of the fields to be filled contained in the form image sample, the visual language model is trained to obtain a field information extraction model; wherein, the actual field attribute information is the actual field attribute information of the fields to be filled contained in the form image sample, and the actual field attribute information includes the actual field name, actual filling type and actual filling position of the fields to be filled contained in the form image sample.

4. The method according to claim 3, characterized in that, Before inputting the form image samples, prompt words, and size information of the form image samples from the training samples into the pre-trained visual language model, and outputting the predicted field attribute information of the fields to be filled contained in the form image samples, the method further includes: Collect multiple form image samples, wherein the form image samples include at least one of the following: a form image in a reference filling scenario, and a form image suitable for a reference device type; The fields to be filled in each of the form image samples are labeled to obtain structured labeled data of the fields to be filled. The structured labeled data includes the field name of the field to be filled and the filling type of the field to be filled. The filling positions of the fields to be filled are marked to obtain the position coordinate information of the filling positions; The structured annotation data and the location coordinate information are determined as the actual field attribute information of the field to be filled.

5. The method according to claim 4, characterized in that, The reference filling scenarios include at least one of the following: license plate number filling scenario, pass filling scenario, passport filling scenario, and driver's license filling scenario; The reference device type includes at least one of the following: candybar device type, foldable device type, and tablet device type.

6. The method according to claim 4, characterized in that, The structured annotation data also includes at least one of the following: the already filled content of the field to be filled, and the fill prompt words for the field to be filled.

7. The method according to claim 4, characterized in that, The fill type includes the input type; The step of marking the fill position of the field to be filled, and obtaining the position coordinate information of the fill position, includes: The annotation method for the filling position is determined based on the filling type of the field to be filled; different filling types correspond to different annotation methods. According to the annotation method of the filling position, the filling position is annotated to obtain the position coordinate information of the filling position; The location coordinate information includes the coordinates of the center point of the filling location.

8. The method according to claim 7, characterized in that, The step of annotating the fill position according to the annotation method to obtain the position coordinate information of the fill position includes: When the input type is an input box, a marker box is displayed at the fill position according to the size information of the input box, and the position coordinate information of the fill position is determined according to the display position of the marker box; When the input type is underline, a marker box is displayed at the fill position, and the position coordinate information of the fill position is determined according to the display position of the marker box; wherein, the reference line of the marker box is the end of the underline or the boundary of the field to be filled.

9. The method according to any one of claims 3-6, characterized in that, The step of training the visual language model based on the predicted field attribute information and the actual field attribute information of the fields to be filled contained in the form image sample to obtain a field information extraction model includes: Based on the predicted field attribute information and the actual field attribute information of the fields to be filled contained in the form image sample, the visual language model is trained for N rounds to obtain N candidate models. The performance of each of the N candidate models is verified based on the validation set to obtain the accuracy of the field attribute information output by each candidate model. The candidate model with the highest accuracy was selected as the field information extraction model.

10. The method according to any one of claims 1-6, characterized in that, Before filling the field information corresponding to the user identity information into the fill position corresponding to the field name according to the fill type, the method further includes: Based on the user identity information database, the field information corresponding to the field name is determined, and the user identity information database includes user identity information.

11. The method according to claim 10, characterized in that, The step of determining the field information corresponding to the field name based on the user identity information database includes: Extract the field keywords from the field names; Based on the field mapping table, the standard field name corresponding to the field keyword is determined. The field mapping table is used to store the mapping relationship between different standard field names and field keywords. Based on the user identity information database, determine the field information corresponding to the standard field name.

12. The method according to claim 10, characterized in that, The method further includes: If the user identity information database does not contain the field name, a first prompt message is output, which is used to indicate that the user identity information database does not contain the field name. Display a user identity information editing interface, which includes the field names not included in the user identity information database; Receive edit input for the name of the field not included in the user identity information database; In response to the edit input, the field information corresponding to the edit input and the field name are stored in the user identity information database.

13. The method according to any one of claims 1-8, characterized in that, Before extracting the form image input field information of the form to be filled from the model and outputting the field attribute information of the fields to be filled contained in the form, the method includes: The form image is input into the form judgment model, and the judgment result information of the form image is output. The judgment result information is used to characterize whether the form corresponding to the form image is a form to be filled. The model for extracting input field information from the form image of the form to be filled, and outputting the field attribute information of the fields to be filled in the form, includes: If the judgment result indicates that the form corresponding to the form image is a form to be filled, the form image is input into the field information extraction model, and the field attribute information of the fields to be filled contained in the form to be filled is output.

14. The method according to claim 13, characterized in that, After inputting the form image into the form judgment model and outputting the judgment result information of the form image, the method further includes: If the judgment result indicates that the form corresponding to the form image is not a form to be filled, a second prompt message is output. The second prompt message is used to indicate that the form corresponding to the form image is not a form to be filled.

15. The method according to any one of claims 1-8, characterized in that, The step of filling the field information corresponding to the user identity information into the fill position corresponding to the field name according to the fill type includes: The form fill preview window displays the names of all fields to be filled in the form image and the field information corresponding to each field name; the form fill preview window is used to preview the filling result of each field to be filled. Receive confirmation input for the filled content in the form fill preview window; In response to the confirmation of the input, the field information corresponding to each field name is filled into the corresponding fill position of the form to be filled according to the fill type corresponding to each field name.

16. A form filling device, characterized in that, include: The extraction module is used to input the form image of the form to be filled into the field information extraction model and output the field attribute information of the fields to be filled contained in the form to be filled. The field attribute information includes the field name, fill type and fill position of the fields to be filled. The fill module is used to fill the field information corresponding to the user identity information into the fill position corresponding to the field name according to the fill type.

17. The apparatus according to claim 16, characterized in that, The device further includes: The training module is specifically used for: The form image sample, prompt words, and size information of the form image sample in the training samples are input into a pre-trained visual language model, which outputs the predicted field attribute information of the field to be filled contained in the form image sample; wherein, the predicted field attribute information is the information predicted by the pre-trained visual language model based on the prompt words and the size information, and the predicted field attribute information is the field attribute information of the field to be filled contained in the form image sample, including the predicted field name, predicted filling type, and predicted filling position of the field to be filled contained in the form image sample; Based on the predicted field attribute information and the actual field attribute information of the fields to be filled contained in the form image sample, the visual language model is trained to obtain a field information extraction model; wherein, the actual field attribute information is the actual field attribute information of the fields to be filled contained in the form image sample, and the actual field attribute information includes the actual field name, actual filling type and actual filling position of the fields to be filled contained in the form image sample.

18. The apparatus according to claim 17, characterized in that, The training module is specifically used for: Based on the predicted field attribute information and the actual field attribute information of the fields to be filled contained in the form image sample, the visual language model is trained for N rounds to obtain N candidate models. The performance of each of the N candidate models is verified based on the validation set to obtain the accuracy of the field attribute information output by each candidate model. The candidate model with the highest accuracy was selected as the field information extraction model.

19. The apparatus according to any one of claims 16-18, characterized in that, The device further includes: The judgment module is used to input the form image into the form judgment model and output the judgment result information of the form image before the extraction module inputs the form image of the form to be filled into the field information extraction model and outputs the field attribute information of the fields to be filled contained in the form to be filled. The judgment result information is used to characterize whether the form corresponding to the form image is a form to be filled. The extraction module is specifically used for: If the judgment result indicates that the form corresponding to the form image is a form to be filled, the form image is input into the field information extraction model, and the field attribute information of the fields to be filled contained in the form to be filled is output.

20. An electronic device, characterized in that, The electronic device includes a processor and a memory, the memory storing programs or instructions that can run on the processor, the programs or instructions being executed by the processor to implement the steps of the method as described in any one of claims 1 to 15.

Images