Information processing method and electronic equipment

Abstract

The embodiment of the invention is applied to the technical field of computers, provides an information processing method and electronic equipment, and is used for automatically filling multiple contents of a table by one key. The method comprises the steps of displaying a first interface; the first interface comprises a table and a first control, and does not comprise a second control used for inputting or triggering display of the first information. Wherein the table comprises a first table label and a to-be-filled control, and the first table label is a table label associated with the to-be-filled control in the first interface. The first information comprises all information to be filled in the first interface. And in response to a user operation on the first control, filling the to-be-filled content matched with the first table tag into the to-be-filled control based on the first table tag.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to an information processing method and an electronic device. Background Technology

[0002] With the development of computer technology, more and more applications (APPs) are being developed and installed on electronic devices to improve the functionality and personalization of these devices and provide users with a richer user experience.

[0003] However, in some information-filling scenarios (such as first-time application registration), users are usually required to fill in relevant information (such as name, mobile phone number, etc.). This means that users need to enter the same information on different applications, which may make the user information entry process cumbersome, waste a lot of user time, and result in a poor user experience. Summary of the Invention

[0004] This application provides an information processing method and an electronic device, which includes multiple controls for automatically filling out forms with a single click.

[0005] To achieve the above objectives, the embodiments of this application adopt the following technical solutions:

[0006] Firstly, an information processing method is provided for use in electronic devices, the method comprising:

[0007] The system displays a first interface. This first interface includes a table and a first control, but does not include a second control for inputting or triggering the display of first information; the first information includes all information to be filled in on the first interface. The table includes a first table label and a control to be filled in, where the first table label is the table label associated with the control to be filled in on the first interface. On this first interface, in response to user operation on the first control, the electronic device fills in the content to be filled in based on the first table label, matching the first table label. In this solution, the electronic device can respond to user operation on the first control, achieving one-click automatic filling of the table, which can greatly improve the efficiency of filling in the form.

[0008] In one possible implementation of the first aspect, the first control is a one-click fill control.

[0009] In one possible implementation of the first aspect, before filling in the field with content matching the first form label based on the first form label, the method further includes: in response to a user operation, identifying the label text of the first form label from the first interface. Based on the label text of the first form label, retrieving the field with content matching the first form label from a user information database. In this solution, the electronic device can automatically identify and locate the content to be filled in the form on the first interface. Thus, the user does not need to search for the required content in other applications beforehand; instead, the electronic device automatically locates the matching field based on the first interface.

[0010] In one possible implementation of the first aspect, the data to be filled in from the user information database that matches the first form label may include multiple data entries. When multiple data entries match the first form label, the electronic device, after obtaining these data entries, displays candidate data entries for the user to select. Furthermore, filling the data entries matching the first form label into the fill-in control based on the first form label may specifically include: responding to a user's triggering action on one of the candidate data entries, filling the candidate data entry into the fill-in control. This ensures that the data automatically filled in by the electronic device in the form is accurate.

[0011] In one possible implementation of the first aspect, when the table includes multiple first table labels and multiple fields to be filled, the above-mentioned method of filling in the fields with content matching the first table labels into the fields to be filled controls is based on the first table labels. Specifically, this requires filling in the fields with content based on a first correspondence between the first table labels, the fields to be filled controls, and the content to be filled. Since there is a correspondence between the first table labels and the fields to be filled controls, and the content to be filled matches the first table labels, a one-to-one correspondence can be established between the first table labels, the fields to be filled controls, and the content to be filled—that is, a first correspondence. This ensures the accuracy of the content automatically filled in by the electronic device in the table.

[0012] In one possible implementation of the first aspect, the first table label in the aforementioned first correspondence can be the label text of the first table label. The control to be filled in in the aforementioned first correspondence can be the coordinate information of the control to be filled in.

[0013] In one possible implementation of the first aspect, based on the first correspondence between the first form label, the control to be filled, and the content to be filled, a filling operation is sequentially performed on each control to fill in the content to be filled. That is, a filling operation is performed on one control at a time. In this way, the electronic device fills in the content sequentially when performing the filling operation, ensuring the accuracy of the automatically filled content in the form.

[0014] In one possible implementation of the first aspect, based on the first correspondence between the first table label, the control to be filled, and the content to be filled, the filling operation is performed sequentially on each control to be filled. Specifically, this may include: selecting a control to be filled as the target control. Then, based on the aforementioned first correspondence, the content to be filled corresponding to the target control is filled into the target control. The step of selecting a control to be filled as the target control is returned, until the filling operation has been completed for all controls to be filled. In this way, the electronic device fills in the data sequentially when performing the filling operation, ensuring the accuracy of the automatically filled content in the table.

[0015] In one possible implementation of the first aspect, based on the aforementioned first correspondence, the content to be filled in corresponding to the target fill control is entered into the target fill control. Specifically, this may include: obtaining the coordinate information of the target fill control from the first correspondence; performing a simulated click operation on the filling area corresponding to the target fill control based on the coordinate information of the target fill control; and then entering the content to be filled in corresponding to the target fill control. In this way, by simulating clicking on the fill control to enter the corresponding content, the possibility of automatic filling errors is reduced, ensuring the accuracy of the content automatically filled in the form by the electronic device.

[0016] In one possible implementation of the first aspect, during the process of filling in one of the fill-in controls, after filling in the content corresponding to the target fill-in control, a keyboard exit operation is also performed. This ensures that the electronic device can continue to perform fill-in operations on subsequent fill-in controls.

[0017] In one possible implementation of the first aspect, before filling in the fill-in control with the content matching the first table label based on the first table label, the method further includes: in response to a user operation, identifying the coordinate information of the fill-in control from the first interface. This coordinate information can be used by the electronic device to locate the display position of the fill-in control on the first interface. In this solution, after obtaining the fill-in content matching the first table label from the user information database, the method further includes: generating a content vector corresponding to the fill-in content according to a first correspondence. The content vector includes the label text of the first table label, the coordinate information of the fill-in control, and the fill-in content. The content vectors corresponding to each fill-in content are stored in a preset data structure. In this solution, selecting a fill-in control as the target fill-in control specifically includes: retrieving the first content vector from the preset data structure, and determining the fill-in control corresponding to the coordinate information of the fill-in control contained in the first content vector as the target fill-in control. Based on the first correspondence, filling the target fill-in control with the fill-in content corresponding to the target fill-in control includes: filling the target fill-in control with the fill-in content from the first content vector. In this solution, after obtaining the content to be filled, the content is stored in a preset data storage structure. This facilitates the subsequent execution of filling operations on each control based on the preset data storage structure, ensuring the accuracy of the automatically filled content in the table.

[0018] In one possible implementation of the first aspect, after filling in the content to be filled into the target fill control, and before returning to the step of selecting a fill control as the target fill control, the method further includes: detecting whether an abnormal situation has occurred. If an abnormal situation occurs, the electronic device may be unable to continue performing the filling operation on subsequent fill controls. Therefore, after each filling operation on a fill control is completed, an abnormality detection is performed, and if an abnormality occurs, abnormality handling is executed. This abnormality handling is used to exit the abnormal situation. This ensures that the electronic device can automatically handle abnormal situations and complete one-click automatic filling.

[0019] In one possible implementation of the first aspect, the abnormal situation specifically includes: keyboard mode not exiting. The aforementioned exception handling includes: performing a keyboard exit operation. In this solution, the step of returning to select a control to be filled as the target control includes: if keyboard mode has already exited, returning to the step of selecting a control to be filled as the target control. This ensures that the electronic device can handle the abnormal situation of keyboard mode not exiting by performing corresponding exception handling, thus completing one-click automatic filling.

[0020] In one possible implementation of the first aspect, the abnormal situation includes: table shifting. The aforementioned exception handling includes: obtaining the content to be filled that matches the second table label in the shifted table. The second table label is the table label associated with the control to be filled in the shifted table. A second correspondence between the second table label, the control to be filled in, and the content to be filled in is obtained. This second correspondence is used to perform a filling operation on the control to be filled in the shifted table. After performing exception handling for table shifting, new content to be filled in can be obtained, and then the remaining control to be filled in the shifted table is filled in based on this new content, ensuring the accuracy of the content automatically filled in by the electronic device.

[0021] In one possible implementation of the first aspect, the abnormal situation includes: table shifting. In the shifted table, the display positions of the remaining fill-in controls and their associated table labels may change, but the fill-in controls themselves will not change. Therefore, the above-mentioned exception handling includes: obtaining the coordinate information of the fill-in controls in the shifted table; updating the first correspondence based on the coordinate information of the fill-in controls in the shifted table; and using the updated first correspondence to perform a fill-in operation on the fill-in controls in the shifted table. Updating the coordinate information of the fill-in controls in the first correspondence allows obtaining the fill-in content corresponding to the shifted table. Then, based on the updated first correspondence, a fill-in operation is performed on the remaining fill-in controls in the shifted table to ensure the accuracy of the content automatically filled in by the electronic device.

[0022] In one possible implementation of the first aspect, the abnormal situation includes: table displacement. Detecting whether an abnormal situation has occurred can specifically include: obtaining the first cursor position when the cursor is activated, and the second cursor position after the filling operation on the target fill control is completed. The difference between the y-coordinate of the first cursor position and the y-coordinate of the second cursor position is calculated and determined as the cursor vertical offset value. Then, it is determined whether the cursor vertical offset value is greater than or equal to an error threshold. If the cursor vertical offset value is greater than or equal to the error threshold, it indicates that the table has been displaced. If the cursor vertical offset value is less than the error threshold, it indicates that the table has not been displaced. Here, cursor activation occurs when a simulated click operation is performed on the target fill control. This method accurately detects whether the table has been displaced and is simple to operate.

[0023] In one possible implementation of the first aspect, abnormal situations include not exiting keyboard mode and table shifting. In this scheme, an abnormal situation is detected, and if it occurs, exception handling is performed, including: checking if keyboard mode has not been exited. If keyboard mode has not been exited, a keyboard exit operation is performed. After performing the keyboard exit operation, if keyboard mode has been exited, it is checked if the table has shifted. If the table has shifted, exception handling for table shifting is performed; this exception handling is used to obtain new content to be filled, and the new content is used to fill in the controls in the shifted table. In this way, by sequentially checking whether keyboard mode has not been exited and whether the table has shifted, and executing the corresponding exception handling if any abnormal situation exists, the electronic device can complete one-click automatic filling.

[0024] In one possible implementation of the first aspect, abnormal situations include failure to exit keyboard mode and table shifting. In this scheme, an abnormal situation is detected, and if it occurs, exception handling is performed, including: checking if keyboard mode has not been exited. If keyboard mode has not been exited, a keyboard exit operation is performed. After performing the keyboard exit operation, if keyboard mode still has not been exited, the filling operation on the control to be filled out is stopped. If exiting keyboard mode is unsuccessful, the electronic device will terminate the automatic form filling process to avoid filling errors.

[0025] In one possible implementation of the first aspect, the method further includes: identifying filled controls in a table on the first interface and table labels associated with the filled controls. Based on the association between the filled controls and the table labels, a vector of filled content is generated. This vector of filled content is used to update the user information database. After one-click autofill, some content may be incorrectly filled, and the user may need to correct or supplement it. Thus, this solution enables the extraction of content from filled controls in the interface, and the user information database can be updated based on the extracted content to ensure the richness and accuracy of the user information database data.

[0026] In one possible implementation of the first aspect, the generated content vector, based on the association between the filled controls and the form labels, can be stored as key-value pairs.

[0027] In one possible implementation of the first aspect, the electronic device may specifically be, in response to a user operation on a third control, identify a filled control of a table in the first interface and a table label associated with the filled control. The third control may specifically be an information retrieval control.

[0028] In one possible implementation of the first aspect, the method further includes displaying a third control on the first interface. For example, the electronic device may automatically display the third control when it detects that a form has been completed, or it may display the third control in response to a user's action.

[0029] In one possible implementation of the first aspect, the information processing method described above can be implemented using a state machine. For example, the method can be specifically implemented using a visual detection state machine, a content-to-be-filled acquisition state machine, a content-to-be-filled maintenance state machine, and a fill-in execution state machine. The visual detection state machine is used to identify whether the first interface includes a table, and when the first interface includes a table, to determine the display positions of the control to be filled in (including input controls and / or dropdown controls), empty fill-in controls, and the table label associated with the control to be filled in, and the display position of that table label, etc. The content-to-be-filled acquisition state machine is used to acquire the content to be filled in that matches the first table label. The content-to-be-filled maintenance state machine is used to maintain and update the acquired content to be filled in during the fill-in operation; for example, updating the content to be filled in when a table error occurs. The fill-in execution state machine is used to execute the fill-in operation, automatically filling the acquired content to be filled in the corresponding control to be filled in. The state machine format facilitates management and enables efficient and accurate automatic table filling.

[0030] In one possible implementation of the first aspect, the above method may further include an exception handling state machine. The exception handling state machine is used to detect whether an exception has occurred after the completion of a fill-in operation for an empty fill-in control. Specifically, the exception handling state machine may include a keyboard exit exception detection state machine, a table shift exception handling state machine, and a keyboard exit exception handling state machine. The exception detection state machine is used to detect whether an exception has occurred indicating that keyboard mode has not been exited. The table shift exception handling state machine is used to detect whether an exception has occurred indicating that table shifting has occurred, and determines the next operation based on the detection result. The keyboard exit exception handling state machine is used to perform a keyboard exit operation, and again determine whether keyboard mode has not been exited, and determines the next operation based on the detection result. The state machine format facilitates management and enables efficient and accurate automatic table filling.

[0031] In one possible implementation of the first aspect, the electronic device stores multiple sets of correspondences between form labels and label data. Retrieving the content to be filled that matches the first form label from the user information database includes: searching the user information database for a target form label that matches the first form label; and, based on the correspondence between form labels and label data, retrieving the label data corresponding to the target form label as the content to be filled.

[0032] Secondly, this application also provides an information processing method for extracting the content of filled controls in an interface. The method includes: identifying filled controls in a table on a first interface and table labels associated with the filled controls. Based on the association between the filled controls and the table labels, a vector of filled content is generated. This vector of filled content is used to update the user information database. After one-click automatic filling, some content may be incorrectly filled, and the user may need to correct or supplement it. Thus, this solution enables the extraction of content from filled controls in the interface, and the extracted content can be used to update the user information database, ensuring the richness and accuracy of the user information database data.

[0033] In one possible implementation of the second aspect, the generated content vector, based on the association between the filled controls and the form labels, can be stored as key-value pairs.

[0034] In one possible implementation of the second aspect, the electronic device may specifically be, in response to a user operation on a third control, identify a filled control of a table in the first interface and a table label associated with the filled control. The third control may specifically be an information retrieval control.

[0035] In one possible implementation of the second aspect, the method further includes displaying a third control on the first interface. For example, the electronic device may automatically display the third control when it detects that the form has been filled out, or it may display the third control in response to a user's action.

[0036] Thirdly, this application also provides an electronic device. The electronic device may include a display screen, a processor, and a memory. The memory stores computer-executable instructions, and when the electronic device is running, the processor executes the computer-executable instructions stored in the memory to cause the electronic device to perform the information processing method as described in any of the first aspects above.

[0037] Fourthly, this application provides a computer-readable storage medium storing instructions that, when executed on a computer, enable the computer to perform any of the information processing methods described in the first aspect above.

[0038] Fifthly, a computer program product containing instructions is provided, which, when run on an electronic device, enables the electronic device to perform any of the information processing methods described in the first aspect above.

[0039] In a sixth aspect, an apparatus (e.g., a system-on-a-chip) is provided, comprising a processor for supporting an electronic device in performing the functions described in the first aspect above. In one possible design, the apparatus further comprises a memory for storing program instructions and data necessary for the electronic device. When the apparatus is a system-on-a-chip, it may be composed of chips or may include chips and other discrete devices.

[0040] The technical effects of any of the design methods in aspects two through six can be found in the technical effects of different design methods in aspect one, and will not be repeated here. Attached Figure Description

[0041] Figure 1 This is a schematic diagram of a table according to an embodiment of this application;

[0042] Figure 2 A schematic diagram of the form-filling interface and operation involved in the information processing method provided in the embodiments of this application;

[0043] Figure 3 A schematic diagram of the selection-while-filling process involved in the information processing method provided in the embodiments of this application;

[0044] Figure 4 A schematic diagram of the interface and operation of the information processing method provided in the embodiments of this application, which involves selecting and filling in information simultaneously;

[0045] Figure 5 A schematic diagram showing the relative positions of table frames involved in the information processing method provided in the embodiments of this application;

[0046] Figure 6 This is a schematic diagram illustrating element matching involved in the information processing method provided in the embodiments of this application;

[0047] Figure 7 A schematic diagram of the automatic form filling process involved in the information processing method provided in the embodiments of this application;

[0048] Figure 8 A screenshot of the first interface involved in the information processing method provided in the embodiments of this application;

[0049] Figure 9 A schematic diagram of table frame matching involved in the information processing method provided in the embodiments of this application;

[0050] Figure 10 A schematic diagram showing the relative positions of the fill box and the matching label box involved in the information processing method provided in the embodiments of this application;

[0051] Figure 11 A schematic diagram of table frame coordinates involved in the information processing method provided in the embodiments of this application;

[0052] Figure 12 A flowchart illustrating an information processing method provided in an embodiment of this application;

[0053] Figure 13 A flowchart illustrating an information processing method provided in an embodiment of this application;

[0054] Figure 14 A schematic diagram of a mobile phone display provided in an embodiment of this application;

[0055] Figure 15 A schematic diagram of a mobile phone display provided in an embodiment of this application;

[0056] Figure 16 A schematic diagram of a state machine provided for an embodiment of this application;

[0057] Figure 17 A schematic diagram of a state machine provided for an embodiment of this application;

[0058] Figure 18 A schematic diagram of a state machine provided for an embodiment of this application;

[0059] Figure 19 A schematic diagram of a state machine provided for an embodiment of this application;

[0060] Figure 20 A schematic diagram of a state machine provided for an embodiment of this application;

[0061] Figure 21 A schematic diagram of a state machine provided for an embodiment of this application;

[0062] Figure 22 A schematic diagram of a state machine provided for an embodiment of this application;

[0063] Figure 23A A schematic diagram illustrating a partial style of a table provided in an embodiment of this application;

[0064] Figure 23B A schematic diagram of five target categories of a table provided for an embodiment of this application;

[0065] Figure 24 This is a schematic diagram of a partial software structure of an electronic device provided in an embodiment of this application;

[0066] Figure 25 A flowchart illustrating a table visual inspection method provided in an embodiment of this application;

[0067] Figure 26A A flowchart illustrating a table visual inspection method provided in an embodiment of this application;

[0068] Figure 26B A schematic diagram illustrating an intersection-union ratio (IUR) provided in an embodiment of this application;

[0069] Figure 27A A flowchart illustrating a table visual inspection method provided in an embodiment of this application;

[0070] Figure 27B A flowchart illustrating a table visual inspection method provided in an embodiment of this application;

[0071] Figure 28 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of this application;

[0072] Figure 29 A schematic diagram of the software structure of an electronic device provided in an embodiment of this application;

[0073] Figure 30 A flowchart illustrating an information entry method provided in this application embodiment;

[0074] Figure 31 This application provides a schematic diagram of an interface for filling in user information.

[0075] Figure 32 This is a schematic diagram of an interface for adding user information provided in an embodiment of this application;

[0076] Figure 33 A schematic diagram of an interface for setting a preset collection application provided in an embodiment of this application;

[0077] Figure 34 This is a schematic diagram of an interface for extracting user information provided in an embodiment of this application;

[0078] Figure 35 A flowchart illustrating a method for standardizing label names provided in this application embodiment;

[0079] Figure 36 A flowchart illustrating a method for updating a user information database, as provided in this application embodiment;

[0080] Figure 37 A flowchart illustrating a method for querying and adjusting user information provided in this application embodiment;

[0081] Figure 38 This is a schematic diagram of an interface for querying user information provided in an embodiment of this application;

[0082] Figure 39 This is a schematic diagram of an interface for deleting user information provided in an embodiment of this application;

[0083] Figure 40A flowchart illustrating an information entry method provided in this application embodiment;

[0084] Figure 41 This is a schematic diagram of an interface for selecting user information provided in an embodiment of this application. Detailed Implementation

[0085] The technical solutions of the embodiments of this application are described below with reference to the accompanying drawings. In the description of the embodiments of this application, the terminology used in the following embodiments is for the purpose of describing specific embodiments only and is not intended to limit the application. As used in the specification and appended claims of this application, the singular expressions "a," "the," "the," "the," and "this" are intended to also include expressions such as "one or more," unless the context clearly indicates otherwise. It should also be understood that in the following embodiments of this application, "at least one" and "one or more" refer to one or more (including two). The term "and / or" is used to describe the relationship between related objects, indicating that three relationships can exist; for example, A and / or B can represent: A alone, A and B simultaneously, or B alone, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship.

[0086] References to "one embodiment" or "some embodiments" in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized. The term "connection" includes direct connections and indirect connections, unless otherwise stated. "First" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated.

[0087] In the embodiments of this application, the words "exemplarily" or "for example" are used to indicate examples, illustrations, or explanations. Any embodiment or design described as "exemplarily" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design solutions. Specifically, the use of the words "exemplarily" or "for example" is intended to present the relevant concepts in a specific manner.

[0088] It should be noted that the user information used in the technical solution of this application is limited to information for which individual consent has been obtained, including but not limited to notifying and reminding users to read the relevant user agreement (notification) and sign the agreement (authorization) which includes the authorization of relevant user information before users use the information collection function.

[0089] A table is a data organization used for structured presentation of information. A table consists of multiple frame boxes, within which text can be displayed. Tables displayed on electronic devices may include groups of frame boxes, each group containing a label and a text field. The label includes a label box and a label name, with the label name located within the label box.

[0090] Electronic devices offer multiple applications (APPs), and the application interface of these applications may include form-filling interfaces. For example, shopping applications on electronic devices may have a shipping address entry interface, application account registration interfaces, and ride-hailing applications may have a starting address entry interface. Electronic devices display forms on these interfaces, which include label boxes and fill-in controls. The fill-in controls include fill boxes, which may or may not contain filled-in text.

[0091] It should be noted that the form displayed on the electronic device's form-filling interface can be of various forms. For example, the form displayed on the electronic device may only include label text and fill boxes, or the form may include label boxes and fill boxes. The information processing method involved in the embodiments of this application is applicable to the information processing flow of electronic devices in the information filling interface of an application, and is not applicable to the information processing flow of conventional Excel format table documents.

[0092] like Figure 1 The image shown is a schematic diagram of a table according to an embodiment of this application. Figure 2 The diagram shown is a schematic representation of the form-filling interface and its operation in the information processing method provided in this application embodiment.

[0093] A fill-in control includes a fill box and fill content, with the fill content residing within the fill box. When fill content exists within the fill box, it can be either prompt text or actual text. If no fill content exists within the fill box, it can be empty. The electronic device can respond to user input by filling in content for the fill box.

[0094] Each table can include multiple table frame groups, and the label box and input field in each table frame group are located close to each other within the table. For example, the label box and input field in each table frame group are usually located in adjacent columns of the same row in the table, or the label box and input field in each table frame group are usually located in adjacent rows of the same column in the table.

[0095] like Figure 1 The image shown is a schematic diagram of a table. Figure 1 In this example, the table consists of multiple frames, each of which can be used to enter text. Figure 1 As shown, the table includes a label 101 and a fill control 102. The label 101 includes a label box 1011 and a label name 1012. The fill control includes a fill box 1021 and content text 1022.

[0096] like Figure 1 As shown, the table may include multiple label boxes 1011, and different label boxes 1011 include different label names 1012, such as name, mobile phone number, ID card number, email address, contact address, etc. Each label box 1011 has a matching fill box 1021, which is used to fill in the content text 1022 that matches the label name, and the content text 1022 in different fill boxes 1021 is different.

[0097] Based on the matching relationship between label 101 and fill control 102, Figure 1 The table can also be divided into multiple table frame groups 103, each table frame group 103 including a label 101 and a fill control 102, with the label 101 and fill control 102 located in adjacent columns of the same row of the table. In other cases, the label 101 and fill control 102 of a table frame group 103 may also be located in adjacent rows of the table.

[0098] like Figure 1 As shown, input box 1021A contains the text "177****6666", input box 1021B contains the prompt text "Please enter your ID number here", input box 1021C does not contain any text, and input box 1021D contains the prompt "Please enter your contact address here".

[0099] Figure 1 The table shown may have visible and displayed borders. For example... Figure 2 The diagram shown is a schematic representation of the form-filling interface involved in the information processing method provided in this application embodiment. Figure 2 As shown in (1) in the table filling interface, the border of the table frame may be absent or hidden.

[0100] The form-filling interface provided by electronic devices typically includes multiple labels and input boxes. The labels and their names are usually pre-existing and fixed, while the text in the input boxes must be manually entered by the user. In various scenarios, electronic devices require users to manually fill in the text, making the process cumbersome. Furthermore, multiple form-filling interfaces provided by electronic devices may involve the same text, requiring users to repeatedly input it, further increasing the complexity and resulting in a poor user experience.

[0101] Based on this, embodiments of this application provide an information processing method. An electronic device responds to a user's touch operation, triggering automatic form filling. The electronic device automatically identifies the fill-in boxes in the form, finds tags that match the fill-in boxes, and automatically matches content text to the fill-in boxes based on the tags, thus achieving automatic form filling. This greatly reduces user interaction on the form-filling interface and improves the user experience.

[0102] For example, the electronic device can be a mobile phone, tablet computer, personal computer (PC), smart screen, desktop, laptop, handheld computer, notebook computer, ultra-mobile personal computer (UMPC), netbook, wearable device such as a smartwatch, artificial intelligence (AI) speaker, and in-vehicle device. It can also be various teaching aids (such as learning machines and early education machines), smart toys, portable robots, personal digital assistants (PDAs), augmented reality (AR) / virtual reality (VR) devices, media players, etc. Furthermore, it can be a device with mobile office functions, a device with smart home functions, a device with audio-visual entertainment functions, or a device supporting smart travel. This application embodiment does not impose special limitations on the specific form of the electronic device. The structure of the electronic device will be described in detail in the later embodiments.

[0103] The following sections, in conjunction with several embodiments, will describe in turn the implementation schemes for triggering automatic form filling and various automatic form filling scenarios involved in the information processing method provided in this embodiment.

[0104] Example 1: In response to the first operation, the electronic device triggers automatic form filling.

[0105] The electronic device responds to the user's touch operation and triggers the execution of the information processing method provided in this embodiment to automatically fill in the form on the form filling interface.

[0106] In one example, the electronic device triggers automatic form filling in response to a touch operation applied by the user on the form filling interface.

[0107] like Figure 2 The diagram shown illustrates the interface and operation schematic of triggering automatic form filling in the information processing method provided in this embodiment of the application. Figure 2 As shown in (1), the electronic device displays a form-filling interface 2101, which can be a recipient information filling interface. The form displayed on the form-filling interface 2101 includes label boxes and filling controls. The label boxes include label names such as name, telephone number, region, and detailed address. The filling controls include or are used to fill in the content text corresponding to the label names in the label boxes, such as the content text corresponding to the name, the content text corresponding to the telephone number, the content text corresponding to the region, and the content text corresponding to the detailed address.

[0108] like Figure 2 In the form-filling interface shown in (1), the form includes multiple blank form frames for filling controls, meaning that the corresponding text content has not yet been filled in the filling controls. The electronic device can respond to the user's touch operation and trigger the automatic form-filling process, automatically filling in the corresponding text content in the multiple filling controls within the form-filling interface.

[0109] Depending on the different input methods, fill-in controls can be divided into input forms and drop-down forms. For example, when the fill-in control is an input form, the electronic device can respond to the user's input and add corresponding text content to the input form. Input forms are suitable for situations where there are many types of text content or where it is impossible to recommend options. Figure 2 As shown in (1), the input controls for name, telephone number and detailed address are all input form boxes.

[0110] For example, user input can be text or voice input on the form interface. Another example is when the fill-in control is a dropdown list; the electronic device can respond to the user's selection by adding corresponding text to the dropdown list. Dropdown lists are suitable for situations where the types of text content are relatively few or where suggested selections are possible. Figure 2 As shown in (1), the input control for the region is a drop-down list.

[0111] Electronic devices can respond to user touch operations and trigger automatic form filling, automatically matching or filling in the corresponding text content in the input boxes and / or drop-down boxes in the form filling interface displayed on the electronic device.

[0112] In one example, the electronic device displays a first control on the form-filling interface, which the user triggers to autofill the form. The electronic device can respond to a user's touch operation on the first control to trigger autofill on the form-filling interface.

[0113] like Figure 2 As shown in (1), the electronic device displays a first control 202 on the form filling interface 201. The first control 202 may include a gesture prompt bar, a table-shaped control, an artificial intelligence (AI)-shaped control, an input pen-shaped control, etc., without limitation.

[0114] like Figure 2 As shown in (1), the electronic device can receive touch operations such as single click, double click or long press by the user on the first control 202, triggering the automatic form filling process.

[0115] In one scenario, the electronic device responds to a user's touch operation on the first control 202, directly triggering the autofill process and initiating the autofill operation on the form filling interface.

[0116] In another scenario, the electronic device, responding to a user's touch operation on the first control 202, can also display autofill options corresponding to different form-filling modes. For example... Figure 2 As shown in (2), the electronic device displays an auto-fill option, including a first mode control 203 corresponding to the first form filling mode and a second mode control 204 corresponding to the second form filling mode.

[0117] The first form-filling mode corresponds to a select-and-fill mode. The electronic device automatically fills in the text for the currently selected fill-in control. The electronic device then automatically fills in the text for the next selected fill-in control.

[0118] It should be noted that the "select-and-fill" method mentioned here refers only to a scheme where the electronic device automatically fills in the corresponding text for the currently selected fill control. In other cases, "select-and-fill" may also be called other names, such as form filling, sequential form filling, single-box form filling, etc., without limitation.

[0119] The second form-filling mode corresponds to the one-click form-filling mode. After the electronic device triggers auto-fill, it automatically fills in the text corresponding to multiple fill-in controls based on all the fill-in controls on the form-filling interface.

[0120] It should be noted that the term "one-click form filling" here refers only to a solution where an electronic device fills out multiple fields on a form interface at once. In other cases, "one-click form filling" may also be called by other names, such as quick form filling or one-click full form filling, without limitation.

[0121] like Figure 2 As shown in (2), the electronic device displays the first mode control 203 and the second mode control 204, which can be displayed as text on the first interface 201. For example, the electronic device displays "Fill in form" to mark the first mode control 203 and "One-click fill" to mark the second mode control 204. In other cases, the electronic device can also mark the first mode control and the second mode control as graphical controls, without limitation.

[0122] like Figure 2 As shown in (2), the electronic device receives a click operation from the user on the first mode control 203, triggering simultaneous selection and filling, such as... Figure 2 As shown in (3) above, the specific implementation scheme for selecting and filling in simultaneously can be found in the relevant description of Embodiment 2 below. Or as shown in... Figure 2 As shown in (4) and (5) in the figure, the electronic device can receive the user's click operation on the second mode control 204 to trigger one-click form filling. The specific implementation scheme of one-click form filling can be found in the relevant description of the following embodiment 3.

[0123] This example provides an information processing method where an electronic device displays a first control on the form-filling interface. The device responds to user touch input on this first control, displaying option controls corresponding to different form-filling modes, such as a first mode control for simultaneous selection and filling, and a second mode control for one-click form filling. The electronic device triggers automatic form filling in the corresponding mode by responding to user clicks on either the first or second mode control. The method provided by the electronic device for triggering form filling is relatively simple, saving user effort and improving the user experience.

[0124] In another example, the electronic device responds to a user's touch input in a smart application and triggers automatic form filling after displaying the form-filling interface.

[0125] For example, electronic devices can automatically initiate an auto-fill process in response to a user's touch operation on a smart application. When an electronic device detects that the currently displayed application interface is a form-filling interface, it can display an auto-fill option within that interface.

[0126] For example, electronic devices can also add an autofill on / off control within the settings interface of smart applications. The electronic device can receive user clicks on the autofill control and automatically start the autofill process. When the electronic device detects that the currently displayed application interface is a form-filling interface, it can display the autofill option on the form-filling interface.

[0127] In other examples, electronic devices may also respond to a user's voice commands, triggering automatic form filling.

[0128] For example, if an electronic device receives voice information from a user that includes keywords indicating automatic form filling, and the electronic device detects that the currently displayed application interface is a form filling interface, it will initiate the automatic form filling process. The keywords indicating automatic form filling mentioned here can include at least one of the following: form filling, selection-based form filling, one-click form filling, automatic form filling, and AI form filling, without limitation.

[0129] Electronic devices can also trigger automatic form filling in other ways. Other schemes that can trigger automatic form filling can be applied to the information processing method provided in the embodiments of this application, and will not be elaborated further.

[0130] Example 2: The automatic form-filling mode executed by the electronic device is "select and fill simultaneously".

[0131] The electronic device responds to user input, triggering a form-filling interface where users select and fill in information simultaneously. The device identifies the currently selected field and determines the matching label. Based on the label's name, the device fills in the corresponding text for that field.

[0132] It should be noted that, in the information processing method provided in this application embodiment, the electronic device fills in matching text in the input box. This can be done by directly displaying the acquired text in the input box, or by displaying the acquired text in the input box or an adjacent area for the user to select. If text already exists in the displayed input box, the electronic device can also respond to user operations by adding, modifying, or deleting text in the input box.

[0133] like Figure 3 The diagram shown is a schematic flowchart of the information processing method provided in this application embodiment, involving selection-while-filling. The electronic device performs selection-while-filling, mainly including the following steps:

[0134] S301: The electronic device displays a first interface, which is a form filling interface. The first interface includes a first fill box and multiple label boxes. Each label box has a corresponding label name, and different label boxes have different label names.

[0135] The information processing method provided in this application embodiment involves an electronic device displaying a first interface. This first interface can be the same as the form-filling interface in the aforementioned example, and will be referred to as the "first interface" thereafter. The electronic device responds to a user's touch operation on the first interface, triggering automatic form filling. The touch operation responded to by the electronic device can include clicks, single-clicks, double-clicks, swipes, etc., and is not limited to any particular type.

[0136] like Figure 4As shown, the electronic device displays a first interface 401, which includes a table. The table includes multiple labels 402, such as 402A, 402B, 402C, and 402D. The label names for different labels are different. For example, the label name for 402A is "Name," the label name for 402B is "Mobile Number," the label name for 402C is "Region," and the label name for 402D is "Detailed Address."

[0137] Furthermore, the table includes multiple fill-in boxes 403, such as 403A, 403B, 403C, and 403D. Some of the fill-in boxes displayed on the first screen may already have text filled in; for example, the text in 403A might be "Zhang San". Conversely, some fill-in boxes displayed on the first screen may be empty; for example, 403B, 403C, and 403D may not have corresponding text filled in.

[0138] For ease of description, the text boxes with filled content in the table can be labeled as "filled table boxes," and the text boxes without filled content can be labeled as "unfilled table boxes."

[0139] The electronic device responds to a user's touch operation on an unfilled form and determines the current form to be filled. For ease of description, the form indicated by the current user's touch operation is designated as the first form. For example... Figure 4 As shown in (1), the electronic device receives a click operation from the user on the fill box 403C and determines that 403C is the first fill box to be filled.

[0140] The electronic device receives touch input from the user onto the first input field and can also move the cursor into the first input field. The electronic device can detect the current cursor position and determine the first input field to be filled based on the current cursor position.

[0141] like Figure 4 As shown in (1), the electronic device receives a touch operation from the user on the fill box 403C and displays the cursor 404 within the fill box 403C, as shown in (1). Figure 4 As shown in (2) of the text.

[0142] In one example, the electronic device acquires a first screenshot of the first interface, analyzes the first screenshot to obtain the current cursor position, and then determines the first input box in the first interface.

[0143] In specific implementation, such as Figure 3 As shown, before the electronic device executes S302 and determines the first label box that matches the first fill box, it may also include the following steps:

[0144] S3011: The electronic device acquires the first screenshot of the first interface.

[0145] S3012: The electronic device extracts features from the first screenshot and identifies the first cursor.

[0146] like Figure 4 As shown in (2), the first screen of the electronic device is captured, and the corresponding first screenshot is obtained, such as... Figure 4 As shown in (3) of the text.

[0147] The electronic device extracts features from the first screenshot, identifying possible features corresponding to the cursor, including color, shape, and texture features. The extracted features can be used to identify the cursor in the first screenshot using pattern recognition or machine learning algorithms, and this identified cursor is designated as the first cursor.

[0148] S3013: The electronic device determines the first fill box based on the pixel coordinates of the first cursor in the first screenshot.

[0149] The electronic device can identify the first cursor in the first screenshot and obtain the pixel coordinates of the first cursor in the first screenshot, thereby determining the first input box. For example, the electronic device can determine the input box whose coordinate range includes the coordinate range of the first cursor as the first input box, or the electronic device can determine the input box whose starting coordinates of the table frame and the coordinates of the first cursor are within a preset range as the first input box.

[0150] In another example, the electronic device can also obtain the cursor position by extracting the parameters associated with the cursor in the first interface, and then determine the first fill box to be filled in in the first interface based on the cursor position.

[0151] For example, for electronic devices with input and textarea elements, the selectionStart and selectionEnd properties can be used to get the cursor position. selectionStart returns the starting position of the cursor within the element, while selectionEnd returns the ending position of the cursor within the element.

[0152] In practice, electronic devices can also obtain the first cursor in the first interface through other means, and then determine the first fill box to be filled based on the first cursor.

[0153] S302: The electronic device responds to a first operation applied to a first input box in a first interface, and determines a first label box that matches the first input box from a plurality of label boxes; the first label box has a first label name.

[0154] The information processing method provided in this embodiment allows an electronic device to identify multiple table frames within a first interface, including multiple label boxes and multiple fill boxes, with the fill boxes including a first fill box to be filled. The table on the first interface comprises multiple table frame groups, each group including a matching label box and a fill box. The positions of the fill box and label box in each table frame group are typically relatively fixed. For example, the label box and fill box in each table frame group are located in adjacent columns of the same row, or they are typically located in adjacent rows of the same column in the table.

[0155] like Figure 5 The diagram shown illustrates the relative positions of table frames in the information processing method provided in this embodiment. Figure 5 As shown in (1), the table includes two table frame groups, wherein the first table frame group 501 includes a label box 5011 and a fill box 5012, and the second table frame group 502 includes a label box 5021 and a fill box 5022.

[0156] The label box and fill box in each table frame group are located in adjacent columns of the same row. For example, label box 5011 and fill box 5012 are in the same row, with label box 5011 located in the column to the left of fill box 5012. Label box 5021 and fill box 5022 are in the same row, with label box 5021 located in the column to the left of fill box 5022.

[0157] like Figure 5 As shown in (2), the table includes two table frame groups, namely table frame group A and table frame group B. Among them, the label box 5031 of table frame group A 503 is located in the column above the fill box 5032, and the label box 5041 of table frame group B 504 is located in the column above the fill box 5042.

[0158] Figure 5 The two relative positions of the form frames shown can include the relative positions of most matching label boxes and fill boxes within a form. Based on this, after determining the first fill box to be filled, the electronic device can determine the label box that matches the first fill box from among multiple label boxes, according to the position of the first fill box and the relative positions of multiple label boxes. For ease of description, the electronic device can refer to the label box that matches the first fill box among multiple label boxes as the first label box.

[0159] S303: The electronic device displays first content text in the associated area of ​​the first input box; the tag attribute of the first content text matches the first tag name.

[0160] The electronic device obtains the first tag that matches the first input field, extracts the tag name from the first tag's first tag box, and records it as the first tag name. The electronic device matches the first content text for the first input field based on the first tag name, and the tag attributes of the first content text match the first tag name.

[0161] like Figure 4 As shown in (3), the first input box determined by the electronic device is 403C, the first tag box matched by 403C is 402C, and the first tag name in 402C is "region". Based on the first tag name "region", the electronic device matches the first content text with the tag attribute "region" for the first input box, such as "Province A, City B, District C".

[0162] In other cases, if the electronic device determines that the first label name entered in the first label box is "gender", the electronic device will use the first content text with the label attribute of the first input box being "gender", such as "female" or "male".

[0163] In one specific implementation, the electronic device matches content text to the first fill box, and the number of content texts can be one or more.

[0164] In one scenario, if the electronic device obtains only one matched text, it can directly fill in the first text field as the first text field.

[0165] In another scenario, the electronic device matches at least one piece of content text to the first input field, and the electronic device can display at least one piece of content text in the associated area of ​​the first input field. The electronic device can fill in the first information table with the content text that received the user's touch operation from the at least one piece of content text as the first piece of content text.

[0166] like Figure 4 As shown in (4), the electronic device matches the content text "Province A, City B, District C" to the first fill box 403C, and displays the content text 405 in the adjacent column of 403C. If the electronic device receives a touch operation applied to the content text 405, the electronic device will fill "Province A, City B, District C" as the first content text into the first fill box 403C, as shown in (4). Figure 4 As shown in (5) of the table.

[0167] Based on the aforementioned implementation, after matching the corresponding first content text to the first input box, the electronic device can further detect when the user triggers other input boxes to be filled, such as a second input box, and match the corresponding content text, such as second content text, to the second input box. Figure 3 As shown, the electronic device can also perform the following steps:

[0168] S304: The electronic device responds to a second operation applied to the second fill box in the first interface, and determines a second label box that matches the second fill box from multiple label boxes; the first label box is a different table frame, and the second label box has a second label name.

[0169] like Figure 4 As shown in (5), after the electronic device matches the first content text to the first fill box 403C, the electronic device can also receive the user's click operation on 403D to determine that 403D is the second fill box to be filled.

[0170] As shown in (5) of Figure (4), the electronic device determines the matching second label box 402D for the second fill box 403D.

[0171] S305: The electronic device displays second content text in the associated area of ​​the second fill box; the label attribute of the second content text matches the second label name.

[0172] like Figure 4 As shown in (5), the electronic device extracts the second label name filled in the second label box 402D as "detailed address". Based on the "detailed address", the electronic device determines the matching second content text for the second input box 403D. The label attribute of the second content text is "detailed address". Figure 4 As shown in (6), the electronic device matches the content text 406 of the second fill box as "D Street E Community F Unit". If the electronic device receives a touch operation on the content text 406, the electronic device fills the content text 406 as the second content text into the second fill box 403D.

[0173] It should be noted that the electronic device can display the input box in the form of a control, which can respond to the user's touch operation. After the electronic device fills in the second input box, it can also receive the user's touch operation on the second input box to add, modify, or delete the second content text entered in the second input box.

[0174] The electronic device executes S301 and S302 to determine the first fill box to be filled and the first label box that matches the first fill box, and the electronic device executes S304 and S305 to determine the second fill box to be filled and the second label box that matches the second fill box. All of these involve the electronic device recognizing and matching the table frames and text within the table. The specific implementation scheme will be explained in detail below through several implementation methods.

[0175] like Figure 6The diagram shown illustrates the element matching involved in the information processing method provided in this application embodiment. The electronic device identifies the cursor position (X, Y) and determines the first fill box 601 to be filled. The electronic device includes an information extraction module and a content matching module. The information extraction module extracts information from the first interface, such as scene information, filled information, and information to be filled. The content matching module is used to match the corresponding content text for the information to be filled.

[0176] For example Figure 6 As shown, the electronic device's information extraction module extracts information such as the title and name of the first interface to obtain the table's scene information, such as "XX shipping address". The electronic device can also extract filled information, such as "Zhang San", whose corresponding tag attribute is "name". The electronic device extracts information to be filled, whose tag attribute is "telephone".

[0177] Based on this, the electronic device's content matching module matches the text with the tag attribute "telephone" to the information to be filled, such as "177****6666". The electronic device then displays the determined text 602 in the associated area of ​​the first fill-in box 601 to be filled.

[0178] It should be noted that the electronic device can determine the content text for the first field to be filled in directly based on the tag attributes of that content text. For example, based on the tag attribute "telephone", the matching content text is "177****6666". Alternatively, the electronic device can determine the matching content text based on both the tag attributes of the element and the already filled information. For example, the electronic device can determine the content text "177****6666" that matches both "Zhang San" and "telephone" based on the tag attribute "telephone" and the already filled information "Zhang San".

[0179] In one specific implementation, the electronic device can identify multiple form boxes and filled elements in the first interface through a pre-trained neural network model, determine the first fill box to be filled by the cursor position, and then search for the first label box that matches the first fill box from multiple label boxes through a pre-configured first algorithm.

[0180] like Figure 7 The diagram shown illustrates the automatic form-filling process involved in the information processing method provided in this application embodiment. The automatic form-filling process executed by the electronic device mainly includes the following steps:

[0181] S701: The electronic device captures the first screenshot of the first interface.

[0182] like Figure 8 As shown in (1), the electronic device acquires a screenshot of the first interface, denoted as the first screenshot. The first screenshot can be a screenshot of the entire interface of the first interface.

[0183] In one scenario, the electronic device displays a first interface, with a table displayed in a portion of the interface area on the first interface, or a pop-up window is added to the first interface to display the table. The electronic device may also take a screenshot of only the portion of the interface area displaying the table or the pop-up window, and use it as the first screenshot.

[0184] S702: Electronic device obtains cursor position.

[0185] Electronic devices can extract image features from the first screenshot, identify the cursor features in the first screenshot, and obtain the cursor position.

[0186] S703: The electronic device cropped the first screenshot to obtain the second screenshot based on the cursor position.

[0187] In the auto-fill process of electronic devices in a select-as-you-go scenario, the currently unfilled fields need to be analyzed and matched, referencing already filled fields during this process. In the initial screen, filled fields are typically positioned above unfilled fields. Electronic devices can retain only the table at the cursor position and above, thus obtaining the necessary information for auto-fill while reducing unnecessary computation.

[0188] like Figure 8 As shown in (1), the electronic device acquires the first screenshot. After acquiring the cursor position, the electronic device retains the image features located in the row and above the cursor in the first screenshot, and crops the first screenshot to obtain the second screenshot, as shown in (1). Figure 8 As shown in (2) of the text.

[0189] S704: The electronic device performs Type I detection on the second screenshot to identify the text bounding box and text information in the second screenshot.

[0190] S705: The electronic device performs a second type of detection on the second screenshot to identify the table frame and table frame type in the second screenshot.

[0191] The electronic device performs a first type of detection and a second type of detection on the second screenshot. The first type of detection is used to detect text information in the table and determine the text bounding box (or text box) corresponding to each segment of text information. The second type of detection is used to detect table frames in the table. The order in which the electronic device performs the first and second types of detection is not limited; for example, the electronic device can perform the first and second types of detection simultaneously.

[0192] The electronic device executes S704 to perform a first type of detection on the second screenshot. This can be done by using Optical Character Recognition (OCR) technology to perform optical character recognition on the second image and recognize the text information in the second screenshot.

[0193] OCR technology scans, processes, and analyzes text information in images, converting the analyzed text into a computer-readable format. The basic steps of OCR technology include image preprocessing, feature extraction, character recognition, and post-processing. Preprocessing typically involves operations such as noise reduction, binarization, and image enhancement to improve image quality and clarity. Feature extraction extracts text features, such as shape, size, and tilt, through image analysis. Character recognition uses these features to identify and classify characters, while post-processing includes operations such as correcting text position and fixing typos.

[0194] The electronic device performs Type I detection on the second screenshot and can also identify the text box corresponding to each piece of text information. For example... Figure 9 As shown, the table includes table frames 901 and 902, and text information 903 and 904. The electronic device, through a first type of detection, can identify text information 903 and 904, as well as the text bounding box 905 corresponding to text information 903 and the text bounding box 906 corresponding to text information 904.

[0195] The electronic device executes step S705 to perform a second type of detection on the second screenshot. This can be achieved using a first model, which can be a visual detection model or a model with frame recognition capabilities, such as the YOLO model. The first model has the ability to identify frames from the input image and obtain the coordinates of each frame in the input image and the frame type.

[0196] The electronic device inputs the second screenshot into the first model to obtain the coordinates of each frame in the second screenshot. For example... Figure 9 As shown, the electronic device inputs the second screenshot into the first model, identifies table frame 901 and table frame 902, and obtains the coordinates of the four vertices (A, B, C and D) of table frame 901 in the second screenshot, as well as the coordinates of the four vertices (a, b, c and d) of table frame 902 in the second screenshot.

[0197] Furthermore, the electronic device inputs the second screenshot into the first model, identifies multiple table frames present in the second screenshot, and determines the table frame type of each table frame based on the existence status or input method of the elements within each table frame. The table frame types obtained by the electronic device using the first model can include label boxes and fill boxes. Fill boxes can be further divided into unfilled input boxes, unfilled drop-down option boxes, text input boxes with content, and filled drop-down option boxes.

[0198] In this context, a label box refers to a form box with a label name entered; a label box can also be defined as a prompt box. An "Unfilled" input box is a fill box where no text is entered and the fill method is input. An "Unfilled" dropdown box is a fill box where no text is entered and the fill method is dropdown. A "Filled" input box is a fill box where text is entered and the fill method is input. A "Filled" dropdown box is a fill box where text is entered and the fill method is input. In some cases, a fill box may support both input and dropdown options; in this case, the fill box can be classified as an input box.

[0199] like Figure 8 In the second screenshot shown in (2), the electronic device recognizes multiple table frames, of which table frames 402A, 402B, 402C and 402D are label boxes, 403A is a filled input box, 403B is a filled input box, 403C is a drop-down option box for not filled, and 403D is a not filled input box.

[0200] S706: Electronic devices match text bounding boxes with table frames, and match text information with table frames.

[0201] like Figure 9 As shown, the electronic device identifies text bounding boxes 905 and 906 in the second screenshot through the first type of detection, and table frames 901 and 902 in the second screenshot through the second type of detection. The electronic device needs to further determine that table frame 901 matches text bounding box 905 and table frame 902 matches text bounding box 906.

[0202] In one example, the electronic device may use a center-point matching algorithm to determine the text bounding box that matches each table frame.

[0203] Specifically, the electronic device obtains the center point of each text box, searches among multiple table frames, and identifies the table frame whose coordinate range includes the coordinates of the text box's center point as the matching table frame. In practice, the electronic device can determine the coordinate range of each table frame by using the coordinates of its two opposite vertices, and then determine whether the table frame's coordinate range includes the coordinates of the text box's center point.

[0204] For example, an electronic device determines the coordinate range of the table frame based on the coordinates of its top-left and bottom-left corners. Figure 9 As shown, the electronic device obtains the center point O1(X) of the text box of the text outer frame 905. O1 Y O1 The coordinates of the four vertices ABCD of table frame 901 and the coordinates of the four vertices abcd of table frame 902.

[0205] In one scenario, the electronic device obtains the coordinates (X, y, y) of the top-left vertex A of the frame 901.A Y A ) and the coordinates (X) of the lower right vertex D D Y D Determine whether the coordinates of the center point O1 of the text box and the coordinate range of the table frame 901 satisfy one of the following conditions:

[0206] (X A ≤X O1 )&(X D ≥X O1 )&(Y A ≥Y O1 )&(Y D ≤Y O1 ), or (X A ≤X O1 ≤X D )&(Y A ≥Y O1 ≥Y D ).

[0207] When the coordinates of the center point O1 of the text box and the coordinate range of the table frame 901 meet the above conditions, the electronic device determines that the text outer box 905 matches the table frame 901, that is, the text information in the table frame 901 is "name".

[0208] In another scenario, the electronic device obtains the coordinates (X) of the top-left vertex a of frame 902. a Y a ) and the coordinates (X) of the lower right vertex D d Y d Determine whether the coordinates of the center point O1 of the text box and the coordinate range of the table frame 902 satisfy one of the following conditions:

[0209] (X a ≤X O1 )&(X d ≥X O1 )&(Y a ≥Y O1 )&(Y d ≤Y O1 ), or (X a ≤X O1 ≤X d )&(Y a ≥Y O1 ≥Y d ).

[0210] If the coordinates of the center point O1 of the text box and the coordinate range of the table frame 902 do not meet the above conditions, the electronic device determines that the text outer box 905 and the table frame 902 do not match, that is, the text information in the table frame 901 is not "name".

[0211] In another scenario, the electronic device can obtain the coordinates (X) of the top-left vertex a of frame 902. a The coordinates (X,Ya) and the coordinates of the lower right vertex D. d (Yd), determine whether the coordinates of the center point O2 of the text box of the text outer box 906 and the coordinate range of the table frame of the table frame 902 satisfy one of the following conditions:

[0212] (X a ≤X O2 )&(X d ≥X O2 )&(Y a ≥Y O2 )&(Y d ≤Y O2 ), or (X a ≤X O2 ≤X d )&(Y a ≥Y O2 ≥Y d ).

[0213] When the electronic device determines that the coordinates of the center point O2 of the text box and the coordinate range of the table frame 902 meet the above conditions, it determines that the text outer box 906 and the table frame 902 match, that is, the text information in the table frame 901 is "Please enter your name".

[0214] It should be noted that the above example uses the top-left and bottom-right vertices of the text outer box as factors in calculating the table frame coordinate range. In other cases, the bottom-left and top-right vertices of the text outer box can also be used as the calculated coordinate values ​​of the table frame, or all four vertices of each text outer box can be used as factors in calculating the table frame coordinate range; there are no restrictions.

[0215] In another specific implementation, the electronic device may employ a table label and fill box pairing algorithm based on two-dimensional Gaussian distribution modeling.

[0216] like Figure 5 The relative positional relationships of the two types of table frame groups shown can cover the relative positional relationships of label boxes and fill boxes in a large number of tables. Electronic devices can acquire a large number of table samples, and based on the relative positional relationships of label boxes and fill boxes within multiple table frame groups in the table samples, use a two-dimensional Gaussian distribution model to statistically analyze the relative distribution between the center point coordinates of the fill box and the center point coordinates of the label box, and calculate the probability of a fill box matching each label box among multiple label boxes based on this relative distribution. For example... Figure 10As shown, this diagram illustrates the relative positions of the fill box and the matching label box. The greater the probability of a match between the label box and the fill box, the more negative the difference in their X-axis coordinates and the greater the difference in their Y-axis coordinates, which are either 0 or positive. Conversely, the less likely a match is, the more positive the X-axis coordinates or the less positive the Y-axis coordinates.

[0217] In this way, electronic devices can achieve high precision and low latency in matching the label box to the fill box.

[0218] The electronic device model, based on Gaussian distribution, for matching fill boxes to form boxes mainly includes the following steps:

[0219] Step 1: The electronic device preprocesses the second screenshot.

[0220] The electronic device uses the first model to identify the label boxes and fill boxes in the second screenshot and records the center point coordinates of each label box and each fill box.

[0221] Step 2: Construct a two-dimensional Gaussian distribution statistical model for the electronic device.

[0222] The electronic device statistically analyzes the relative position distribution of the center point coordinates of label boxes and fill boxes from a large number of form samples, and defines the relative position vector d. ij =C Ij -C Ti C Ti C represents the coordinates of the center point of the label box. Ti =(x Ti ,y Ti ), C Ij C represents the coordinates of the center point of the fill box. Ij =(x Ij ,y Ij Based on this, the relative position vector d ij =(x Ij -x Ti ,y Ij -y Ti The relative position vector corresponds to the center distance between the label box and the fill box, which is the distance between the center points of the matching label boxes and the center points of the matching fill boxes.

[0223] Based on the distribution results of the relative position vectors of a large number of table samples, electronic devices construct a two-dimensional Gaussian distribution model p(d ij ),

[0224]

[0225] Where μ represents the mean, ε represents the covariance matrix, exp represents the natural exponential function, and T represents the transpose matrix.

[0226] Step 3: The electronic device constructs a first candidate set, which includes candidate label boxes whose center distance from the first fill box is within a first threshold.

[0227] Electronic devices for each fill box I j Calculate the center distance to other candidate boxes, and find candidate label boxes T whose center distance is within a predefined first threshold r. i Forming candidate set T j T j =T i ||d ij |≤r. For example, the first threshold r can be less than or equal to 100 pixels.

[0228] Step 4: The electronic device determines the probability that each candidate label box matches the fill box.

[0229] For each candidate label box T in the candidate set, the electronic device i Calculate the probability I that it matches the fill box. j The calculation formula is:

[0230] The electronic device obtains the probability of each candidate label box relative to the fill box in the candidate set, and selects the label box with the highest probability. As fill box I j The paired label boxes, where,

[0231] Based on this, the electronic device can determine the first matching label box for the first fill-in box indicated by the user. The electronic device constructs a Gaussian model by analyzing the relative position distribution of matching fill-in boxes and label boxes in a large number of form samples, and then uses this Gaussian model to find the first matching label box. By constructing a Gaussian model to find the matching first label box for the first fill-in box, the electronic device requires less computation and has higher accuracy.

[0232] S77: The electronic device obtains the first label name that matches the first input box corresponding to the cursor position.

[0233] S78: The electronic device outputs the first content text that matches the first tag name.

[0234] The electronic device matches the content text for the first input box based on the first label name of the first label box.

[0235] In practice, the electronic device can maintain a form information database, which stores the text to be filled in and its corresponding tag names. The electronic device determines the first tag name corresponding to the first fill-in box, searches the form information database for the text that matches the first tag name, and uses this text as the first content text that matches the first fill-in box. The electronic device then fills in this first content text into the first fill-in box on the first interface. This saves users from manually entering the form information and optimizes the user experience.

[0236] Based on the above implementation method, before executing the automatic form filling process, the electronic device can also detect whether the current scene is a form filling scene by recognizing the cursor trajectory. If it is determined to be a form filling scene, the subsequent automatic form filling process will be executed.

[0237] like Figure 11 The diagram shown illustrates the coordinates of the table frames involved in the information processing method provided in this application embodiment. The table includes input boxes 1101, 1102, and 1103. The vertical coordinate of input box 1101 is y1, the vertical coordinate of input box 1102 is y2, and the vertical coordinate of input box 1103 is y3. The electronic device can detect the area of ​​the user's current input operation based on the cursor coordinates.

[0238] In one scenario, the user touch operation received by the electronic device is a horizontal input operation or a line break operation within a fill box, and the y-value of the cursor changes little.

[0239] In another scenario, the user touch operation received by the electronic device is switching from one input field to another, resulting in a significant change in the cursor's y-value. For example... Figure 11 As shown, when the electronic device switches from fill box 1101 to fill box 1102, the change in the y-value is Δy1, where Δy1 = y1 - y2. When the electronic device switches from fill box 1102 to fill box 1103, the change in the y-value is Δy2, where Δy2 = y2 - y3. Based on this, the electronic device can set a second threshold, where the second threshold ∈ y If the change in the y-value of the electronic device exceeds the second threshold, confirm that a new line is filled in the form.

[0240] In one example, the electronic device provides a low-power listening strategy to monitor whether a form-filling scenario is in progress.

[0241] For example, electronic devices at the initial listening time T init Inside, monitor changes in cursor coordinates and calculate the change in the Y-value of the cursor coordinates. When |Δy i |=|y i+1 -y i |>ò y When the form is filled out and a new line is added, the electronic device enters continuous listening mode.

[0242] Electronic devices monitor cursor coordinates and analyze cursor trajectory characteristics. The cursor trajectory is then segmented, and the cursor state is determined based on the number of segments.

[0243] The electronic device segments the cursor movement sequence according to the changes in Y values, treating each segment as a line of input. It calculates the average change in X values ​​within each segment, the average Y values ​​for each segment, and then the average change in Y values ​​across all segments.

[0244] The electronic device first segments the sequence in which the cursor moves.

[0245] The electronic device will trace the cursor's trajectory within the time series T as follows: Where i represents the time step and n represents the total number of time steps. After the electronic device determines to begin continuous monitoring based on the change in the y-value, it continuously monitors and determines whether the change in the y-value exceeds the error threshold. y . When |Δy i |>ò y When the cursor is in a certain position, the current segment is considered to have ended, and a new segment begins. The start and end times of each segment are... Where k is the segment number.

[0246] When the number of segments K is greater than the specified value N segments At that time, the electronic device calculates the mean value of the change in X value within each segment. and the mean of the change in Y value between each segment

[0247] Electronic devices calculate the average change of X value within each segment The formulas include:

[0248] Electronic devices calculate the mean Y value within each segment The formulas include:

[0249] The electronic device then calculates the average change of X value within each segment. and the mean Y value within each segment Calculate the change in Y value between each segment Moving to the right means that the electronic device responds to the user's action and enters text content from left to right in the input box.

[0250] The electronic device acquires the mean change Δμ of the Y value between each segment. y , Δμ y >ΔY threshold , where ΔY threshold If the preset Y-axis interval threshold is used, it means that the Y-axis interval of each input box is relatively large, which is different from inputting long text.

[0251] If the electronic device determines that it meets the requirements of each segment And Δμ y >ΔY threshold If the cursor trajectory indicates a form-filling scenario, the electronic device can trigger automatic form filling, saving the user the effort of triggering automatic form filling and further improving the convenience of the form-filling process.

[0252] The information processing method provided in the above-described embodiments of this application can automatically trigger form filling by an electronic device based on cursor recognition in a form-filling scenario. The electronic device can also trigger automatic form filling based on user touch operations. Once the electronic device initiates the automatic form filling process, it identifies the fill-in box to be filled, matches a label box to the fill-in box, and matches content text to the fill-in box based on the label name of the label box. This saves users the trouble of manually entering content text and improves the convenience of form filling.

[0253] Example 3: The automatic form filling mode executed by the electronic device is one-click filling.

[0254] This application proposes an information processing method that supports automatic filling of all content in a form. The method includes: displaying a first interface; the first interface includes a table and a first control, excluding a second control for inputting or triggering the display of first information; the table includes a first table label and a control to be filled in, the first table label being the table label associated with the control to be filled in on the first interface. The first information includes all information to be filled in on the first interface. In response to a user operation on the first control, based on the first table label, the content to be filled in that match the first table label is entered into the control to be filled in. Thus, the electronic device can automatically fill in the form on the first interface with a single user operation, greatly improving the efficiency of form filling.

[0255] In some embodiments of this application, during the process of automatically filling out a form, the electronic device may display a prompt message to remind the user that the form is currently being automatically filled out.

[0256] Depending on the filling method, form fill controls can be divided into two different types. One type requires input, called an input control; the other provides a drop-down menu, requiring the user to select appropriate content from the menu to fill in, and is called a drop-down control. In some embodiments, the form autofill process can automatically fill in empty input controls and empty drop-down controls. That is, the aforementioned empty fill control specifically refers to an empty fill control, or it could be an empty drop-down control. Empty fill controls can also be named by other names, such as fill-in control, etc.

[0257] In the embodiments of this application, the automatic form filling process can be divided into the following parts: visual detection part, content to be filled acquisition part, content to be filled maintenance part, and filling execution part.

[0258] The system comprises several parts: a visual detection section and a filling execution section. The visual detection section performs visual inspection on the first interface after the automatic form filling process is triggered. It determines whether the first interface includes a table, and if a table is included, it identifies empty fill controls (including input controls and / or dropdown controls), their display positions, and the table labels associated with them, as well as their display positions. The content to be filled section acquires the content to be filled corresponding to the empty fill controls when the first interface includes a table. The content to be filled maintenance section maintains and updates the acquired content during the filling operation. For example, the phone can maintain and update the content to be filled based on the current form filling progress, such as deleting completed content or updating the coordinate information of the input controls corresponding to the content to be filled when the table shifts. The filling execution section executes the filling operation, automatically filling the acquired content to be filled into the corresponding empty fill controls.

[0259] In some embodiments, the automatic form filling process in this application also includes an exception handling section. This exception handling section is used to detect whether any exceptions occur during the filling operation and to handle any such exceptions. Specifically, the exception handling section is used to detect whether any exceptions occur after the filling operation for an empty fill control has been completed.

[0260] The information processing method proposed in the embodiments of this application will be described in detail below with reference to the accompanying drawings. Figure 12 The specific flow of the information processing method in some embodiments is shown.

[0261] S1201. In response to user operation, identify the table in the first interface.

[0262] In this system, user actions can trigger an automatic form filling process. The first interface includes a form and first controls, but does not include second controls for inputting or triggering the display of first information. The first information includes all information to be filled in on the first interface. In some embodiments, the user action may correspond to... Figure 2 The user triggers the one-click fill control. The one-click fill control can be the primary control.

[0263] In some embodiments, S1201 may specifically include: the mobile phone responding to a user operation by acquiring a screenshot of a first interface. Then, the screenshot of the first interface is identified to obtain a table therein. Acquiring the screenshot of the first interface can be achieved in any manner.

[0264] Optionally, as an example, the mobile phone can recognize the table in the first interface using a model. For example, the mobile phone can use a first model to recognize the table in the screenshot of the first interface. It should be noted that the specific implementation of recognizing the table using the first model will be described in detail in later embodiments. It should also be noted that the first model can be named by other names, such as a visual detection model.

[0265] The mobile phone identifies the first interface to determine whether it includes a table, and if so, identifies and obtains the table labels and associated fill controls. Furthermore, if the first interface includes a table, the phone can identify and determine the label area corresponding to each table label and the fill area corresponding to the associated fill control. This is used to locate the display positions of the table labels and fill controls on the first interface, facilitating automatic table filling later. Common label and fill areas are rectangular, and their display positions on the first interface can be represented by bounding boxes (bboxes), such as label boxes and fill boxes. For example, if the first interface includes a table, the phone can obtain the coordinates of the bounding boxes corresponding to the label and fill areas.

[0266] The mobile phone identifies the table in the first interface and obtains all table labels and the fill controls associated with each table label. The specific implementation of the mobile phone identifying the table in the first interface and determining the association between table labels and fill controls can be found in other embodiments or in related technologies. In practical scenarios, the table in the first interface may include some filled controls (i.e., filled controls) and some unfilled controls (i.e., empty fill controls). It is understood that in the embodiments of this application, when the mobile phone executes the automatic form filling process, it only needs to automatically fill in the empty fill controls. Therefore, after identifying the table in the first interface, the empty fill controls and the table labels associated with those empty fill controls can be obtained to facilitate automatic filling of the unfilled fill controls; as in S1202.

[0267] S1202. Obtain the empty fill control in the table, and the first table label associated with the empty fill control.

[0268] An empty fill control is a fill control in a table that needs to be filled in. A first table label is used to indicate the type of content to be filled in the empty fill control associated with the first table label. In the embodiments of this application, the empty fill control is specifically an input control to be filled in and / or an empty drop-down control in the table.

[0269] In some embodiments, S1202 may specifically include: obtaining the coordinate information of an empty fill control and the coordinate information of a first table label associated with the empty fill control. Specifically, the coordinate information of the empty fill control is the coordinate information of the fill area corresponding to the empty fill control; the coordinate information of the first table label is the coordinate information of the label area corresponding to the first table label. For example, the information obtained in S1202 may be represented as follows: Fill control 1: coordinate information of fill area 1, coordinate information of label area 1 associated with fill control 1; Fill control 2: coordinate information of fill area 2, coordinate information of label area 2 associated with fill control 2; ... Fill control n: coordinate information of fill area n, coordinate information of label area n associated with fill control n. Subsequently, a fill operation can be performed on the empty fill control based on the coordinate information of the fill area and the coordinate information of the label area associated with the fill area.

[0270] S1203. Retrieve the content to be filled that matches the label of the first form from the user information database.

[0271] To determine what should be filled in the blank fill control, after obtaining the coordinate information of the first form label associated with the blank fill control in S1202 above, it is also necessary to identify the content of the first form label in order to find the corresponding content to be filled. In some embodiments, S1203 may specifically include: identifying the label text corresponding to the first form label, and obtaining the content to be filled that matches the first form label from the user information database based on the label text.

[0272] In some embodiments, the label text corresponding to the first table label is identified, which can be achieved through text recognition technology. For example, a mobile phone can identify the label text corresponding to the first table label using an optical character recognition (OCR) model.

[0273] The user information database stores information that needs to be filled in for various forms, such as mailing addresses, personal information, and return addresses. For example, the user information database stores the following: form labels and associated label data. The label data is the data that needs to be entered into the empty fields in the form during the automatic form filling process. In some embodiments, the form labels and label data are stored in the user information database as key-value pairs; the form labels are the keys, and the label data is the values. In this embodiment, step S1203 includes: retrieving label data matching the first form label from the user information database, and using the matched label data as the content to be filled in.

[0274] The specific implementation of retrieving the content to be filled that matches the labels in the first form from the user information database will be described in detail in later embodiments.

[0275] S1204. According to the correspondence between the first table label, the blank fill control and the content to be filled, perform the filling operation on each blank fill control to fill in the content to be filled into the blank fill control.

[0276] In step S1202 above, the association between the empty fill control and the first form label is obtained. The content to be filled is retrieved from the user information database based on the first form label. Therefore, based on the above information, the mobile phone can determine the correspondence between the first form label, the empty fill control, and the content to be filled. Subsequently, the mobile phone can fill in each piece of content to be filled into the corresponding empty fill control based on the correspondence between the three.

[0277] In some embodiments, in S1204 above, the mobile phone performs a filling operation on each blank fill control, specifically by performing the filling operation on each blank fill control sequentially. Optionally, as an example, in S1204 above, the specific implementation logic for performing the filling operation on each blank fill control sequentially is as follows: the mobile phone selects one blank fill control as the target fill control and performs a filling operation on that target fill control; then it selects another blank fill control as the new target fill control and performs a filling operation on that new target fill control, and so on, until all blank fill controls have been filled with the corresponding content to be filled. For example, with Figure 2 In the interface 201 shown, taking the table corresponding to the mailing address as an example, there are four blank fill controls, which are the fill controls associated with the table labels "Name", "Phone Number", "Region", and "Detailed Address" (i.e., the first table label mentioned above). When performing fill operations on each blank fill control in sequence, the mobile phone can first perform fill operations on the blank fill control associated with the table label "Name", then perform fill operations on the blank fill controls associated with the table labels "Phone Number" and "Region", and finally perform fill operations on the blank fill control associated with the table label "Detailed Address".

[0278] In some embodiments, performing a filling operation on the target fill control may specifically include: simulating a click on the target fill control based on the coordinate information of the filling area corresponding to the target fill control; and filling the content to be filled in the filling area corresponding to the target fill control with the content matched by the first table label associated with the target fill control. The coordinate information of the filling area corresponding to the target fill control can be obtained in S1202.

[0279] Typically, after a user clicks on a fill control, the phone displays a keyboard for input, or enters keyboard mode. Optionally, in some embodiments of this application, the phone also displays a keyboard after a simulated click on the target fill control. After filling in the content to be filled into the corresponding target fill control, the phone needs to exit keyboard mode before another empty fill control can be selected as the new target fill control for filling. That is, in some embodiments, during the filling operation on the target fill control, after filling in the content to be filled into the corresponding fill area of ​​the target fill control, a keyboard exit operation is also performed. Alternatively, in other embodiments of this application, after a simulated click on the target fill control, the phone enters keyboard mode, but the keyboard may not be displayed on the display interface.

[0280] It should be noted that the specific implementation of the keyboard exit operation can be found in the descriptions in relevant technologies, and will not be elaborated here.

[0281] In some embodiments, the specific implementation of S1204 may include: generating a content vector based on the obtained content to be filled, the first form label, and the empty fill control. Then, multiple content vectors are stored in a preset data structure. The empty fill controls in the content vector can be specifically represented by the coordinate information of the fill area corresponding to the empty fill control. Afterwards, the mobile phone performs a fill operation on each empty fill control based on the preset data structure.

[0282] In some embodiments, the mobile phone performs a filling operation on each empty fill control based on a preset data structure. Specifically, this may include: retrieving a content vector from the preset data structure, such as a first content vector; determining the target fill control based on the coordinate information of the fill area in the first content vector; filling the target fill control with the content to be filled from the first content vector; returning to the step of retrieving a content vector from the preset data structure; and then, based on the newly retrieved content vector, performing a filling operation on the next empty fill control until the preset data structure is empty; that is, the filling operation has been completed for all empty fill controls.

[0283] For example, the preset data structure can be a queue, such as a queue to be filled.

[0284] The queue to be filled can include multiple queue elements. Each queue element stores a content vector, including a first form label, an empty form label, and content to be filled. The first form label, the empty form label, and the content to be filled correspond one-to-one. For example, a queue element is composed as follows: (first form label, content to be filled, coordinate information of the filling area corresponding to the empty form label).

[0285] In other embodiments, in order to better locate the display position of the empty fill control, the composition of the queue element also includes the coordinate information of the first table label, such as: the composition of a queue element is as follows: (first table label, content to be filled, coordinate information of the label area, coordinate information of the fill area).

[0286] Furthermore, the aforementioned mobile phone selects an empty fill control as the target fill control, and the specific implementation of the fill operation on this target fill control is as follows: The first queue element in the fill queue is retrieved, let's call it queue element A; based on the coordinate information of fill area A in queue element A, a simulated click operation is performed on fill area A (i.e., the empty fill control). The content to be filled in queue element A is then filled into fill area A. It can be understood that the empty fill control corresponding to fill area A in queue element A is the target fill control for this fill operation. Since the first queue element in the fill queue has already been retrieved, the second queue element in the previous fill queue is retrieved, becoming the first queue element in the subsequent fill queue. After the fill operation on the empty fill control corresponding to fill area A based on queue element A is completed, the first queue element can be retrieved again from the fill queue, let's call it queue element B. Then, using the empty fill control corresponding to fill area B in queue element B as the new target fill control, the above steps are repeated, performing a fill operation on this new fill control based on queue element B, until the fill queue is empty. Understandably, in this embodiment, when the queue to be filled is empty, it means that the filling operation has been completed for all empty fill controls.

[0287] By using the above method, based on the queue of fields to be filled, and performing a filling operation on each empty field control, the filling operation on all empty fields in the table can be completed.

[0288] In the technical solution proposed in the embodiments of this application, the mobile phone can automatically fill in the form on the first interface with one click in response to a single user operation, which can greatly improve the efficiency of filling in the form.

[0289] Furthermore, after completing the filling operation of one empty fill control in the table, and before moving to the next empty fill control for filling operation, some abnormal situations may occur. To respond smoothly to a single user operation and automatically fill in all empty fill controls with a single click, it is also necessary to detect whether abnormal situations have occurred and to handle them accordingly. The following describes in detail the process of detecting and handling abnormal situations, using an implementation method based on a queue of fill controls to be filled as an example. Figure 13 The corresponding process is shown.

[0290] In this embodiment, the queue to be filled includes n queue elements, and the structure of a queue element is: (k n ,v n bbox 表格标签n bbox 填写控件n ), where k n The label text representing the table label, v n Indicates the relationship with k n Matched tag data, bbox 表格标签n This represents the coordinate information of the label area corresponding to table label n, bbox. 填写控件n This indicates the coordinate information of the input area corresponding to the input control n.

[0291] S1301. Retrieve the first queue element (k1, v1, bbox). 表格标签1 bbox 填写控件1 According to bbox 填写控件1 Determine the display position of fill control 1, and fill v1 into fill control 1.

[0292] The specific implementation of S1 can be found in the description of performing a filling operation on an empty filling control based on the filling queue in the above embodiment.

[0293] Retrieve the first queue element (k1, v1, bbox) from the queue to be filled. 表格标签1 bbox 填写控件1 After that, the position of the element in the queue to be filled will change. For example, the second element in the queue before being taken out will become the first element in the queue after being taken out.

[0294] S1302. Check for any abnormalities.

[0295] In some embodiments, abnormal situations may include: keyboard mode not exiting. As described in the above embodiments, after the content to be filled in is entered into the filling area corresponding to the target filling control, the mobile phone will also perform a keyboard exit operation. Therefore, under normal circumstances, the keyboard should exit after the filling operation on the empty filling control is completed.

[0296] Optionally, as an example, the phone can detect whether there is an abnormal situation where keyboard mode has not been exited by identifying whether the first screen includes a keyboard. Specifically, if the first screen includes a keyboard, it indicates that there is an abnormal situation where keyboard mode has not been exited. Conversely, if the first screen does not include a keyboard, it indicates that there is no abnormal situation where keyboard mode has not been exited. Identifying whether the first screen includes a keyboard can be achieved in any of the following ways.

[0297] As another example, in an embodiment where the phone is in keyboard mode but the keyboard is not displayed, the phone can determine whether there is an abnormal situation where the phone has not exited keyboard mode by obtaining keyboard mode-related information from the background.

[0298] In other embodiments, abnormal situations may include table displacement. When a fill-in operation is performed on a fill-in control in a table, the table may shift. After the table shifts, the display position of the fill-in control changes. This change in the display position of the fill-in control may affect the automatic form filling process, causing the automatic fill-in operation to fail. For example, after a simulated click operation is performed on a fill-in control in the table, the cursor is activated; that is, the phone displays the cursor in the fill-in area corresponding to the empty fill-in control. The cursor is used to indicate the current operation position; the cursor position moves with the filled content. Figure 14 After a simulated click is performed on the fill control associated with detailed information, the cursor position is 1401; after the fill operation on the fill control is completed, the cursor position is 1402. In other scenarios, such as Figure 15 In the example shown, the cursor position is 1501 when the cursor is activated; after the cursor is activated, the table will shift. In this example, after the fill operation on the fill control is completed, the cursor position is 1502.

[0299] Optionally, as an example, detecting whether a table has shifted can be achieved as follows: Obtain the initial cursor position when the cursor is active, and the second cursor position after the filling operation on the current control has been completed. Calculate the difference between the y-coordinates of the first and second cursor positions. If the difference is greater than or equal to an error threshold, the table is considered to have shifted. The error threshold can be set according to the actual situation; specifically, the error threshold is a value greater than 0. The current filling control refers to the currently selected filling control that has completed the filling operation, i.e., the filling control where the cursor is located.

[0300] like Figure 14In the example, the first cursor position is 1401, and the second cursor position is 1402. These two positions have the same ordinate, and the difference is less than the error threshold; that is, in... Figure 14 In the example, the table was not shifted. However, in... Figure 15 In the example shown, the first cursor position is 1501, and the second cursor position is 1502. These two positions have different ordinates. The difference is greater than or equal to the error threshold; that is, in... Figure 15 The table in the example has been shifted.

[0301] If the result of S1302 is negative, it means that no abnormal situation has occurred, and S1304 can be executed directly. If the result of S1302 is positive, it means that an abnormal situation has occurred, so S1303 is executed first, and then S1304 is executed.

[0302] S1303. Execute exception handling.

[0303] The main purpose of handling exceptions where the keyboard mode is not exited is to control the phone to exit keyboard mode.

[0304] When the phone is in keyboard mode, a common way to exit the keyboard is to tap an area outside the keyboard on the first screen that is not associated with a fill control; for example, an empty area on the first screen or the area corresponding to a table label in a table. In some embodiments, exception handling for not exiting keyboard mode may include: determining the first table label associated with the current fill control, obtaining the coordinate information of the label area corresponding to the first table label, and performing a simulated tap operation on the label area based on the coordinate information of the label area.

[0305] When the phone is in keyboard mode, another common way to exit keyboard mode is to tap the controls corresponding to the "Done" or "Enter" keys in the keyboard area on the first screen. For example, the "Done" key control can display the text "Done" / "Next" / ..., or something like... Figure 14 The keyboard layout shows symbols such as "√". Similarly, the Enter key control can be displayed as the text "Enter" or the Enter symbol. In other embodiments, exception handling for not exiting keyboard mode may include: obtaining the coordinate information of a preset control within the keyboard area of ​​the first interface, and performing a simulated click operation on the preset control based on the coordinate information. The preset control may be the control corresponding to the Complete key or the Enter key.

[0306] When an anomaly occurs due to table shifting, the display position of empty fill controls that have not yet been filled in changes. The queue of fill controls generated before this will no longer correspond to the shifted table, and the empty fill controls that have not yet been filled in will not be able to automatically fill in based on this queue. Therefore, the main purpose of handling table shifting anomalies is to ensure that fill operations can continue to be performed on subsequent empty fill controls.

[0307] In some embodiments, exception handling for table shifting includes: after table shifting, re-acquiring the content to be filled in the shifted table and generating a new queue of content to be filled. That is, after determining that the table has shifted, the shifted table is re-identified to obtain the empty fill control and the table label associated with the empty fill control (denoted as the second table label). Then, based on the second table label in the shifted table, new content to be filled can be retrieved from the user database, and a new queue of content to be filled can be generated. Finally, the mobile phone can fill the new content to be filled into the empty fill control in the shifted table based on the new queue of content to be filled. The specific implementation of identifying the shifted table, obtaining the content to be filled, and filling the content to be filled into the empty fill control is consistent with the corresponding operation implementation in the above embodiments, and will not be repeated here.

[0308] In other embodiments, exception handling for table shifting includes updating the fill queue based on the shifted table. Typically, when a table is shifted, only the display positions of the table labels and fill controls change compared to the original table; the association between the table labels and fill controls, and the fill controls themselves, remains unchanged. However, the display position of empty fill controls that have not yet been filled at the time of table shifting may affect the execution of subsequent fill operations. As described in the above embodiments, the k elements in the fill queue... n The label text of the table labels. Therefore, the above-mentioned update of the queue to be filled based on the shifted table may specifically include: re-identifying the empty fill controls in the table (denoted as remaining empty fill controls), and the table labels associated with these remaining empty fill controls (denoted as remaining table labels), obtaining the coordinate information of the remaining empty fill controls, and the label text of the remaining table labels. Then, based on the label text of the remaining table labels, updating the coordinate information of the remaining empty fill controls to the corresponding queue elements in the queue to be filled.

[0309] It should be noted that the new queue to be filled and the updated queue to be filled generated for the shifted table correspond to the empty fill control that was not filled when the table was shifted.

[0310] In other embodiments, abnormal situations may include not exiting keyboard mode and table shifting. Optionally, as an example, when detecting whether an abnormal situation exists, it is possible to first check whether keyboard mode has not been exited, and then check whether the table has shifted if keyboard mode has been exited. In the case of the abnormal situation of not exiting keyboard mode, exception handling for keyboard mode not exiting is first performed, and then the keyboard mode is checked again after the exception handling is performed. If keyboard mode has been exited after the exception handling for keyboard mode not exiting is performed, then the table is checked for shifting. If keyboard mode has not been exited after the exception handling for keyboard mode not exiting is performed, the automatic form filling process can be terminated, and the check for table shifting is no longer performed.

[0311] Furthermore, during the initial check of whether keyboard mode has exited, if it is detected that keyboard mode has exited, it can further check whether the table has shifted abnormally. Subsequently, if the table has shifted, it can perform the abnormal handling for table shifting as described in the above embodiment, regenerating a new queue of cells to be filled, or updating the queue of cells to be filled; then it can continue to perform the filling operation on the next empty fill control, i.e., execute S1304. If it is detected that keyboard mode has exited and the table cell has not shifted, it can directly perform the filling operation on the next empty fill control, i.e., execute S1304.

[0312] In embodiments where abnormal situations may include not exiting keyboard mode and table shifting, the judgment result of S1302 is "yes," corresponding to the following scenario: when the abnormal situation is detected for the first time, keyboard mode has exited, and the table has not shifted. S1302 first checks whether keyboard mode has not exited. If so, S1303 is executed to perform exception handling for not exiting keyboard mode. Then, it checks whether keyboard mode has not exited. If keyboard mode has not exited, the automatic table filling process ends directly. If keyboard mode has exited, S1302 is executed again to check whether the table has shifted. If the table has shifted, S1303 is executed again to perform exception handling for table shifting. It should be noted that after performing exception handling for table shifting, S1304 can be executed directly.

[0313] S1304. Retrieve the first queue element (k2, v2, bbox). 表格标签2 bbox 填写控件2 According to bbox 填写控件2 Determine the display position of fill control 2, and fill v2 into fill control 2.

[0314] In the technical solution proposed in this application embodiment, after completing the filling operation on an empty fill control, an abnormal situation will be detected. Furthermore, if an abnormal situation occurs, the mobile phone will automatically perform the corresponding exception handling, without requiring user intervention. This ensures that the mobile phone can automatically complete the one-click filling of the entire form.

[0315] In some embodiments, the various parts of the above form autofill process can be implemented using state machines. State machines facilitate management and enable efficient and accurate form autofill. The following provides a detailed description of the state machines corresponding to each part of the above form autofill process.

[0316] Each state machine can include two parts: one part is the state execution action, which is used by the state machine to execute relevant logic; the other part is the state transition condition, which is used to determine the next operation to be performed based on the corresponding judgment conditions, such as jumping to another state machine or ending the automatic form filling process.

[0317] 1. The state machine corresponding to the visual inspection part, such as state machine 1 or visual inspection state machine, is implemented as follows: Figure 16 .

[0318] State of Action: Take a screenshot of the first interface to obtain a screenshot of the first interface. Use the first model to identify the screenshot of the first interface, detect the different categories of table elements included in the table in the first interface, and provide their respective bboxes. In this embodiment, the table elements are divided into five categories, including: ① Table label L t ② Unfilled input controls (i.e., blank fill controls) u ③ Input controls I that have already been filled f ④ Unfilled dropdown control (i.e., empty dropdown control) D u ⑤ The dropdown control D that has already been filled in f .

[0319] Furthermore, when the phone uses the first model to recognize the screenshot of the first interface, the table labels and associated input controls or dropdown controls will be matched in a sequence of bounding boxes. The following is provided. Each sequence element represents a pair of bboxes for a fill-in control and a table label. Specifically, the bbox of the input control is the bbox of the fill-in area corresponding to the input control; the bbox of the table label is the bbox of the label area corresponding to the table label. It should be noted that this part corresponds to S1201 and S1202 above. The following explanation uses the input control as an example.

[0320] For example, the bbox of an input control is represented as Its upper left corner coordinates are The coordinates of the bottom right corner are The bbox of the table label is represented as Its upper left corner coordinates are The coordinates of the bottom right corner are Paired bbox sequences The input control portion in the code can be represented as:

[0321]

[0322] in:

[0323]

[0324] Understandable, paired bbox sequences The dropdown control portion is represented similarly to the input control.

[0325] State transition condition: First determine the paired bounding box sequence. Is it empty? If the paired bbox sequence If empty, it means there are currently no fill controls in the table, and the automatic form filling process ends. If the paired bbox sequence... If not empty, then check if the queue Q to be filled is empty. If the paired bounding box sequence... If the field is not empty and the queue Q is empty, it means that no content matching the empty fill control has been retrieved yet; in this case, we can jump to state machine 2. If the paired bbox sequence... If the field is not empty and the queue Q is not empty, it means that a field matching the empty field control has been found; at this point, the process can jump to state machine 3.

[0326] 2. The state machine corresponding to the part where the content to be filled is to be retrieved, such as state machine 2 or the content to be filled retrieval state machine. The specific implementation of state machine 2 is as follows: Figure 17 .

[0327] State execution action: For the empty input control I identified in state machine 1 using visual inspection. u Related table label (the first table label mentioned above) L t and empty dropdown control D u Associated table label L t Combine OCR model to obtain table label L t The corresponding label text. And, the table label L... t The corresponding tag text is used as the key (key, K), and the tag data that matches the tag text corresponding to the table tag Lt is retrieved from the user information database and used as the value (value, V).

[0328] Use an empty input control I u The bbox, and the empty input control I u Associated table label L t The bounding boxes (bboxes) form a key-value pair vector, and this key-value pair vector forms a queue Q to be filled. The queue Q to be filled, including the content to be filled for empty input controls, can be represented as:

[0329]

[0330] Similarly, an empty dropdown control D u The bbox, and the empty dropdown control D u Associated table label L t The bounding boxes form a key-value pair vector.

[0331] The state operation actions of state machine 2 specifically correspond to S1203 in the above embodiment.

[0332] State transition condition: Check if the queue to be filled, Q, is empty. If the queue to be filled, Q, is not empty, then the filling operation can be performed; at this time, jump to state machine 4. If the queue to be filled, Q, is empty, then no matching table label L was found. t The corresponding tag data cannot be automatically filled in the table, so the automatic table filling process ends.

[0333] 3. The state machine corresponding to the content to be filled maintenance section, such as state machine 3 or the content to be filled maintenance state machine, is implemented as follows: Figure 18 .

[0334] State operation action: Obtain the bbox of the empty fill control re-detected by the visual inspection state machine. Update the bbox of the empty fill control in the fill queue Q to match the bbox of the empty fill control re-detected by the visual inspection state machine. This part corresponds to the exception handling process in S1302 and S1303 of the above embodiment when there is an abnormal table situation.

[0335] State transition condition: After the state operation action is completed, it can jump to state machine 4.

[0336] 4. Fill in the state machine corresponding to the execution part, such as state machine 4 or the execution state machine. The specific implementation of state machine 4 is as follows: Figure 19 .

[0337] Status operation action: First, check if the queue to be filled Q is empty. If the queue to be filled is empty, it means that all empty fill controls have been automatically filled, and the automatic form filling process ends.

[0338] Conversely, if the queue Q to be filled is not empty, it means that there are still empty input controls in the first interface that need to be automatically filled. The following explanation uses the empty input control corresponding to the first queue element as an example. Retrieve the first queue element from the queue Q to be filled. Based on the first queue element Determine empty input control I u The coordinates of the top left corner of the bbox and the coordinates of the bottom right vertex Calculate the empty input control I u Center coordinates:

[0339]

[0340] against center coordinates Perform a simulated click to activate the cursor. Obtain the cursor's position (i.e., the initial cursor position mentioned above). and the value v in the first queue element i Inject the empty input control I u The cursor position can specifically be the coordinates of the top-left corner of the cursor. Finally, control the phone to exit keyboard mode.

[0341] It should be noted that if the first queue element in the queue to be filled Q corresponds to an empty drop-down control, its implementation is similar to the specific implementation of the filling process of an empty input control, and will not be elaborated here.

[0342] The state operation action of state machine 4 corresponds to the process of performing a filling operation on an empty filling control in S1204 of the above embodiment.

[0343] State transition condition: Check again whether the queue Q to be filled is empty. If the queue Q to be filled is empty, it means that all empty fill controls have been automatically filled, and the form autofill process ends. Otherwise, jump to state machine 5.

[0344] 5. State Machine for Exception Handling. The following explanation uses exceptions such as failure to exit keyboard mode and table shifting as examples. The state machine for exception handling includes state machines 5, 6, and 7.

[0345] State machine 5, also known as the keyboard exit anomaly detection state machine, is used to detect whether there is an abnormal situation where the keyboard mode has not been exited. In this embodiment, the example of the phone displaying the keyboard while in keyboard mode is not exited is used for illustration. The specific implementation of state machine 5 is as follows: Figure 20 :

[0346] Status operation action: Detect whether the keyboard exists.

[0347] The state operation of state machine 6 corresponds to the process in the above embodiment for detecting whether there is an abnormal situation where the keyboard mode has not been exited.

[0348] State transition condition: Determine if the keyboard exists. If the keyboard exists, it indicates an exception where keyboard mode has not been exited, and the process jumps to state machine 7. Conversely, if the keyboard does not exist, it indicates no exception where keyboard mode has not been exited, and the process jumps to state machine 6.

[0349] State machine 6, also known as the table shift exception handling state machine, is used to detect whether an exception has occurred in table shifting and, if so, to determine the next action to be taken. The specific implementation of state machine 6 is as follows: Figure 21 :

[0350] Status execution action: Get the cursor position (i.e., the second cursor position mentioned above). according to The cursor position obtained in state machine 4 Calculate the cursor vertical offset value

[0351] The state operation of state machine 6 corresponds to the process of detecting whether the table has shifted in the above embodiment.

[0352] State transition condition: Determine if the cursor's vertical offset value is greater than or equal to the error threshold. If If the cursor's vertical offset is less than the error threshold ε, the automatic table filling process can continue, i.e., jump to state machine 4. Conversely, if the cursor's vertical offset is greater than or equal to the error ε, it indicates that an abnormal situation of table shifting has occurred after the current filling control has been filled. When an abnormal situation of table shifting occurs, the mobile phone can re-identify the shifted table, thus jumping to state machine 1.

[0353] State machine 7, also known as the keyboard exit exception handling state machine, is used to re-execute the keyboard exit operation if the exit fails. The specific implementation of state machine 7 is as follows... Figure 22 :

[0354] Status running action: Execute keyboard exit operation.

[0355] The keyboard exit operation can be performed in two ways: Method 1 and / or Method 2.

[0356] Method 1: Get the cursor position According to location Determine the target click location. Then, perform a simulated click at that location to exit the keyboard.

[0357] For example, the mobile phone can calculate the center coordinates of the bbox of the form label associated with the current fill-in control based on the bbox of the fill-in control and the bbox of the form label, and use the center coordinates of the form label as the target click position. The center coordinates of the form label's bbox... as follows:

[0358]

[0359] Method 2: Locate the display position of the corresponding control on the keyboard, such as the Complete key or Enter key, using the first model. Then, perform a simulated click operation on the display position of the corresponding control. This method exits the keyboard by confirming the input.

[0360] In some embodiments, when an exception occurs where keyboard mode is not exited, the keyboard that failed to exit can be a safety keyboard. The state operation of state machine 7 corresponds to the exception handling process for keyboard mode not exiting in the above embodiments.

[0361] State transition condition: Check if the keyboard exists again. If it still exists, the form autofill process ends. If, after handling the exception of not exiting keyboard mode, the keyboard does not exist, then jump to state machine 6.

[0362] In the above embodiments, the information processing method implemented through a state machine can efficiently and accurately achieve one-click automatic filling of forms. One-click filling of multiple fill controls in the form can improve form filling efficiency.

[0363] Furthermore, after the form is filled out, the content can be extracted from the completed form and updated to the user information database. Therefore, in some embodiments, the above method may further include:

[0364] In response to user actions, the first model is used to identify the first interface screenshot, detecting the different categories of table elements included in the table and providing their corresponding bounding boxes (bboxes). It should be noted that the table in this first interface is already filled out. Since this method extracts content from a completed table, it primarily focuses on the already filled input controls (I). f And the already filled dropdown control D f The table labels and their associated table tags are extracted. Furthermore, during table recognition, it is also necessary to determine the relationship between the table labels and the filled-in input controls. f The relationships between the table labels and the already filled dropdown controls D f The relationship between the table labels and the filled-in input controls. For example, in the output of the first model, the table labels and the filled-in input controls are related. fThe relationships between the table labels and the already filled dropdown controls D f The relationships between elements can be represented using key-value pairs.

[0365] Furthermore, based on the OCR model, the filled input control I is obtained. f The text, and the associated table label L t The label text. Then obtain the filled input control I. f The corresponding key-value pairs can be represented as follows:

[0366]

[0367] Similarly, based on the OCR model, it is also possible to obtain the filled dropdown control D. f The text, and the associated table label L t The label text in the dropdown control. This allows you to obtain the filled dropdown control D. f The corresponding key-value pairs can be represented as follows:

[0368]

[0369] To avoid retrieving duplicate form content, after obtaining the already filled input control I... f The corresponding key-value pairs, and the already filled dropdown control D f After the corresponding key-value pairs, the mobile phone can also perform a union operation on the two key-value pairs, and the union is used as the final extracted table content. Specifically, the union operation on two key-value pairs can be performed with the table label's bbox as the reference: for a table label's bbox, only one associated fill control, input control, or dropdown control is retained. Specifically, the union of formula (6) and formula (7) can be expressed as:

[0370]

[0371] In obtaining KV total Afterwards, the mobile phone can transmit the KV. total Stored in the user information database. Optionally, as an example, the phone will store the KV (Keywords, Values, etc.) total Stored in the user information database, specifically the key-value pair (KV). total Stored in the user information database. As another example, the phone will store the KV (Keywords, Values, etc.) total Before storing the extracted key-value pairs in the user information database, further processing can be performed on the extracted key-value pairs. total The data is compared with the user information database. If duplicate data is found, only the key-value pairs extracted this time can be used. total The updated data was stored in the user information database.

[0372] In the technical solution proposed in this application embodiment, by performing screenshot recognition on the first interface, the control text of the filled-in control and the tag text of the table label associated with the filled-in control can be extracted, thereby generating key-value pairs between the table label and the filled-in control. These key-value pairs can help update the user information database, maintaining the richness and accuracy of the data in the user information database.

[0373] Furthermore, this application also proposes an information processing method specifically for extracting content from completed forms. For example, this method can be applied after a user has manually filled out a form. The specific implementation of this method is detailed in the above embodiments regarding the extraction of content from a completed form.

[0374] Based on the exemplary information processing method described in the above embodiments, during the automatic form filling process, the electronic device needs to obtain user information from a user information database and fill in the user information into the correct position in the form. Therefore, the electronic device needs to know the position of the fill boxes in the form and the type of the form that has a pairing relationship with the fill boxes in order to obtain the correct user information.

[0375] For example, embodiments of this application provide a table visual detection method, an electronic device, a computer-readable storage medium, and a computer program product. With the help of a table visual detection module and a non-maximum suppression (NMS) algorithm, the electronic device can not only locate and classify table labels and fill boxes in the table, but also match table labels with fill boxes in the table, thereby realizing automatic table filling.

[0376] The electronic devices may include, but are not limited to: mobile phones, tablets, laptops, ultra-mobile personal computers (UMPCs), netbooks, personal digital assistants (PDAs), wearable devices, in-vehicle devices, augmented reality (AR) / virtual reality (VR) devices, smart TVs, and other electronic devices. This application does not limit the scope of these devices.

[0377] The table styles are diverse, such as Figure 23A and Figure 23B The interface displays various table styles. These tables have a tabular structure; that is, they are tables. Tables can include, but are not limited to, tables within applications, web forms in browsers, such as online registration forms, completed surveys, and courier information forms.

[0378] A table may include one or more objectives, and the objective categories may include table labels and fill boxes, such as... Figure 23A and Figure 23B As shown.

[0379] Table labels are typically identifying text or symbols used in tables to describe their content, attributes, or categories. For example, table labels may include a label name, or a label name and a label box. The label name can be... Figure 23A and Figure 23B The "Name" is shown. The label box can be... Figure 23A and Figure 23B The box shown encloses the "Name".

[0380] In this application, the label box mentioned in the embodiments can also be called a prompt box or a prompt control. The label name mentioned in this application embodiments can also be called label text.

[0381] A fill box typically refers to an area where a user can input or edit data. In this application's embodiments, the fill box may also be referred to as a fill control.

[0382] Below, we will elaborate on various specific implementation methods for the description fill-in box.

[0383] In some embodiments, such as Figure 23A and Figure 23B As shown, the input fields can include: an unfilled input field, a filled input field, an unfilled dropdown list, and a filled dropdown list.

[0384] In some embodiments, such as Figure 23A and Figure 23B As shown, the fill-in box can include: an input box and a drop-down box. The input box includes both blank and filled input boxes. The drop-down box includes both blank and filled drop-down boxes.

[0385] An input box typically refers to an interactive control that allows users to input text, numbers, and other characters. A dropdown box, also known as a dropdown list or combo box, is an interactive control that provides a limited number of options for the user to choose from. The input box mentioned in the embodiments of this application can also be called an input control. The dropdown box mentioned in the embodiments of this application can also be called a dropdown control.

[0386] In some embodiments, such as Figure 23A and Figure 23B As shown, the fill box can include: an unfilled fill box and a filled fill box. An unfilled fill box includes an unfilled input box and an unfilled dropdown list. A filled fill box includes a filled input box and a filled dropdown list.

[0387] As can be seen, the target categories in the table can include five types: table labels, blank input boxes, filled input boxes, blank dropdown lists, and filled dropdown lists.

[0388] Additionally, the input field may include information elements. When the input field is not filled in, the information element may display a prompt, such as... Figure 23A The text appears to be a series of seemingly unrelated phrases and sentences, making it difficult to translate coherently. A literal translation would be nonsensical. Therefore, a more accurate approach is to identify the individual phrases and translate them separately, acknowledging the lack of context and grammatical structure. Figure 23B The name "Zhao Yiyi" is shown. The prompt content mentioned in this application embodiment can also be called prompt text, and the filled content mentioned in this application embodiment can also be called filled text.

[0389] Of course, the target categories can also include two types: table labels and fill boxes.

[0390] Alternatively, the target category can include three types: table labels, input boxes, and dropdown lists.

[0391] Alternatively, the target category can include three types: form labels, unfilled fields, and filled fields.

[0392] Please see Figure 24 , Figure 24 This is a schematic diagram of a partial software structure of an electronic device provided in an embodiment of this application. For example... Figure 24 As shown, the first model in the electronic device may include: an image preprocessing module 2411, a table visual detection module 2412, and an NMS algorithm module 2413.

[0393] The first model is used to perform visual inspection of the tables in the interface, which can identify the matching targets in the table that have a pairing relationship, so that based on one of the matching targets, the user information to be filled in by the other target in the matching target can be accurately determined.

[0394] The image preprocessing module 2411 can preprocess parameters of the interface, such as size, dimensions, pixels, or grayscale, to obtain a preprocessed interface. The interface includes tables. For example, Figure 24 In this context, the interface can be the first interface described later. The format of the preprocessed interface conforms to the input format of the table visual inspection module 2412.

[0395] In some embodiments, the image preprocessing module 2411 can adjust the size or space occupied by the table interface, thereby reducing storage space and accelerating transmission speed. And / or, the image preprocessing module 2411 can normalize the table interface, adjusting the pixel values ​​of the interface to a specific range, facilitating subsequent use of the interface and ensuring the performance and stability of interface processing.

[0396] Therefore, the image preprocessing module 2411 can transmit the preprocessed interface to the table visual inspection module 2412.

[0397] The image preprocessing module 2411 is software code or a software module. The image preprocessing module 2411 can be located in the memory of an electronic device and can be accessed by a processor such as a central processing unit (CPU) or a graphics processing unit (GPU) in the electronic device.

[0398] The table visual inspection module 2412, based on the preprocessed interface, can perform detection such as localization, classification and matching on multiple targets in the table, and obtain multiple detection results.

[0399] Each detection result corresponds to a set of paired bounding boxes. Each set of paired bounding boxes can include multiple bounding boxes, such as two bounding boxes. The multiple bounding boxes correspond to targets of different target categories in the table.

[0400] For example, in two bounding boxes, if the target category of one bounding box is a table / note, then the target category of the other bounding box can be a fill box, or the target category of the other bounding box can be a drop-down list or an input box, or the target category of the other bounding box can be an unfilled fill box or a filled fill box, or the target category of the other bounding box can be any one of an unfilled input box, a filled input box, an unfilled drop-down list, and a filled drop-down list.

[0401] Each detection result can include relevant feature information of each bounding box in the paired bounding boxes, such as location and confidence score for each target category. Here, "each target category" refers to the specific class of the target.

[0402] In some examples, the table visual detection module 2412 can extract feature maps from the preprocessed interface. The feature map F can be represented as... Typically, it represents the set of real numbers, where C is the number of channels, H is the height of the feature map, and W is the width of the feature map.

[0403] The table visual detection module 2412 can apply multiple convolutional layers to the feature map. That is, the table visual detection module 2412 performs convolution operations on the feature map, which can reduce the number of channels C to the output dimension required by the table visual detection module 2412.

[0404] In some embodiments, the table visual detection module 2412 can represent the above convolution operation process using the tensor C in Formula 7:

[0405] C = Conv2D(F, 2 × (4 + K)) Formula 7;

[0406] Here, Conv2D represents a two-dimensional convolution operation; 2×(4+K) indicates that two bounding boxes are predicted at each spatial location of the feature map. Each bounding box contains four coordinate values ​​and confidence scores for K categories. The four coordinate values ​​represent the position of the corresponding bounding box, such as coordinates and size. For example, the four coordinate values ​​include the coordinates of the top-left vertex of the bounding box, the width of the bounding box, and the height of the bounding box. K is the number of target categories.

[0407] After the convolution operation, the tensor C output by the table visual detection module 2412 has a size of (2×(4+K))×H×W, meaning that the table visual detection module 2412 can output the relevant feature information of multiple paired bounding boxes at each spatial location of the feature map. In this way, the table visual detection module 2412 can realize the localization, classification, and pairing of various targets in the table.

[0408] Thus, the table visual detection module 2412 can transmit the relevant feature information of multiple sets of paired bounding boxes to the NMS algorithm module 2413.

[0409] In this application, the specific implementation of the table visual detection module 2412 is not limited. In some embodiments, the various functions of the table visual detection module 2412 are further combined with... Figure 24 The table visual detection module 2412 may include: a feature extraction module 24121, a feature post-processing module 24122, and a table label matching detection head module 24123.

[0410] The feature extraction module 24121 can extract features from the preprocessed interface to generate a feature map. The feature extraction module 24121 can transmit the feature map to the feature post-processing module 24122. The feature post-processing module 24122 performs post-processing and feature selection on the feature map to obtain a processed feature map. Therefore, the feature post-processing module 24122 can transmit the processed feature map to the table label matching detection head module 24123.

[0411] The table label pairing detection head module 24123 can locate, classify, and pair various targets in the table based on the processed feature map, obtaining multiple detection results, i.e., multiple sets of related feature information of paired bounding boxes. In other words, the table label pairing detection head module 24123 can detect various targets in the table and determine the target category, location, and pairing relationship of each target.

[0412] Thus, the table label pairing detection head module 24123 transmits the relevant feature information of multiple sets of paired bounding boxes to the NMS algorithm module 2413.

[0413] In the table visual detection module 2412, the feature extraction module 24121 and the feature post-processing module 24122 can be different software modules or integrated into the same software module. The feature extraction module 24121 and the feature post-processing module 24122 can employ traditional feature extraction algorithms, such as the object detection algorithm (You Only Look Once, YOLO). The feature extraction module 24121 can employ algorithms such as convolutional neural networks. The feature post-processing module 24122 can employ algorithms such as those based on feature pyramid networks (FPN).

[0414] The table label matching detection head module 24123 is a software module. The table label matching detection head module 24123 can be set separately from the feature extraction module 24121 and the feature post-processing module 24122, or it can be integrated with the feature extraction module 24121 and the feature post-processing module 24122.

[0415] The table label matching detection head module 24123 needs to be trained in order to locate, classify, and match the various targets in the table. In some embodiments, the table label matching detection head module 24123 can be trained alone, or it can be trained together with the feature extraction module 24121 and the feature post-processing module 24122, that is, the entire table visual detection module 2412 is trained.

[0416] In some instances, training samples are typically used to train the table visual detection module 2412, enabling the table label matching detection head module 24123 in the table visual detection module 2412 to accurately locate, classify, and match each target in the table.

[0417] The training samples include multiple tables, and each table can be labeled with attributes such as location, target category, and pairing relationship.

[0418] Additionally, when the target category is a fill box, the target's style and / or the number of words in the target's prompt can also be tagged, which helps to distinguish whether the target category is a table label or a fill box.

[0419] Additionally, when the target category is a dropdown list, a symbol is used to indicate that the target is a dropdown list, such as... Figure 23A and Figure 23B The symbol ">" shown can also be used for labeling, which helps to distinguish whether the target category is an input box or a drop-down box.

[0420] In addition, when the target category is a fill box, the status of whether the target is filled can also be tagged, which helps to distinguish whether the target category is an unfilled fill box or a filled fill box.

[0421] It should be noted that the embodiments of this application do not limit the information of the labeled targets in the training samples.

[0422] In other instances, by combining a multi-task loss function and training samples to train the table visual detection module 2412, the table label matching detection head module 24123 in the table visual detection module 2412 can more accurately locate, classify, and match each target in the table, taking into account the losses caused by localization, classification, and matching.

[0423] The NMS algorithm module 2413 can filter out paired bounding boxes with high confidence and low overlap from multiple paired bounding boxes based on the relevant feature information of multiple paired bounding boxes, remove duplicate and low-quality paired bounding boxes, and obtain filtered multiple paired bounding boxes, thereby ensuring that these paired bounding boxes do not overlap excessively in the same category.

[0424] In this way, the NMS algorithm module 2413 can determine each target in the table and the targets that have a pairing relationship with the corresponding targets based on the selected multiple sets of paired bounding boxes, that is, multiple sets of paired targets.

[0425] For example, for each table label in the table, the NMS algorithm module 2413 can determine the fill box that has a pairing relationship with each table label based on the multiple sets of paired bounding boxes selected.

[0426] Thus, the NMS algorithm module 2413 can obtain the pairing relationships and related feature information of multiple pairs of paired targets in the table, such as the confidence level of location and target category.

[0427] The NMS algorithm module 2413 is software code or a software module. The NMS algorithm module 2413 can be set in the memory of the electronic device and called by the processor such as the CPU or GPU of the electronic device.

[0428] Below, embodiments of this application will be used to illustrate the use of... Figure 24 Taking the structure shown as an example, and in conjunction with the accompanying drawings and application scenarios, the table visual inspection method provided in this application will be described in detail.

[0429] Please see Figure 25 , Figure 25 This is a flowchart illustrating a table visual inspection method provided in an embodiment of this application. Figure 25 As shown, the table visual detection method provided in this application embodiment may include:

[0430] S2501. Obtain the first interface, which includes multiple targets of the table.

[0431] When the first interface includes multiple objects such as forms, the electronic device can acquire the first interface in various ways based on the user's filling intention. In some embodiments, after receiving an operation on a first control or a hardware button, the electronic device can capture an image displayed on the screen.

[0432] The first control is a virtual control on the image, and operations on the first control may include, but are not limited to, touch operations such as single-click, double-click, and swipe. The hardware button is a physical entity in the electronic device, and operations on the hardware button may include, but are not limited to, physical pressing operations such as single-click and double-click.

[0433] When the format of the image conforms to the input format of the table visual inspection module 2412, the electronic device can identify the image as the first interface.

[0434] When the image format does not conform to the input format of the table visual detection module 2412, the electronic device can preprocess the image through the image preprocessing module 2411 to make the image format conform to the input format of the table visual detection module 2412, thus obtaining a preprocessed interface. Therefore, the electronic device can determine the preprocessed interface as the first interface.

[0435] In summary, electronic devices can obtain the first interface based on the user's input.

[0436] S2502. Detect multiple targets in the table and obtain multiple detection results. The detection results include the position of a set of paired bounding boxes and the confidence of each target category. A set of paired bounding boxes includes multiple bounding boxes that have a pairing relationship.

[0437] The electronic device can perform detection such as locating, classifying and matching the targets in the table based on the first interface, and obtain multiple detection results.

[0438] Each detection result includes relevant feature information of a set of paired bounding boxes. A set of paired bounding boxes may include multiple bounding boxes, and there are pairing relationships between the multiple bounding boxes. In some embodiments, the multiple bounding boxes may be two bounding boxes, or three or more bounding boxes.

[0439] A detection result includes relevant feature information for all bounding boxes within a paired bounding box. The relevant feature information for a bounding box can include its location and the confidence score for each target category. The confidence score for a target category refers to the probability that the target belongs to a particular target category.

[0440] For example, the position of a bounding box can be represented using the coordinates of one of its vertices, its width, and its height. Alternatively, the position of a bounding box can be represented using the coordinates of its four top and bottom points. Or, the position of a bounding box can be represented using the coordinates of the intersection of its two diagonals, its width, and its height.

[0441] In a set of paired bounding boxes, the target categories of each bounding box must be at least different. For example, when multiple bounding boxes include a first bounding box and a second bounding box, the target categories of the first bounding box and the second bounding box must be different.

[0442] When the target category of the first bounding box is a table label, the target category of the second bounding box can be any one of the following: an unfilled input box, a filled input box, an unfilled dropdown list, and a filled dropdown list. Alternatively, the target category of the second bounding box can be either an unfilled or a filled input box. Alternatively, the target category of the second bounding box can be either an input box or a dropdown list. Alternatively, the target category of the second bounding box can be either a filled input box.

[0443] For example, when multiple bounding boxes include a first bounding box, a second bounding box, and a third bounding box, at least two of the target categories corresponding to the first bounding box, the second bounding box, and the third bounding box are different.

[0444] S2503. Based on multiple detection results, pair multiple targets in the table to obtain one or more pairing results. The pairing results include the location of a set of paired targets and the confidence level of the target category. A set of paired targets includes multiple targets that have a pairing relationship.

[0445] The electronic device can obtain all paired bounding boxes from multiple detection results. Among all paired bounding boxes, there may be multiple pairs of paired bounding boxes corresponding to a single target in the table. Therefore, the electronic device can use the NMS algorithm module 2413 to determine a pair of paired bounding boxes corresponding to each target in the table from all paired bounding boxes based on multiple detection results.

[0446] Thus, the electronic device can determine a set of paired bounding boxes corresponding to each target in the table. In this way, the electronic device can determine which targets in the table have paired relationships. The two targets that have a paired relationship are called paired targets.

[0447] The electronic device can also determine the relevant characteristic information of each target in the table. This relevant characteristic information may include the target's location and target category.

[0448] For example, in a pair of bounding boxes corresponding to target 1, the target corresponding to the first bounding box is target 1, and the target corresponding to the second bounding box is target 2, which has a pairing relationship with target 1.

[0449] Therefore, Target 1 and Target 2 can be considered paired targets. The position of the first bounding box can be regarded as the position of Target 1, and the target category of the first bounding box can be regarded as the category of Target 1. The position of the second bounding box can be regarded as the position of Target 2, and the target category of the second bounding box can be regarded as the category of Target 2.

[0450] In summary, electronic devices can obtain relevant feature information of one or more pairs of paired targets in the table.

[0451] Taking a pair of bounding boxes where one bounding box corresponds to a table label and the other bounding box corresponds to an input box / dropdown list as an example, a pairing result can be represented as:

[0452] Label box (bbox): Category confidence score for the label box; Input box / dropdown box (bbox): Category confidence score for the input box / dropdown box.

[0453] In this context, the label box (bbox) refers to the position of the table label. The category confidence score of the label box refers to the confidence score of the table label. Similarly, the input / dropdown box (bbox) refers to the position of the input / dropdown box. The category confidence score of the input / dropdown box refers to the confidence score of the input / dropdown box. "bbox" is usually an abbreviation for bounding box.

[0454] In some embodiments, the electronic device can store relevant feature information of each pair of targets as queue elements in a queue to be filled. During the form filling process, the electronic device can automatically fill in the form based on the queue elements in the queue to be filled and the user information in the user information database.

[0455] The table visual detection method of this application embodiment acquires a first interface through an electronic device, enabling the identification of multiple targets within the table on the first interface. The electronic device can detect the multiple targets in the table to locate, classify, and pair each target, thereby obtaining multiple detection results. Each detection result includes the position of each pairing bounding box and the confidence level of each target category, such that each target corresponds to one or more pairs of paired bounding boxes. Based on the multiple detection results, the electronic device can pair the multiple targets in the table, obtaining one or more pairing results. It can filter out paired bounding boxes with high repetition and low confidence, ensuring that each target corresponds to a pair of paired bounding boxes. Thus, the electronic device can determine one or more pairs of paired targets among the multiple targets. Furthermore, based on the position of a pairing bounding box and the confidence level of each target category, the electronic device can obtain the position and category of each pair of paired targets. In other words, the electronic device can determine which targets(s) each target is paired with, as well as the position and category of that target, from among the multiple targets.

[0456] Thus, the electronic device can obtain the location and target category of one or more pairs of targets in the table, which facilitates the recommendation of user information to be filled in for the other target in the pairing based on one of the targets during the automatic filling process. This allows for a process of writing and filling in simultaneously, or the user information to be filled in for the other target in the pairing can be filled in directly on the interface, allowing the filling process to be completed in one go.

[0457] In this way, multiple targets are matched for the tables in the interface, enabling electronic devices to automatically fill in various forms based on the matching relationships. This is not limited to specific form styles and can comprehensively cover all types of forms, avoiding problems such as filling errors, frequent filling, and high filling requirements, thus improving the user's filling experience and efficiency.

[0458] Based on the description of the above embodiments, taking the position of the bounding box as represented by the coordinates of a vertex of the bounding box, the width of the bounding box, and the height of the bounding box, and taking a pair of paired bounding boxes as an example, in some embodiments, the multiple detection results obtained by S2501 are represented by tensor O in Formula 1.

[0459]

[0460] Where B is the number of first interfaces; N is the total number of multiple detection results; (x1,y1,w1,h1) is the coordinates and size of one of the bounding boxes in a set of paired bounding boxes; (x2,y2,w2,h2) is the coordinates and size of the other bounding box in a set of paired bounding boxes. It is the confidence score of one of the bounding boxes across K categories; It is the confidence score of the K target categories in another bounding box; K is the number of target categories.

[0461] In summary, an electronic device can process a single interface to determine the matching targets in the tables contained within that interface. Alternatively, the electronic device can process multiple interfaces in batches to determine the matching targets in the tables contained within each interface.

[0462] Based on the description of the above embodiments, in S2502, the electronic device can combine the table visual detection module 2412 mentioned above to locate and classify multiple targets in the table.

[0463] In some embodiments, the electronic device can input a first interface into the table visual detection module 2412. The table visual detection module 2412 can locate, classify, and match targets on the first interface, and output multiple detection results.

[0464] By combining the multi-task loss function L, the table visual detection module 2412 can be trained using training samples, so that the table visual detection module 2412 can take into account the losses caused by localization, classification and matching, and can more accurately locate, classify and match each target in the table.

[0465] The multi-task loss function L can be expressed by Equation 2:

[0466] L=αL cls +βL bbox Formula 2;

[0467] Among them, L cls It is a classification loss function, used to characterize the accuracy of the bounding box in classifying the target category; L bbox α is the paired bounding box regression loss function, which is used to characterize the accuracy of paired bounding boxes; α and β are weight coefficients, which are used to balance the influence of the classification loss function and the paired bounding box regression loss function.

[0468] In summary, the multi-task loss function L can include the classification loss function L. cls and paired bounding box regression loss function L bbox The weighted sum is used to optimize the performance of classifying, locating, and matching the various targets in the table.

[0469] In some embodiments, the classification loss function may be the classification loss function L. cls This is used to measure the accuracy of classifying and locating the various targets in the table.

[0470] The cross-entropy loss function can be expressed by Equation 3;

[0471]

[0472] Where N is the total number of samples; K is the number of target categories; y i,1,k and y i,2,k p is the true label of the paired bounding box of the i-th sample; i,1,k and p i,2,k γ is the probability that the i-th sample belongs to the target class k; γ is the penalty coefficient used for the loss of the weighted paired bounding boxes.

[0473] In some embodiments, the bounding box regression loss function L bbox A smooth L1 loss function can be used. This is used to measure the accuracy of matching the various targets in the table.

[0474] Among them, the paired bounding box regression loss function L bbox This can be represented by Formula 4;

[0475]

[0476] Where N is the total number of samples; It is the true paired bounding box of the i-th sample; These are the predicted paired bounding boxes; smooth L1 loss function. The definition is as follows:

[0477]

[0478] Based on the description of the above embodiments, in S2503, the NMS algorithm module 2413 in the electronic device can use various methods to effectively process the paired bounding boxes output by the table visual detection module 2412, filter out duplicate and low-quality paired bounding boxes, and obtain one or more paired results.

[0479] Below, in conjunction with Figure 26A and Figure 26B This document details the process by which an electronic device filters out duplicate and low-quality bounding boxes based on their confidence level and overlap. The confidence level refers to the probability that the target corresponding to the bounding box belongs to a specific target category. The overlap refers to the degree or area of ​​positional overlap with other bounding boxes.

[0480] Please see Figure 26A and Figure 26B , Figure 26A This is a flowchart illustrating a table visual inspection method provided in an embodiment of this application. Figure 26B This is a schematic diagram illustrating an intersection-union ratio (IUR) according to an embodiment of this application. Figure 26A As shown, the table visual detection method in this application embodiment may include:

[0481] S2601. Determine the weighted confidence and weighted overlap of all paired bounding boxes in multiple detection results.

[0482] The electronic device can obtain relevant feature information of all paired bounding boxes from multiple detection results. For any set of paired bounding boxes, the electronic device can determine the confidence level of each target type for each bounding box in that set. Therefore, the electronic device can weight the confidence levels of each target type for all bounding boxes in each set to obtain the weighted confidence level for each set of paired bounding boxes.

[0483] In some embodiments, taking a set of paired bounding boxes including two bounding boxes as an example, the weighted confidence ω of each set of paired bounding boxes can be expressed by Formula 5;

[0484]

[0485] Where λ is a weighting coefficient; λ is used to control the contribution of one bounding box in the paired bounding boxes to the confidence score; 1-λ is used to control the contribution of the other bounding box in the paired bounding boxes.

[0486] For any set of paired bounding boxes, the electronic device can determine the position of each bounding box within that set. Therefore, the electronic device can perform a weighted average of the positions of all bounding boxes in each set of paired bounding boxes to obtain the weighted overlap of each set.

[0487] In some embodiments, taking a set of paired bounding boxes including two bounding boxes as an example, the weighted overlap IoUφ of each set of paired bounding boxes can be expressed by Formula 6;

[0488]

[0489] in, It is a bounding box B1 in the paired bounding box and the first reference bounding box. Weighted intersection and union ratio (IoU) value; It is another bounding box B2 in the paired bounding box and the second reference bounding box. The weighted intersection-union ratio (IoU) is denoted by μ; μ is the weighting coefficient; μ is used to control the contribution of one bounding box B1 to the weighted intersection-union ratio (IoU); 1-μ controls the contribution of another bounding box B2 to the weighted intersection-union ratio (IoU).

[0490] The concept of intersection over union (IoU) is used in calculating the weighted overlap. IoU originates from sets in mathematics and is used to describe the relationship between two sets A and B. It is equal to the number of elements in the intersection of the two sets divided by the number of elements in their union. The specific formula is as follows:

[0491]

[0492] Electronic devices use the concept of Intersection over Union (IoU) to describe the degree of overlap or coincidence between two bounding boxes. Two bounding boxes can be viewed as sets of two pixels, and the IoU between these two bounding boxes is equal to the area of ​​the overlapping portion of the two bounding boxes divided by the combined area of ​​the two bounding boxes.

[0493] For example, such as Figure 26B As shown, the intersection region 1 in "Intersection" is the overlapping area of ​​the two bounding boxes, and the slash region 2 in "Union" is the combined area of ​​the two bounding boxes. Dividing the area of ​​intersection region 1 by the area of ​​slash region 2 gives the intersection-union ratio between the two bounding boxes.

[0494] S2602. Based on the weighted confidence and weighted overlap of all paired bounding boxes, determine the first detection result among multiple detection results. The first detection result includes one or more detection results, and the targets corresponding to each pair of paired bounding boxes in different detection results are different.

[0495] The electronic device filters out duplicate and low-quality paired bounding boxes from multiple detection results based on the weighted confidence and weighted overlap of all paired bounding boxes, selecting those with high confidence and low overlap to ensure that each target corresponds to a set of paired bounding boxes. Thus, the electronic device can determine the first detection result from multiple detection results.

[0496] S2603. Based on each detection result in the first detection result, determine the position of each pair of paired targets in the multiple targets as the position of a pair of paired bounding boxes in the corresponding detection result, and determine the confidence of the target category of each pair of paired targets as the target category with the highest confidence in a pair of paired bounding boxes in the corresponding detection result.

[0497] Based on the first detection result, the electronic device can determine each pair of targets among multiple targets, as well as the relevant feature information of each pair of targets.

[0498] For any given detection result, the position of each paired target is the position of a set of paired bounding boxes in that detection result; that is, the position of each target in each paired target is the position of the corresponding bounding box in that set of paired bounding boxes. The target category of each paired target is the target category with the highest confidence in that set of paired bounding boxes in that detection result; that is, the position of each target in each paired target is the target category with the highest confidence in the corresponding bounding box in that set of paired bounding boxes.

[0499] In summary, the electronic device can effectively process the paired bounding boxes output by the table visual inspection module 2412, filter out paired bounding boxes with high confidence and low overlap, and remove duplicate and low-quality paired bounding boxes, so that each target in the table can correspond to a set of paired bounding boxes.

[0500] Based on this, in S2602, the electronic device can first consider the confidence level and then the overlap level. Alternatively, the electronic device can first consider the overlap level and then the confidence level. Or, the electronic device can consider both the confidence level and the overlap level simultaneously. Thus, the electronic device can obtain the first detection result.

[0501] For example, regarding the screening principle that considers confidence level first and then overlap, please refer to [link / reference]. Figure 27A , Figure 27A This is a flowchart illustrating a table visual inspection method provided in an embodiment of this application. Figure 27A As shown, the table visual detection method in this application embodiment may include:

[0502] S2701. The electronic device calculates the weighted confidence ω for each pair of bounding boxes.

[0503] S2702. The electronic device calculates the weighted overlap IoUφ for each pair of bounding boxes.

[0504] S2703. The electronic device sorts all paired bounding boxes according to the weighted confidence ω of each pair of paired bounding boxes.

[0505] The sorting of all paired bounding boxes can be in descending order of weighted confidence or in ascending order of weighted confidence; this embodiment does not limit the sorting.

[0506] S2704. The electronic device performs NMS operation, which can filter out paired bounding boxes with high confidence and low overlap, and obtain the first detection result.

[0507] Below, in conjunction with Figure 27B This section details the specific implementation process of S2703 and S2704 by the electronic device.

[0508] Please see Figure 27B , Figure 27BThis is a flowchart illustrating a table visual inspection method provided in an embodiment of this application. Figure 27B As shown, the table visual detection method in this application embodiment may include:

[0509] S2801. Sort all paired bounding boxes in descending order of weighted confidence.

[0510] Electronic devices can be based on weighted confidence levels All paired bounding boxes will be arranged in descending order of size. The bounding boxes are sorted. The weighted confidence scores of the sorted paired bounding boxes satisfy the following condition:

[0511]

[0512] S2802. Traverse all paired bounding boxes after sorting, except for the first group of paired bounding boxes. If the difference between the weighted overlap between a group of paired bounding boxes in other groups and the first group of paired bounding boxes is greater than or equal to a first threshold, delete a group of paired bounding boxes. If the difference between the weighted overlap between a group of paired bounding boxes in other groups and the first group of paired bounding boxes is less than the first threshold, delete the first group of paired bounding boxes and update a group of paired bounding boxes to the first group of paired bounding boxes to obtain the first detection result.

[0513] In all sorted paired bounding boxes, the electronic device compares whether the difference between the weighted overlap of the second set of paired bounding boxes and the first set of paired bounding boxes is less than a first threshold.

[0514] If the difference between the weighted overlap of the second set of paired bounding boxes and the first set of paired bounding boxes is greater than or equal to a first threshold, the second set of paired bounding boxes is deleted. If the difference between the weighted overlap of the second set of paired bounding boxes and the first set of paired bounding boxes is less than the first threshold, the first set of paired bounding boxes is deleted, and the second set of paired bounding boxes is updated to match the first set of paired bounding boxes.

[0515] Next, the electronic device compares whether the difference between the weighted overlap of the third set of paired bounding boxes and the first set of paired bounding boxes is less than a first threshold. The comparison process of the weighted overlap between the third set of paired bounding boxes and the first set of paired bounding boxes can be found in the description of the weighted overlap between the second set of paired bounding boxes and the first set of paired bounding boxes, and will not be repeated here.

[0516] By analogy, the electronic device can iterate through and compare the last pair of paired bounding boxes in all sorted pairs. Thus, the electronic device can determine the first detection result as the final selected paired bounding boxes.

[0517] In one specific embodiment, NMS operation may include the following process:

[0518] An electronic device can initialize a result list B of a control, which is used to store the final filtered pair bounding boxes.

[0519] Next, the electronic device sorts all the paired bounding boxes. Perform iterations:

[0520] If result list B is empty, the electronic device will pair the bounding boxes. Add it to result list B.

[0521] If the result list B is not empty, then for each pair of bounding boxes... Electronic devices can calculate bounding boxes and bounding box The weighted IoU of the bounding boxes in the results list B with the weighted IoU of the filtered bounding boxes, and whether the weighted IoU is less than the first threshold, can be expressed by the following expression:

[0522]

[0523] if The electronic device will then set the pairing bounding box Add it to result list B.

[0524] Among them, IoU thresh This is the first threshold.

[0525] In summary, electronic devices can filter out paired bounding boxes with high confidence and low overlap, remove duplicate and low-quality paired bounding boxes, ensure that paired bounding boxes in the first detection result do not overlap excessively in the same target category, and have the highest weighted confidence, so that each target in the table can correspond to a set of paired bounding boxes.

[0526] Based on the exemplary information processing method described in the above embodiments, during the use of any application on an electronic device, the electronic device may display a data entry interface for that application. This data entry interface is the first interface mentioned throughout the text. This data entry interface displays information awaiting user input; that is, it is the interface in the application where user information needs to be entered. The data entry interface may include multiple fill-in controls and corresponding name controls, meaning each name control corresponds one-to-one with a fill-in control. The fill-in control displays the user information entered or selected by the user. The name control is the table label mentioned earlier. The name control displays the name of the information to which the fill-in control belongs. Subsequently, in response to the user's input operation on any fill-in control in the data entry interface, the electronic device displays the user's input. In other words, the user needs to sequentially fill in the user information in the data entry interface to complete the information entry.

[0527] For example, the aforementioned application can be an application installed on an electronic device (such as a shopping app), a web-based application, or a mini-program within an application, etc., without specific limitations. The web-based application can be an application found in a browser. The mini-program within an application can be an application that can be used without downloading or installing. The information name associated with the aforementioned input control can include at least one of the following: username, name, login password, gender, ID type, contact information, and shipping address.

[0528] However, if users need to fill in the same information on different interfaces (such as the account registration interface of different applications), that is, they need to enter the same user information multiple times, the information filling process may be cumbersome, waste a lot of time, and result in a poor user experience.

[0529] Therefore, to simplify the information entry process and improve efficiency, this application provides an information entry method. In this method, an electronic device displays an entry interface, which includes multiple fill-in controls and corresponding name controls. Then, in response to a user's triggering operation on any fill-in control in the entry interface, the electronic device obtains the label name of the name control corresponding to the triggered fill-in control. Next, the electronic device selects candidate user information corresponding to the label name of the name control from a user information database. This user information database includes multiple user records collected by the electronic device from multiple applications. Finally, the electronic device displays the candidate user information on the entry interface.

[0530] In this embodiment, after detecting a user's trigger operation on any fill control in the input interface, the electronic device can select the corresponding candidate user information from the user information database based on the label name of the name control corresponding to the triggered fill control, and display it on the input interface. This enables intelligent information filling; users only need to select the corresponding user information according to their actual needs, eliminating the need for manual entry, simplifying the information filling process, improving efficiency, and enhancing the user experience. Furthermore, since the candidate user information is collected by the electronic device from multiple applications, meaning it can select from all previously collected user information and is not limited to information collected by the application to which the input interface belongs, comprehensive display of user information is achieved. This reduces the likelihood of users repeatedly entering the same user information in different applications, providing convenience for users to quickly enter information.

[0531] For example, the aforementioned electronic devices may be mobile phones, tablets, desktops, laptops, handheld computers, notebook computers, ultra-mobile personal computers (UMPCs), netbooks, as well as cellular phones, personal digital assistants (PDAs), augmented reality (AR) / virtual reality (VR) devices, and other devices capable of installing and using applications that require user information. This application does not impose any special limitations on the specific form of the electronic device.

[0532] Figure 28 This is a schematic diagram of the hardware structure of the electronic device 2800 provided in the embodiments of this application, as shown below. Figure 28 As shown, the electronic device 2800 may include a processor 2810, an external memory interface 2820, an internal memory 2821, a universal serial bus (USB) interface 2830, a charging management module 2840, a power management module 2841, a battery 2842, antenna 1, antenna 2, a mobile communication module 2850, a wireless communication module 2860, an audio module 2870, a speaker 2870A, a receiver 2870B, a microphone 2870C, a headphone jack 2870D, a sensor 2880, a button 2890, a motor 2891, an indicator 2892, a camera 2893, a display screen 2894, and a subscriber identification module (SIM) card interface 2895, etc.

[0533] It is understood that the structures illustrated in the embodiments of this application do not constitute a specific limitation on the electronic device 2800. In other embodiments of this application, the electronic device 2800 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0534] Processor 2810 may include one or more processing units, such as an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural network processing unit (NPU). These different processing units may be independent devices or integrated into one or more processors.

[0535] In some embodiments, the electronic device 2800 may use the processor 2810 to perform the information entry method provided in this application.

[0536] The wireless communication function of electronic device 2800 can be implemented through antenna 1, antenna 2, mobile communication module 2850, wireless communication module 2860, modem processor and baseband processor, etc.

[0537] Electronic device 2800 implements display functions through a GPU, a display screen 2894, and an application processor. The GPU is a microprocessor for image processing, connecting the display screen 2894 and the application processor. The GPU performs mathematical and geometric calculations for graphics rendering. Processor 2810 may include one or more GPUs, which execute program instructions to generate or modify display information.

[0538] The display screen 2894 is used to display images, videos, etc. In some embodiments, the electronic device 2800 can display a data entry interface and candidate user information corresponding to any fill-in control on the data entry interface through the display screen 2894.

[0539] The electronic device 2800 can perform shooting functions through an ISP, camera 2893, video codec, GPU, display 2894, and application processor.

[0540] Electronic device 2800 can implement audio functions, such as music playback and recording, through audio module 2870, speaker 2870A, receiver 2870B, microphone 2870C, headphone jack 2870D, and application processor. In some embodiments, electronic device 2800 can acquire voice commands input by the user through microphone 2870C. Then, electronic device 2800 can determine the information input mode of the interface to be entered based on the voice command. This information input mode includes a one-click input mode and a separate input mode. The one-click input mode indicates that electronic device 2800 displays all candidate user information corresponding to the fill-in controls at once. The separate input mode indicates that electronic device 2800 displays candidate user information corresponding to the fill-in control triggered by the user.

[0541] Buttons 2890 may include a power button, volume buttons, etc. Buttons 2890 may be mechanical buttons or touch buttons. Electronic device 2800 can receive button input and generate key signal inputs related to user settings and function control of the electronic device. Motor 2891 can generate vibration alerts. Motor 2891 can be used for incoming call vibration alerts or for touch vibration feedback. Indicator 2892 may be an indicator light, used to indicate charging status, battery level changes, or messages, missed calls, notifications, etc. SIM card interface 2895 is used to connect a SIM card. The SIM card can be inserted into or removed from the SIM card interface 2895 to achieve contact and separation with the electronic device.

[0542] For example, the software system of the aforementioned electronic device 2800 can adopt a layered architecture, event-driven architecture, microkernel architecture, microservice architecture, or cloud architecture. This application embodiment uses a layered Android system as an example to illustrate the software structure of the electronic device 2800.

[0543] Figure 29 This is a software structure block diagram of an electronic device 2800 according to an embodiment of this application.

[0544] A layered architecture divides software into several layers, each with a clear role and function. Layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into four layers, from top to bottom: the application layer (application layer), the application framework layer (framework layer), the Android runtime and system libraries, and the kernel layer (or driver layer). The application layer can include a series of application packages.

[0545] like Figure 29As shown, the application package may include SMS, gallery, notes, form filling assistant, information management module, function settings module, etc.

[0546] The form-filling assistant described above is used to determine the name control corresponding to the triggered fill-in control in the input interface. In some embodiments, in response to a user's triggering operation on any fill-in control in the input interface, the form-filling assistant determines the name control corresponding to the triggered fill-in control. The input interface includes multiple fill-in controls and their corresponding name controls. That is, there is a one-to-one correspondence between the fill-in control and the name control.

[0547] The aforementioned information management module is used to receive user-defined operations on user information in the user information database. These operations include information query operations and / or information adjustment operations. The information query operation causes the electronic device to display the user information the user wants to query. The information adjustment operation causes the electronic device to adjust the user information in the user information database. For example, the information adjustment operation could be a deletion operation on any user's information in the user information database, a modification operation on any user's information in the user information database, or an addition operation on information in the user information database, etc., and is not specifically limited.

[0548] The aforementioned function setting module is used to set the collection source of user information in the user information database (i.e., the preset collection application). In some embodiments, if the user turns on the switch control with automatic collection function in the function setting interface (or information collection setting interface), it indicates that the electronic device is allowed to collect user information saved by all applications, and the function setting module can collect user information saved by the user in any application on the electronic device. In other embodiments, if the user does not turn on the switch control with automatic collection function in the function setting interface, and the user turns on the switch control of any application, the function setting module can collect user information saved by the application corresponding to the turned-on control.

[0549] The application framework layer provides application programming interfaces (APIs) and a programming framework for applications in the application layer. The application framework layer includes some predefined functions.

[0550] like Figure 29 As shown, the application framework layer may include a window manager (Window Manager Service, WMS), a content provider, a view system, an image memory module, etc.

[0551] The window manager is used to manage windowed applications. It can retrieve screen size, determine the presence of a status bar, lock the screen, and capture screenshots, among other things.

[0552] Content providers store and retrieve data, making that data accessible to applications. This data may include videos, images, audio, made and received phone calls, browsing history and bookmarks, phone books, etc.

[0553] A view system includes visual controls, such as controls for displaying text and controls for displaying images. View systems can be used to build applications. A display interface can consist of one or more views. For example, a display interface including a text notification icon could include views for displaying text and views for displaying images.

[0554] The profile memory module is used to determine at least one candidate user information that matches a target tag name. This target tag name is obtained by standardizing the tag name of the name control corresponding to the triggered fill-in control. In some embodiments, upon receiving an information query instruction from the form-filling assistant, the profile memory module can standardize the tag name of the name control corresponding to the triggered fill-in control to obtain the target tag name. Then, the profile memory module can select at least one user information that matches the target tag name from the user information database and display this at least one user information as at least one candidate user information in the input interface for the user to select.

[0555] Optionally, after obtaining at least one candidate user information, the profile memory module can sort the candidate user information according to a preset display rule to obtain an information candidate list. This preset display rule may include at least one of the following: chronological order of information update time, descending order of information entry count, and association with historical user information. Information update time refers to the time when the candidate user information was added to the user information database. Information entry count refers to the number of times the candidate user information was matched, i.e., the number of times the candidate user information was selected by the user. Historical user information refers to user information already entered in the input interface. Afterwards, the profile memory module can send this information candidate list to the form-filling assistant so that the electronic device can display the candidate user information in the order listed.

[0556] The Android runtime consists of the core libraries and the virtual machine. The Android runtime is responsible for scheduling and managing the Android system. The core libraries comprise two parts: one part contains the functionalities that the Java language needs to call, and the other part contains the core Android libraries.

[0557] The application layer and application framework layer run in a virtual machine. The virtual machine executes the Java files of the application layer and application framework layer as binary files. The virtual machine is used to perform functions such as object lifecycle management, stack management, thread management, security and exception management, and garbage collection.

[0558] System libraries can include multiple functional modules. For example: surface manager, 3D graphics processing library (e.g., OpenGL ES), 2D graphics engine (e.g., SGL), media libraries, etc.

[0559] The Surface Manager manages the display subsystem and provides fusion of 2D and 3D layers for multiple applications. The 3D graphics processing library implements 3D graphics drawing, image rendering, compositing, and layer processing. The 2D graphics engine is the drawing engine for 2D graphics. The media library supports playback and recording of various common audio and video formats, as well as still image files. The media library supports multiple audio and video encoding formats, such as MPEG4, H.264, MP3, AAC, AMR, JPG, and PNG.

[0560] The kernel layer is the layer between hardware and software. The kernel layer includes at least display drivers, camera drivers, audio drivers, and sensor drivers.

[0561] For example, the software architecture of the aforementioned electronic device 2800 may further include a hardware abstraction layer. This hardware abstraction layer is a wrapper around Linux kernel drivers, providing interfaces to higher-level systems. It hides the hardware interface details of a specific platform, providing a virtual hardware platform for the operating system. In this embodiment, the hardware abstraction layer may include modules such as a camera HAL, audio HAL, GPS HAL, and Wi-Fi HAL.

[0562] Understandable, Figure 29 The layers in the illustrated structure and the components contained in each layer do not constitute a specific limitation on the electronic device 2800, i.e., the mobile phone. In other embodiments of this application, the structure may include more or fewer layers than illustrated, and each layer may include more or fewer components; this application does not impose any limitations.

[0563] Below, based on the exemplary information processing method of the above embodiments, in the automatic form filling process or automatic form filling process, the content to be filled can be obtained from the following user information database and displayed on the input interface of the electronic device. That is to say, the user information in the user information database can serve as the source of the content to be filled during the automatic filling process.

[0564] Next, the specific implementation of the aforementioned process will be described in detail. For example, this application provides an information entry method. This method can enter user information from a user information database into the entry interface. The following description uses a mobile phone as an example to illustrate the method of this application embodiment. Specifically, as... Figure 30 As shown, the information entry method may include S3001 to S3014.

[0565] S3001, the form filling assistant in the mobile phone receives the user's trigger operation on any fill control in the input interface.

[0566] The aforementioned input interface refers to the interface in the application where user information needs to be entered. This input interface may include multiple fill-in controls and corresponding name controls, meaning there is a one-to-one correspondence between the name control and the fill-in controls. The fill-in controls are used to display the user information entered or selected by the user. The name controls are used to display the name of the information to which the fill-in control belongs. For example, such as... Figure 31 As shown, the screen displayed on the phone is the interface for adding a shipping address (i.e., ...). Figure 31 (a) of the interface, the add shipping address interface may include a "Recipient" control 3111, a corresponding input control 3121, a "Mobile Number" control 3112, and a corresponding input control 3122. The "Recipient" control 3111 and the "Mobile Number" control 3112 are both name controls.

[0567] For example, the aforementioned triggering operation refers to the operation performed when a user touches the mobile phone screen with their finger. This triggering operation can be a single click on a fill control, a double click on a fill control, etc., and is not specifically limited.

[0568] It should be noted that the application to which the above-mentioned input interface belongs can be an application installed on the mobile phone, a web application, or a mini-program within any application on the mobile phone, etc., and there are no specific restrictions.

[0569] S3002, in response to the user's trigger operation on any fill control in the input interface, the form filling assistant sends an information query instruction to the portrait memory module in the mobile phone.

[0570] Specifically, upon receiving a user's trigger action on any fill-in control in the data entry interface, the form-filling assistant can send an information query instruction to the profile memory module based on the triggered fill-in control. This information query instruction carries the attribute information of the name control corresponding to the triggered fill-in control.

[0571] In some embodiments, the form-filling assistant can determine the triggered fill-in control based on the user's trigger location and the display location of the fill-in control on the input screen. The user's trigger location is determined based on the touch point generated when the user's finger touches the phone screen. Then, the form-filling assistant can determine the name control corresponding to the triggered fill-in control based on the positional mapping relationship between the fill-in controls and name controls on the input screen, and obtain the attribute information of that name control.

[0572] In this embodiment, the attribute information of the name control may include the label name of the name control, the application of the interface to which the name control belongs (or the application to which the interface belongs), and the title name of the interface to which the name control belongs (or the interface title name). The label name of the name control refers to the name of the information displayed on the interface to be entered. For example, if the name control is a "Recipient" control, then the label name is "Recipient". The application to which the interface belongs refers to the application to which the interface to be entered belongs. The interface title name is used to characterize the purpose of filling in user information on the interface to be entered. For example, the interface title name may be interface information displayed at the top of the interface to be entered. For example, using... Figure 31 Taking the input interface shown in (a) as an example, the title of this interface can be "Add shipping address".

[0573] S3003, when the portrait memory module receives the information query instruction sent by the form filling assistant, it detects the label name in the attribute information of the name control corresponding to the triggered filling control according to the information query instruction, and obtains the detection result.

[0574] Specifically, upon receiving an information query instruction from the form-filling assistant, the profile memory module can detect the tag name of the triggered form-filling control based on the attribute information of the name control carried in the query instruction, to determine whether the tag name can be standardized. In other words, the detection result indicates whether the tag name of the name control can be standardized. Standardization refers to converting the tag name into a tag name stored in the user information database, so that the mobile phone can determine candidate user information based on the tag name, thereby improving the accuracy of information recommendations.

[0575] In some embodiments, the mobile phone can detect the aforementioned tag name by determining whether the attribute information of the name control meets preset conditions. That is, the preset conditions are used to assess whether the tag name can be standardized. Specifically, if the attribute information of the name control meets the preset conditions, it indicates that the tag name can be standardized, and the profile memory module can execute S3004 to standardize the tag name. If the attribute information of the name control does not meet the preset conditions, it indicates that the tag name cannot be standardized. Therefore, the profile memory module can execute S3011 to send the tag name to the cloud service, providing a basis for subsequent standardization processing of the tag name.

[0576] The aforementioned preset conditions may include the name control's attribute information satisfying any mapping relationship in the preset mapping rules, and / or, the similarity between a tag name in the user information database and the tag name of the name control being greater than a preset similarity. It should be noted that the specific judgment process for these preset conditions will be explained in detail below.

[0577] S3004, if the detection result indicates that the label name can be standardized, the image memory module standardizes the label name to obtain the target label name.

[0578] Specifically, after the above detection results indicate that the label name of the name control can be standardized, the profile memory module standardizes the label name of the name control to obtain the target label name of the name control. This target label name can be any label name existing in the user information database. For example, if the label name is "recipient," the profile memory module can standardize the recipient to "name," meaning the target label name is "name." Similarly, if the label name is "phone number," the profile memory module can standardize the phone number to "phone number," meaning the target label name is "phone number."

[0579] The standardization process for the aforementioned label names will be explained in detail below.

[0580] S3005, the profile memory module determines whether there is user information in the user information database that matches the target tag name.

[0581] In some embodiments, after obtaining the target tag name, the profile memory module can determine whether there is user information in the user information database that matches the target tag name. If there is user information in the user information database that matches the target tag name, it means that the user has previously filled in user information for the target tag name. Therefore, the profile memory module can execute S3007. If there is no user information in the user information database that matches the target tag name, it means that the user has not previously filled in user information for the target tag name. Therefore, the profile memory module can execute S3006.

[0582] In one example, the aforementioned user information database may include multiple user information entries for the account holder and a tag name for each entry. For instance, if the database includes the account holder's gender as "female," the tag name would be gender, and the user information entry would be "female." In another example, the user information database may also include multiple user information entries for at least one associated user and a tag name for each entry. An associated user refers to a user who has a connection with the account holder. For example, an associated user could be a relative (such as a child or parent), a friend, or a customer of the account holder, etc., without specific limitations.

[0583] In one implementation, the user information in the aforementioned user information database can be pre-filled by the user in the information editing interface of the settings application. Specifically, upon detecting a user's information entry event, the phone can store the user-entered information as new user information in the user information database. This information entry event can be the user's entry operation in the information editing interface. This provides a basis for subsequent retrieval of user information.

[0584] In some embodiments, in response to a user's information entry operation on the information editing interface, the mobile phone can obtain the newly entered user information. The mobile phone can then store the newly entered user information in a user database for later retrieval. For example, such as... Figure 32 As shown, the mobile phone is displaying the personal information interface shown in interface (a). If the user clicks the add control 3201 in the personal information interface, the mobile phone can display... Figure 32 The information editing interface shown in (b) is shown below. Subsequently, in response to the user's input of the username information control 3202 in the information editing interface, the mobile phone can obtain and display the input content of the information control 3202, "Zhao Yiyi".

[0585] In another implementation, the user information in the aforementioned user information database can also be extracted by the mobile phone from a preset collection application. This preset collection application is an application with information collection capabilities. That is, before extracting user information stored by any application on the phone, the phone can determine whether the application has information collection capabilities. Only if the application has information collection capabilities can the phone extract information from that application to obtain the user information stored therein.

[0586] In some embodiments, the aforementioned preset collection application may be a mobile phone pre-configured, that is, a default setting on the mobile phone. For example, please see Figure 33 The phone is displaying Figure 33 Interface (a) shows the information collection settings interface, which includes a switch control 3301A for automatic collection, a switch control 3302 for the SMS application, a switch control 3303 for the Gallery application, and a switch control 3304 for the Notes application. It can be seen that the automatic collection switch control 3301, the SMS application switch control 3302, and the Gallery application switch control 3303 are all in the off state, while the Notes application switch control 3304 is in the on state. This means that the phone only defaults to the Notes application as the pre-set collection application.

[0587] In other embodiments, the aforementioned preset data collection applications can be those pre-configured by the user. That is, the mobile phone can identify applications whose information collection functions are enabled by the user as preset data collection applications. For example, such as... Figure 33 As shown, if the user turns on the automatic collection function switch 3301A in the information collection settings interface, it means the user allows the phone to extract user information from any application on the phone. Therefore, the phone can use all installed applications as preset collection applications, meaning it can extract user information saved by each application. The switch 3301A indicates that the automatic collection function is off. Afterwards, the phone can display... Figure 33 The target setting interface shown in (b) includes a switch control 3301B corresponding to the automatic collection function. The switch control 3301B is used to indicate that the automatic collection function is in the enabled state.

[0588] If the user does not enable the automatic collection function switch 3301A in the above information collection settings interface, but clicks the switch 3302 belonging to the SMS application in the above information collection settings interface 3300, it means that the user allows the mobile phone to extract user information from the SMS application. Therefore, the mobile phone can use the SMS application as the default collection application, that is, the mobile phone can extract the user information saved by the SMS application.

[0589] It is understandable that if the default collection app is an SMS app, the phone can extract user information from SMS messages saved by the SMS app. These SMS messages can be received by the phone or edited by the user. If the default collection app is a gallery app, the phone can extract user information from images saved by the gallery app. These images can be taken by the user or downloaded and cached from any app on the phone (such as a social media app). If the default collection app is a notes app, the phone can extract user information from notes saved by the notes app. These notes are edited by the user. If the default collection app is a third-party app (such as a social media app), the phone can extract user information from the information entry interface of that third-party app. This information entry interface can be any interface within the app that allows user information to be entered. This interface can be an account registration interface, a page for adding traveler information, a page for adding a shipping address, etc., and is not specifically limited.

[0590] In some embodiments, the process of the mobile phone extracting user information from different applications can be automated. For example, taking a gallery application as a preset collection application, when a new image is detected in the gallery application, the mobile phone can extract information from the new image to obtain new user information and add it to the user information database. This achieves automatic information extraction without requiring user intervention, simplifying the process and facilitating the subsequent display of candidate user information.

[0591] In some embodiments, the process of the mobile phone extracting user information from different applications can also be achieved by the mobile phone extracting information based on the user's information extraction operation. Specifically, in response to the user's information extraction operation on the display interface of a preset collection application, the mobile phone can extract information from the display interface to obtain new user information. The display interface may include at least one piece of user information. The information extraction operation can be a user's click operation on an information extraction control, or a voice command from the user regarding information extraction, etc., and is not specifically limited. For example, the voice command could be to extract user information. Afterwards, the mobile phone can add the new user information to the user information database. This allows for accurate extraction of user information, confirms with the user before extraction, reduces the risk of information security issues due to excessive extraction, ensures the security of the extraction environment, and provides convenient conditions for subsequently displaying candidate user information.

[0592] For example, such as Figure 34 As shown, taking the aforementioned preset collection application as an example, the phone is displaying... Figure 34 The image display interface shown in (a) includes an ID card image 3410. Subsequently, in response to the user's wake-up operation for the voice assistant, the phone can display as shown... Figure 34 The interface shown in (b) displays a "Save to Personal Information" control, a "Copy ID Number" control, a "Copy Address" control, and a voice assistant prompt card 3420 overlaid on the image display interface. The wake-up operation can be initiated by the user uttering a wake-up phrase for the voice assistant, such as "Hello Youyou." The "Save to Personal Information" control is used to extract user information from the ID card image 3410 and save it to the user information database. It can be understood that the types of controls overlaid on the phone are determined by the content displayed on the phone when the voice assistant is activated.

[0593] Afterwards, with the voice assistant activated, the user can input dialogue. This dialogue can be in voice format, such as inputting a command like "Save user information." Alternatively, it can be in text format, such as inputting a command like "Save user information" into the voice assistant's prompt card 3420. Upon detecting the user's input, the phone can perform the corresponding operation, namely saving the user information from the ID card image 3410 to the user information database. Alternatively, in response to the user's click on the "Save to Personal Information" control, the phone can save the user information from the ID card image 3410 to the user information database.

[0594] In one implementation, before extracting information from the aforementioned display interface, the mobile phone needs to authenticate the user. That is, the phone can only extract information from the display interface if the authentication is successful (i.e., the user is the owner of the device). Specifically, the phone can authenticate the user and obtain an authentication result. This result indicates whether the user is the owner of the device. If the authentication result indicates the user is the owner, the phone can extract information from the display interface to obtain new user information. If the authentication result indicates the user is not the owner, i.e., the user is not the owner, the phone does not perform the information extraction operation, i.e., it does not extract information from the display interface. This reduces the possibility of non-owner users extracting user information, improves the security of information extraction, and ultimately enhances the user experience.

[0595] The mobile phone can authenticate users through facial recognition, fingerprint recognition, or password input, among other methods; the specific method is not limited. For example, please see... Figure 34If the "Save to Personal Information" control in interface (b) is clicked by the user, the phone can display the following: Figure 34 The security verification interface shown in (c) is used to authenticate the user's identity. This security verification interface is a fingerprint recognition interface. After successful authentication, the mobile phone can save the user information from the ID card image 3410 to the user information database.

[0596] In another implementation, upon obtaining new user information, the mobile phone can retrieve multiple user information entries corresponding to the user to whom the new user information belongs from the user information database. Then, the mobile phone can determine whether user information with that tag name exists among the multiple user information entries based on the tag name corresponding to the new user information. If the tag name does not exist among the multiple user information entries, it means that the user information database does not store user information with that tag name. Therefore, the mobile phone can save the new user information to the user information database and display the personal information interface. This personal information interface includes the new user information. This allows for accurate saving of user information, reducing the occurrence of missed or incorrectly saved user information, and providing convenient conditions for timely retrieval of user information later.

[0597] For example, such as Figure 34 As shown, the phone is displaying... Figure 34 In interface (c), if the user's identity authentication is successful but the user information in ID card image 3410 is not stored in the user information database, the mobile phone can display the personal information interface of the user to whom ID card image 3410 belongs, that is, display... Figure 34 The (d) interface is shown in the image. This personal information interface includes the newly added ID card (i.e., ID card 3411). Optionally, if the user clicks on ID card 3411, it indicates that the user wants to view an image of the ID card. The phone can then display an image of ID card 3410 for the user to view and verify the information.

[0598] If user information with the tag name exists among the multiple user information entries, it means that user information with the tag name has already been stored in the user information database. Therefore, in order to improve the accuracy of information storage in the user information database, the mobile phone can use the user information with the tag name among the multiple user information entries as the original user information and determine whether the original user information is the same as the new user information. If the original user information is the same as the new user information, it means that the new user information has been saved to the user information database. Therefore, the mobile phone does not need to save the new user information again, so as to avoid wasting too much storage space due to duplicate saving of user information and improve the utilization rate of storage resources.

[0599] If the original user information differs from the new user information, it indicates an error in either. Therefore, the phone can display an information update interface for the user to determine whether an update is necessary. This update interface may include at least one of the following: original user information, new user information, a prompt about "updating user information," a back control, and an update control. If the user clicks the back control, it means they do not wish to update the user information, and the phone can continue to save the original information. If the user clicks the update control, it means they want to update the user information, and the phone can replace the original information with the new information in the user database. This ensures accurate user information updates, reduces errors, and improves the user experience.

[0600] For example, such as Figure 34 As shown, the phone is displaying... Figure 34 In interface (c), if the user's identity authentication is successful and the user's ID number is stored in the user information database, but the ID number stored in the user information database is different from the ID number in ID card image 3410, the mobile phone can display... Figure 34 The information update interface shown in (e) includes a prompt (i.e., "Update card information?"), the current number (i.e., the original user information), the new number (i.e., the new user information), a "Save Current Information" control (i.e., a return control), and an "Update" control. If the user clicks the "Save Current Information" control, the phone can continue to save the original ID number in the user information database. If the user clicks the "Update" control, the phone can replace the ID number in ID image 3410 with the original ID number and save it to the user information database.

[0601] S3006, if there is no user information in the user information database that matches the target tag name, the profile memory module sends a matching result instruction to the form filling assistant.

[0602] The matching result indicates that there is no user information in the user information database that matches the target tag name, meaning that the profile memory module did not find any user information that matches the tag name.

[0603] In some embodiments, when the form-filling assistant receives a matching result instruction sent by the profile memory module, the mobile phone can continue to display the input interface, that is, it does not display the recommended user information (i.e., candidate user information) on the input interface, so that the user can fill in the information for the aforementioned triggered input control.

[0604] S3007, if user information matching the target tag name exists in the user information database, the profile memory module will identify the user information matching the target tag name as candidate user information.

[0605] Specifically, once user information matching the target tag name exists in the user information database, the profile memory module can display the user information matching the target tag name as candidate user information in the input interface.

[0606] In some embodiments, the profile memory module can determine the number of candidate user information entries. If the number of candidate user information entries is 1, the profile memory module can directly execute S3008. If the number of candidate user information entries is not 1, that is, if there are multiple user information entries in the user information database that match the target tag name, the profile memory module can sort the multiple candidate user information entries according to a preset display rule to obtain a candidate information list. This candidate information list is used to represent the arrangement order of the multiple candidate user information entries.

[0607] The aforementioned preset display rules may include at least one of the following: the order of information update time, the decreasing order of information entry count, and association with historical user information. Information update time refers to the time when candidate user information is added to the user information database. Information entry count refers to the number of times a candidate user is selected by the user. Historical user information refers to user information already entered in the input interface.

[0608] In one implementation, the user profile memory module can prioritize placing candidate user information closer to the current information entry time at the top of the candidate information list, and placing candidate user information further away from the current information entry time at the bottom. Specifically, the user profile memory module can obtain the information update time of each candidate user. Then, the user profile memory module can sort the multiple candidate user information according to the order of their information update times to obtain the candidate information list. For example, the candidate user information includes user A, user B, and user C, and user A's information update time is T1, user B's is T2, and user C's is T3. Here, T1, T2, and T3 include, but are not limited to, year, month, date, clock, and minute. Therefore, the user profile memory module can place user C first in the candidate information list, user B second, and user A third, meaning the candidate information list consists of user C, user B, and user A.

[0609] In another implementation, the user profile memory module can prioritize placing candidate user information with higher query counts at the top of the candidate information list and lower query counts at the bottom. Specifically, the module obtains the query count for each candidate user. Then, it sorts the candidate user information in descending order of query count to obtain the candidate information list. For example, if the candidate user information is user A, user B, and user C, and user A has 10 queries, user B has 5 queries, and user C has 1 query, the module can place user A first, user B second, and user C third in the candidate information list. In other words, the candidate information list consists of users A, B, and C.

[0610] In another implementation, the user profile memory module can prioritize placing candidate user information associated with historical user information at the top of the candidate information list, and place candidate user information not associated with historical user information at the bottom. Specifically, the user profile memory module can retrieve the associated user information for each candidate user information in the user information database. Then, the module can sequentially determine whether the associated user information of historical user information and candidate user information is the same. If the associated user information of historical user information is the same as that of any candidate user information, the module can place that candidate user information at the top of the candidate information list. For example, the candidate user information includes user A's mobile phone number "189****4045" and user C's mobile phone number "189****4123", and the historical user information is user A. As can be seen, the profile memory module can identify the mobile phone number "189****4045" as a candidate user information associated with historical user information. Therefore, the profile memory module can place the mobile phone number "189****4045" in the first position of the information candidate list and the mobile phone number "189****4123" in the second position of the information candidate list. In other words, the information candidate list consists of user A and user C.

[0611] It should be noted that the aforementioned historical user information can be carried by the aforementioned information query instruction. That is, upon receiving a user's trigger action on any fill-in control in the data entry interface, the form-filling assistant can send an information query instruction carrying historical user information and the attribute information of the name control corresponding to the triggered fill-in control to the user profile memory module. This provides a foundation for the subsequent accurate display of recommended information. Afterwards, the user profile memory module can store this historical user information in the user information database, providing a basis for subsequent retrieval of user information.

[0612] In some cases, before storing historical user information in the user information database, the profile memory module can map the tag names associated with that historical user information to obtain mapped tag names. Then, the profile memory module can store these mapped tag names as both the tag names and the historical user information in the user information database. This ensures that all tag names in the user information database are standardized, meaning that user information of the same type has the same tag name, facilitating accurate retrieval of user information and improving retrieval efficiency.

[0613] It's understandable that historical user information may follow certain rules (e.g., phone numbers are 11 digits long, email addresses contain the "@" symbol). Therefore, the user profile memory module can map the historical user information to its corresponding tag name based on the historical user information and preset information rules, obtaining the mapped tag name. These preset information rules include, but are not limited to, at least one of the following: the historical user information contains the "@" character, the total number of characters in the historical user information is 11 digits, and the number of characters in the historical user information is within a preset range. For example, the historical user information is "189****4045," and the tag name for this historical user information is "Contact Information." Since the preset format for phone numbers is 11 digits, the user profile memory module can determine that the mapped tag name for this historical user information is a phone number; that is, the user profile memory module can standardize "Contact Information" to a phone number. Another example is the historical user information "Zhao Yiyi," and the tag name for this historical user information is "Sender." Since the number of characters in a name is generally 2 to 4, which is the preset range of 2 to 4, the profile memory module can determine that the mapping tag name to which the historical user information belongs is the name. In other words, the profile memory module can standardize the sender as the name.

[0614] S3008, the profile memory module sends candidate user information to the form filling assistant.

[0615] In some embodiments, after obtaining the aforementioned candidate user information, the profile memory module can send the candidate user information to the form-filling assistant. For example, if the number of candidate user information entries is 1, the profile memory module can directly send the candidate user information to the form-filling assistant. Alternatively, if the number of candidate user information entries is not 1, the profile memory module can send the aforementioned candidate information list to the form-filling assistant, that is, send the sorted candidate user information to the form-filling assistant.

[0616] S3009, when the form filling assistant receives the candidate user information sent by the profile memory module, it displays the candidate user information on the input interface.

[0617] In some embodiments, after receiving candidate user information from the profile memory module, the form-filling assistant can display that candidate user information on the input interface. For example, if the number of candidate user information entries is one, the form-filling assistant can directly display that entry. Alternatively, if the number of candidate user information entries is not one, the form-filling assistant can display the candidate user information sequentially on the input interface according to the order in the candidate information list. This allows users to easily select the appropriate user information, providing convenient conditions for quickly filling in information and thus improving the user experience.

[0618] In one implementation, the form-filling assistant can display candidate user information floating below the triggered form-filling control. For example, such as... Figure 31 As shown, the phone is displaying... Figure 31 In interface (a), if the input control 3121 corresponding to the "Recipient" control is triggered by the user, the mobile phone can display the candidate user information for the "Recipient" in a floating manner below the input control 3121. In other words, the mobile phone can display... Figure 31 The interface is shown in (b). The candidate user information for "Recipient" may include Zhao Yiyi and Zhao Sansan.

[0619] In another implementation, the form-filling assistant can display candidate user information as a dropdown list based on the triggered form-filling control. The dropdown list allows the user to select user information from the candidate list to fill in.

[0620] It's understandable that, to simplify the display of user information, if the number of candidate user information entries is not one, the form-filling assistant can only display a preset number of candidate user information entries that appear at the beginning of the candidate list. This preset number of entries can be set according to actual needs; for example, it can be one, three, etc., without specific limitations. For instance, if the candidate list includes 10 candidate user information entries, the form-filling assistant can retrieve the preset number (e.g., three) of candidate user information entries that appear at the beginning of the list and display that preset number. In other words, the phone can only display the first three candidate user information entries. This allows for accurate display of user information, reducing interface clutter caused by displaying too much information, improving interface cleanliness, and facilitating subsequent selection of user information by the user.

[0621] S3010, in response to the user's selection operation for any candidate user information, the form filling assistant displays the selected candidate user information in the triggered fill-in control.

[0622] In some embodiments, after candidate user information is displayed on the input interface, if any candidate user information is selected by the user, the form-filling assistant can display the selected candidate user information in the triggered fill-in control. For example, please refer to... Figure 31 The phone is displaying Figure 31 In the interface shown in (b), if the candidate user information "Zhao Yiyi" is clicked by the user, the mobile phone can directly display "Zhao Yiyi" in the input control 3121, that is, display Figure 31 The interface shown in (c) allows for intelligent filling of user information. Users only need to select and fill in the corresponding user information according to their actual needs, eliminating the need for manual entry. This simplifies the information filling process, improves the efficiency of information filling, and ultimately enhances the user experience.

[0623] In other embodiments, if the candidate user information displayed on the input interface does not meet the user's needs, the user may not select any candidate user information. The user can directly fill in the corresponding user information in the triggered fill control. That is, in response to the user's fill operation on the triggered fill control, the form filling assistant displays the filled user information in the triggered fill control.

[0624] It's understandable that when the triggered fill-in controls display user information—that is, when selected candidate user information or already filled-in user information—the form-filling assistant can send this user information to the profile memory module, allowing the module to store it in the user information database. This not only enables timely saving of user information and improves storage efficiency, but also allows the phone to easily identify the input content of blank fill-in controls on the current screen (i.e., the input screen), ensuring that the input content is associated with the user information—that is, belonging to the same user. This improves the efficiency of selecting candidate user information and ultimately enhances the user experience.

[0625] S3011, if the detection result indicates that the label name cannot be standardized, the profile memory module sends the label name to the name management module in the cloud service.

[0626] In some embodiments, after the above detection results indicate that the tag name cannot be standardized, the profile memory module can send the tag name to the name management module in the cloud service (or server) so that the tag name can be standardized and learned in the future, thereby improving the standardization efficiency of the attribute name.

[0627] The cloud service mentioned above may include a name management module, which is used to standardize the label names of the name controls.

[0628] In other embodiments, after the detection result indicates that the tag name cannot be standardized, the profile memory module can send the detection result to the form-filling assistant. Then, upon receiving the detection result from the profile memory module, the form-filling assistant continues to display the input interface, meaning it does not display any candidate user information on the input interface. Subsequently, in response to the user's information entry operation on the triggered fill-in control, the form-filling assistant can display the entered user information in the triggered fill-in control. It can be understood that this entered user information is obtained based on the user's information entry operation. In other words, if the tag name cannot be standardized, it means the phone cannot find the corresponding user information in the user information database, meaning the phone cannot display candidate user information on the input interface. Therefore, the phone can continue to display the input interface to remind the user to manually fill in the user information.

[0629] Afterwards, the form-filling assistant can send the entered user information as historical user information to the profile memory module. Upon receiving the historical user information from the form-filling assistant, the profile memory module maps the tag names of the historical user information according to the aforementioned preset information rules, obtaining mapped tag names. The profile memory module then stores these mapped tag names, along with the historical user information, in the user information database. This ensures that all tag names in the user information database are standardized, meaning that the tag names corresponding to the same type of user information are identical, facilitating accurate retrieval of user information and improving the efficiency of user information retrieval.

[0630] S3012, when the name management module receives the tag name sent by the profile memory module, it performs standardized learning on the tag name to obtain the target attribute name.

[0631] Specifically, after receiving the tag name from the profile memory module, the name management module can save the tag name. Then, at preset time intervals, the name management module can standardize the tag name to obtain the target tag name. This improves the accuracy of target tag name determination, thus providing a foundation for subsequent target tag name determination.

[0632] In some embodiments, operations and maintenance personnel can standardize the aforementioned label names to obtain target label names. This manual standardization improves the accuracy of target label name determination, thus providing a foundation for subsequent target label name determination.

[0633] In other embodiments, the name management module can standardize the aforementioned tag names according to a name adjustment model to obtain target tag names. Specifically, the name management module can input tag names into the name adjustment model to obtain target tag names. This name adjustment model is a supervised learning algorithm. It is trained based on a set of tag names, which includes multiple tag names and the actual tag names carried by each tag name. In other words, the name management module can input tag names into the name adjustment model to be trained to obtain predicted tag names. Then, the name management module can adjust the parameters of the name adjustment model based on preset tag names and actual tag names to obtain a trained name adjustment model. In this way, the accuracy of target tag name determination can be improved through machine learning, eliminating the need for manual standardization learning, thus improving the standardization efficiency of tag names and providing a foundation for subsequent determination of target tag names.

[0634] S3013, the name management module sends the mapping relationship between the target tag name and the tag name to the image memory module.

[0635] S3014, when the image memory module receives the mapping relationship between the target tag name and the tag name sent by the name management module, it adds the mapping relationship between the target tag name and the tag name to the preset mapping rules.

[0636] Specifically, after receiving the mapping relationship between target tag names and tag names sent by the name management module, the profile memory module can add this mapping relationship to the preset mapping rules to facilitate subsequent standardization of tag names. This preset mapping rule includes multiple mapping relationships between tag names and target tag names.

[0637] In one implementation, the profile memory module can display preset mapping rules through a mapping table. This mapping table represents the mapping relationship between multiple tag names and target tag names.

[0638] In some embodiments, the mapping table may include multiple target tag names and the tag name corresponding to each target tag name. That is, the target tag names in the above preset mapping rules may be determined based on the tag names.

[0639] For example, as shown in Table 1, the mapping table may include the tag name "Contact Person", the target tag name "Name" corresponding to the tag name "Contact Person", the tag name "Recipient", the target tag name "Name" corresponding to the tag name "Recipient", the tag name "Contact Person Number", the target tag name "Mobile Number" corresponding to the tag name "Contact Person Number", the tag name "Mobile Number", and the target tag name "Mobile Number" corresponding to the tag name "Mobile Number". In other words, both the tag name "Contact Person" and the tag name "Recipient" correspond to the target tag name "Name", and both the tag name "Contact Person Number" and the tag name "Mobile Number" correspond to the target tag name "Mobile Number".

[0640] Table 1

[0641] Label Name Target label name Contact Person Name recipient Name Contact number phone number Phone number phone number

[0642] The mapping table mentioned above can also include the interface title name and the application to which each target tag name belongs. In other words, the target tag name in the preset mapping rules can be determined based on the interface title name, the application to which the interface belongs, and the tag name. The interface title name is the interface information displayed at the top of the phone's screen. It can be understood that this interface title name is used to represent the purpose of entering user information on the phone's screen. This provides a foundation for accurately determining the target tag name subsequently.

[0643] For example, as shown in Table 2, the mapping table includes the target tag name "Mobile Number" corresponding to the application "Application 1", the title name "Add Shipping Information" and the tag name "Contact Information"; the target tag name "Email Number" corresponding to the application "Application 1", the title name "Apply for Invoice" and the tag name "Contact Information"; the target tag name "File Number" corresponding to the application "Application 2", the title name "Add Student Record" and the tag name "ID Number"; and the target tag name "ID Card Number" corresponding to the application "Application 3", the title name "Add Traveler Information" and the tag name "ID Card Number".

[0644] Table 2

[0645] Application to which the interface belongs Interface title name Label Name Target label name Application 1 Add shipping information Contact information phone number Application 1 Apply for making an invoice Contact information Email address Application 2 Add student profile ID number File number Application 3 Add traveler information ID number ID number

[0646] The above describes the specific implementation process of how a mobile phone displays corresponding candidate user information based on a triggered fill-in control, allowing the user to select. The following will combine the above... Figure 29 The structure shown and Figure 35 The standardization process for the label name shown is explained in detail.

[0647] S3501, the image memory module calls the preset mapping rules.

[0648] Specifically, after obtaining the attribute information of the aforementioned name control, the profile memory module can invoke the aforementioned preset mapping rules. It can be understood that the multiple mapping relationships included in these preset mapping rules can be obtained from the name management module in the cloud service. For example, the profile memory module can obtain new mapping relationships within a preset time period from the name management module at preset intervals. Alternatively, when the name management module generates new mapping relationships, it can directly send the new mapping relationships to the profile memory module.

[0649] S3502, the portrait memory module determines whether the attribute information of the name control satisfies any of the mapping relationships in the preset mapping rules.

[0650] In some embodiments, after invoking the aforementioned preset mapping rules, the profile memory module can determine whether the attribute information of the name control satisfies any mapping relationship in the preset mapping rules. If the attribute information of the name control satisfies any mapping relationship in the preset mapping rules, it indicates that the attribute information of the name control meets the aforementioned preset conditions, that is, the tag name can be standardized. Therefore, the profile memory module can execute S3503. If the attribute information of the name control does not satisfy any mapping relationship in the preset mapping rules, it indicates that the tag name may not be standardized. Therefore, in order to further determine whether the tag name can be standardized, the profile memory module can execute S3504.

[0651] In one implementation, if the mapping table representing the preset mapping rule only includes the tag name and the target tag name (meaning the target tag name can be directly determined based on the tag name), then the profile memory module only needs to determine if there is a tag name in the preset mapping rule that is the same as the tag name of the name control. If there is a tag name in the preset mapping rule that is the same as the tag name of the name control, the profile memory module can determine that the attribute information of the name control meets the preset conditions. If there is no tag name in the preset mapping rule that is the same as the tag name of the name control (meaning the tag name is different from all the tag names in the preset mapping rule), the profile memory module can determine that the attribute information of the name control does not meet the preset conditions.

[0652] For example, please refer to Table 1. If the label name of the aforementioned name control is "Recipient," and the mapping table shown in Table 1 also includes the label name "Recipient," it means that there exists a label name in the mapping table that is the same as the label name "Recipient" of the name control. Therefore, the profile memory module can determine that the attribute information of the name control meets the preset conditions, that is, the label name "Recipient" of the name control can be standardized. If the label name of the aforementioned name control is "Consignee," and the mapping table shown in Table 1 does not include "Consignee," it means that there does not exist a label name in the mapping table that is the same as the label name "Consignee" of the name control. Therefore, the profile memory module can determine that the attribute information of the name control does not meet the preset conditions, that is, the label name "Consignee" of the name control cannot be standardized.

[0653] In another implementation, if the mapping table representing the preset mapping rule includes the application to which the interface belongs, the interface title name, the tag name, and the target tag name—meaning the target tag name is determined based on the interface title name, the application to which the interface belongs, and the tag name—then the profile memory module needs to determine whether there exists a mapping relationship in the preset mapping rule that is identical to the interface application, interface title name, and tag name included in the attribute information. If there is a mapping relationship in the preset mapping rule that is identical to the interface application, interface title name, and tag name included in the attribute information, the profile memory module can determine that the name control's attribute information meets the preset conditions. If there is no mapping relationship in the preset mapping rule that is identical to the interface application, interface title name, and tag name included in the attribute information—meaning that any item in the attribute information is different from the mapping relationship in the preset mapping rule—the profile memory module can determine that the name control's attribute information does not meet the preset conditions.

[0654] For example, please refer to Table 2. If the application to which the interface belongs is Application 1, the interface title is "Add Item Sending Information," and the label name of the name control is "Contact Information," and the mapping table shown in Table 2 includes mapping relationships that simultaneously satisfy the application "Application 1," the interface title "Add Item Sending Information," and the label name of the name control "Contact Information," the profile memory module can determine that the attribute information of the name control meets the preset conditions, that is, the label name "Contact Information" of the name control can be standardized. If the application to which the interface belongs is Application 2, the interface title is "Add Item Sending Information," and the label name of the name control is "Contact Information," and the mapping table shown in Table 2 does not include mapping relationships that simultaneously satisfy the application "Application 2," the interface title "Add Item Sending Information," and the label name of the name control "Contact Information," the profile memory module can determine that the attribute information of the name control does not meet the preset conditions, that is, the label name "Contact Information" of the name control cannot be standardized.

[0655] S3503, the portrait memory module uses the target label name included in the mapping relationship satisfied by the attribute information as the target label name of the name control.

[0656] It is understandable that if the attribute information of the aforementioned name control satisfies any of the mapping relationships in the preset mapping rules, it means that the label name of the name control has been standardized. Therefore, the profile memory module can directly use the target label name in the mapping relationship satisfied by the attribute information of the name control as the target label name of the name control. In this way, not only can the power consumption loss caused by repeated standardization be reduced and the utilization rate of mobile phone resources be improved, but the standardization efficiency of label names can also be improved, thereby speeding up the efficiency of information display and providing convenient conditions for users to select the corresponding information later.

[0657] S3504, the profile memory module determines whether there exists a tag name in the user information database whose similarity to the tag name of the name control is greater than the preset similarity.

[0658] It is understandable that if the attribute information of the aforementioned name control does not satisfy any of the mapping relationships in the preset mapping rules, it indicates that the label name of the name control has not been standardized. Therefore, in order to further determine whether the label name of the name control can be standardized, the profile memory module can continue to determine whether there is a label name in the user information database whose similarity to the label name of the name control is greater than a preset similarity. This preset similarity can be pre-set based on actual conditions. In this way, multi-angle standardization of label names can be achieved, meaning that label names can be standardized in different ways, providing a foundation for the accurate display of candidate user information subsequently.

[0659] In some embodiments, if the similarity between a tag name in the user information database and the tag name of the name control is greater than a preset similarity, it indicates that the tag name of the name control is likely the same as the tag name in the user information database. In other words, the tag name of the name control can be standardized. Therefore, the profile memory module can execute S3505. If no tag name in the user information database has a similarity greater than the preset similarity with the tag name of the name control, meaning that the similarity between each tag name in the user information database and the tag name of the name control is less than or equal to the preset similarity, then the tag name of the name control cannot be standardized. Therefore, the profile memory module can execute S3506.

[0660] Specifically, for each tag name in the user information database, the profile memory module can calculate the similarity between that tag name and the tag name of the aforementioned name control. This similarity represents the degree of similarity between the tag name of the name control and other tags sharing the same tag name. Then, if the similarity between any tag name in the user information database and the tag name of the name control is greater than a preset similarity, the profile memory module can determine that the attribute information of the name control meets the preset conditions, meaning the tag name of the name control can be standardized. If the similarity between every tag name in the user information database and the tag name of the name control is less than or equal to the preset similarity, the profile memory module can determine that the attribute information of the name control does not meet the preset conditions, meaning the tag name of the name control cannot be standardized.

[0661] The aforementioned similarity can be determined based on the semantic vector of the tag name of the aforementioned name control (or the first semantic vector) and the semantic vector of the tag name in the user information database (or the second semantic vector). Specifically, the profile memory module can extract features from the tag name of the aforementioned name control according to the semantic vector model to obtain the first semantic vector. This semantic vector model maps words to vectors, so that each word corresponds to a semantic vector. Then, the profile memory module can obtain the second semantic vector of each tag name in the user information database. This second semantic vector is obtained through feature extraction based on the semantic vector model. Finally, for each piece of user information in the user information database, the profile memory module can calculate the similarity between the second semantic vector and the first semantic vector, that is, calculate the similarity between the tag name of the user information and the aforementioned tag names.

[0662] For example, taking semantic vectors as four-dimensional vectors, if the semantic vector of the username is [0.99, 0.12, 0.03, 0.8], and the semantic vector of the name in the user information database is [0.79, 0.42, 0.08, 0.7], and the semantic vector of the mobile phone number in the user information database is [0.19, 0.22, 0.48, 0.7], the profile memory module calculates the similarity between the semantic vector of the username and the semantic vector of the name as 0.88, and the similarity between the semantic vector of the username and the semantic vector of the mobile phone number as 0.60. It can be seen that the similarity of 0.88 between the semantic vector of the username and the semantic vector of the name is greater than the preset similarity of 0.85. Therefore, the profile memory module can determine that the label name "username" of the name control meets the preset conditions.

[0663] S3505, the portrait memory module uses the tag name with a similarity greater than the preset similarity as the target tag name of the name control.

[0664] It's understandable that if a tag name in the user information database has a similarity score greater than the preset similarity score between it and the tag name in the name control, it means the tag name in the user information database is quite similar to the tag name in the name control. Therefore, the profile memory module can directly use the tag name with a similarity score greater than the preset similarity score as the target tag name for the name control. In this way, the target tag name can be determined quickly and accurately, providing a foundation for subsequently determining candidate user information.

[0665] In some embodiments, if multiple tag names in the user information database have a similarity greater than a preset similarity with the tag names of the name control, the profile memory module can use all of these tag names as the target tag names of the name control. For example, if the tag name of the name control is an ID number, and the tag names in the user information database can include ID card number, student ID, and registration number, and the similarity scores 1 between ID card number and ID number, 2 between student ID and ID number, and 3 between registration number and ID number are all greater than the preset similarity, the profile memory module can use ID card number, student ID, and registration number as the target tag names of the name control. This reduces the occurrence of missing user information and improves the accuracy of information display.

[0666] S3506, the profile memory module uploads the label name of the name control to the cloud service.

[0667] Specifically, after determining that there is no tag name in the aforementioned user information database whose similarity to the tag name of the name control is greater than a preset similarity, it indicates that the tag name of the name control cannot be directly standardized. Therefore, the profile memory module can upload the tag name of the name control to the cloud service. Subsequently, the name management module in the cloud service can perform standardization learning on the tag name of the name control. This provides a foundation for subsequent comprehensive standardization of tag names.

[0668] It should be noted that the above-described standardization process for tag names is based on the premise that the attribute information of the name control satisfies any mapping relationship in the preset mapping rules, and that the similarity between a tag name in the user information database and the tag name of the name control is greater than a preset similarity. However, if the preset condition only includes that the attribute information of the name control satisfies any mapping relationship in the preset mapping rules, the mobile phone can execute only processes S3501 to S3503, and upload the tag name of the name control to the cloud service if the attribute information of the name control does not satisfy any mapping relationship in the preset mapping rules. If the preset condition only includes that the similarity between a tag name in the user information database and the tag name of the name control is greater than a preset similarity, the mobile phone can execute only processes S3504 to S3506.

[0669] The above describes the specific implementation process of how the portrait memory module in a mobile phone standardizes the label names of name controls. The following will combine the above... Figure 29 The structure shown and Figure 36 The process of obtaining user information from the user information database is shown in detail. It describes how user information is obtained from a preset collection application and stored in the user information database. Specifically, it may include steps S3601 to S3606:

[0670] S3601, the function settings module in the mobile phone receives the user's trigger operation on any switch control in the settings interface.

[0671] The aforementioned settings interface (or information collection settings interface) is used to configure whether applications on the phone can collect user information. For example, this settings interface can be... Figure 33 The interface shown in (a) is shown in the middle.

[0672] S3602, in response to the user's trigger operation on any switch control in the settings interface, the function setting module sends a function setting instruction to the image memory module.

[0673] In some embodiments, if any switch control in the settings interface is triggered by a user, it indicates that the user wants to adjust the information collection function of the application to which that switch control belongs, that is, the user wants to enable or disable the information collection function. Therefore, the function settings module can send a function settings instruction to the profile memory module. This function settings instruction carries the setting result of the information collection function. This setting result is used to indicate whether the information collection function of the application to which the triggered switch control belongs is enabled.

[0674] It is understandable that the application to which each switch control belongs is pre-defined. For example, using... Figure 33 Taking interface (a) as an example, the application to which switch control 3301A belongs is all applications on the phone. That is to say, as long as switch control 3301A is turned on by the user, the phone can collect user information saved by all applications. The application to which switch control 3302 belongs is the SMS application on the phone. That is to say, as long as switch control 3302 is turned on by the user, the phone can collect user information saved by the SMS application.

[0675] S3603, upon receiving a function setting instruction from the function setting module, the portrait memory module updates the collection application list according to the setting result carried by the function setting instruction. This collection application list includes at least one preset collection application.

[0676] Specifically, upon receiving the aforementioned function setting instruction, the profile memory module can update the list of collected applications based on the settings provided in the instruction. This not only protects user privacy but also provides a foundation for subsequently obtaining user information.

[0677] In one implementation, if the above setting result indicates that the information collection function of the application to which the triggered switch control belongs is enabled, the profile memory module can add the application to which the triggered switch control belongs to the collection application list to obtain a new collection application list.

[0678] In another implementation, if the above settings result indicates that the information collection function of the application to which the triggered switch control belongs is turned off, the profile memory module can delete the application to which the triggered switch control belongs from the collection application list to obtain a new collection application list.

[0679] S3604, the profile memory module determines whether the application to which the triggered switch control belongs is an application in the collected application list.

[0680] In some embodiments, after receiving the above-mentioned function setting instruction, the profile memory module can determine whether the application to which the triggered switch control belongs is an application in the collection application list. If the application to which the triggered switch control belongs is an application in the collection application list, it means that the user wants to turn off the information collection function of the application. Therefore, the profile memory module can execute S3605. If the application to which the triggered switch control belongs is not an application in the collection application list, it means that the user wants to turn on the information collection function of the application. Therefore, the profile memory module can execute S3606.

[0681] S3605, if the application to which the triggered switch control belongs is an application in the collection application list, the profile memory module deletes all user information associated with that application from the user information database.

[0682] Specifically, if the triggered switch belongs to an application in the application collection list, the profile memory module can delete all user information associated with that application from the user information database. In this way, while disabling the application's information collection function, all user information associated with that application can be deleted, thereby protecting user privacy and improving mobile phone security.

[0683] In some embodiments, if the triggered switch control belongs to an application in the collection application list, it indicates that the user does not wish to disclose the user information entered by that application. Therefore, the profile memory module can directly delete all user information associated with that application from the user information database. This protects the user's information privacy and provides security for the user.

[0684] In other embodiments, when the application to which the triggered switch control belongs is an application in the collected application list, the mobile phone can display a pop-up window. This pop-up window may include a prompt message, a delete control, and a retain control. The prompt message indicates whether user information stored in the user database regarding the application to which the triggered switch control belongs should be retained. Subsequently, in response to the user's triggering operation to delete the control in the pop-up window, the user profile memory module in the mobile phone can delete all user information associated with that application from the user database. Alternatively, in response to the user's triggering operation to retain the control in the pop-up window, the user profile memory module in the mobile phone can retain all user information associated with that application in the user database.

[0685] For example, such as Figure 33 As shown, taking the switch control 3304 belonging to the note-taking application as an example, if the user clicks the switch control 3304 belonging to the note-taking application, it means that the user wants to turn off the information collection function of the note-taking application. Therefore, the phone can display as follows: Figure 33The pop-up interface shown in (c) is a pop-up window displayed on top of the settings interface. This pop-up window may include a prompt asking "Do you want to retain the data collected in the notes?", a delete control, and a retain control. If the user clicks the delete control, the phone's profile memory module can delete all user information associated with the notes application from the user information database. If the user clicks the retain control, the phone's profile memory module can retain all user information associated with the notes application from the user information database.

[0686] S3606, if the application to which the triggered switch control belongs is not an application in the collection application list, the profile memory module obtains all user information saved by that application and stores the user information in the user information database.

[0687] Specifically, if the triggered switch control belongs to an application not listed in the application collection list, the user profile memory module can retrieve all user information saved by that application and store it in the user information database. In this way, while enabling the application's information collection function, all user information saved by the application can be obtained, thus providing a basis for subsequently displaying recommendation information.

[0688] It is understood that the above steps S3605 and S3606 are parallel solutions. That is, if the application to which the triggered switch control belongs is an application in the collection application list, the profile memory module can execute step S3605 without executing step S3606; if the application to which the triggered switch control belongs is not an application in the collection application list, the profile memory module can execute step S3606 without executing step S3605.

[0689] In one implementation, the process of the portrait memory module updating the collection application list and the process of the portrait memory module determining whether the application to which the triggered switch control belongs is an application in the collection application list can be executed simultaneously or sequentially. For example, the portrait memory module can execute step S3603 first, and then execute steps S3604 to S3606. Alternatively, the portrait memory module can execute steps S3604 to S3606 first, and then execute step S3603.

[0690] The above describes the specific implementation process of how the user profile memory module in a mobile phone obtains user information from pre-set collection applications and stores it in the user information database. The following will combine the above... Figure 29 The structure shown and Figure 37 The user information processing procedure shown details how to query and adjust user information in the user information database, specifically including steps S3701 to S3708:

[0691] S3701, the information management module in the mobile phone receives user queries for the user information database.

[0692] The aforementioned query operation can be a click on any query control with information display functionality on the mobile phone. In other words, if a user clicks on a query control on the mobile phone, the phone can directly display the user information from the aforementioned user information database.

[0693] In one example, the query control described above could be a query control for all user information in the user information database. For example, see [link to example]. Figure 38 The phone is displaying Figure 38 The settings interface shown in (a) includes a "Personal Information" control (i.e., a query control). In another example, the query control could also be a query control for user information of any user in the user information database. For example, see [link to example]. Figure 38 The phone is displaying Figure 38 The information list interface shown in (b) includes multiple information query controls (i.e., query controls) belonging to various users, such as the information query control for user A, the information query control for user B, etc. In another example, the query control can also be a query control for any user's information in the user information database, etc., without any specific limitation.

[0694] S3702, in response to the user's query operation on the user information database, the information management module sends an information retrieval instruction to the profile memory module.

[0695] In some embodiments, after detecting a user's query operation on user information in the user information database, the information management module can send an information retrieval instruction to the profile memory module, so that the profile memory module can retrieve and display the user information from the user information database. This information retrieval instruction carries the tag name of the user information corresponding to the user to be queried. The user to be queried is determined based on the query control corresponding to the user's query operation. For example, if the query control is for all user information in the user information database, the user to be queried is all users in the user information database. Or, if the query control is for any user information in the user information database, the user to be queried is the user corresponding to the query control.

[0696] S3703, when the profile memory module receives an information retrieval instruction sent by the information management module, it retrieves the target user information from the user information database according to the tag name of the user to be queried carried in the information retrieval instruction.

[0697] Specifically, after receiving the aforementioned information retrieval instruction, the profile memory module can retrieve user information from the user information database that matches the tag name of the user to be queried, i.e., retrieve the target user information. This provides a basis for subsequently displaying the target user information.

[0698] In one example, if the user to be queried is user A, and the tag name is mobile phone number, the profile memory module can retrieve user A's mobile phone number from the user information database and use that mobile phone number as the target user information. In another example, if the user to be queried is all users, and the tag name is username, the profile memory module can retrieve all users' usernames from the user information database and use those usernames as the target user information.

[0699] S3704, the profile memory module sends the target user information to the information management module.

[0700] S3705, the information management module displays the target user information upon receiving it from the profile memory module.

[0701] In some embodiments, after receiving the target user information, the information management module can display the target user information. This enables the user information query function, improves the accuracy of user information display, and thus enhances the user's query experience.

[0702] For example, such as Figure 38 As shown, if the "Personal Information" control in the settings interface (a) is clicked by the user, the phone can display the usernames of all users in the user information database (such as the owner user A, user B, etc.), that is, display... Figure 38 The user list interface is shown in (b). Afterwards, if the user clicks the information query control belonging to user A in the user list interface, the phone can display all user information of user A (such as phone numbers "131****5613", "189****7894", etc.), that is, display... Figure 38 The information details interface shown in (c) is shown in the middle.

[0703] S3706, the information management module receives user adjustments to the user information database.

[0704] The aforementioned information adjustment operations can be deletion, modification, or addition of information for any user in the user information database, etc., without specific limitations. It can be understood that the information deletion, modification, and addition operations are all pre-set according to actual circumstances. For example, the information deletion control can be a swipe operation on a query control. The information modification operation can be a long-press operation on a query control, where the user's finger presses the query control for a preset time. The information addition operation can be a single click operation on an add control, used to add new user information to the user information database. This new user information can be from a new user or new information from an existing user in the database, without specific limitations.

[0705] In one example, with Figure 32 Taking the user list interface shown in (a) as an example, the above-mentioned information deletion operation can be a sliding operation (such as a right-to-left sliding operation) on the information query control belonging to the owner user A. The above-mentioned information addition operation can be a click operation on the add control 3201. In another example, taking Figure 38 Taking the information details interface shown in (c) as an example, the above information modification operation can be a long press operation on the student ID control.

[0706] S3707, in response to the user's operation to adjust information in the user information database, the information management module sends an information adjustment instruction to the profile memory module.

[0707] Specifically, after detecting a user's adjustment operation on the user information database, the information management module can send an information adjustment instruction to the profile memory module to facilitate subsequent adjustments to the user information in the database. This information adjustment instruction carries the information adjustment operation.

[0708] In some embodiments, if the information adjustment operation is an information addition operation, it means the user wants to add new user information. Therefore, the information adjustment instruction only needs to carry the information addition operation. However, if the information adjustment operation is an information deletion operation or an information modification operation, it means the user wants to process user information already stored in the user information database. Therefore, to accurately determine the user information to be processed (i.e., the user information the user wants to process), the information adjustment instruction can also carry the tag name corresponding to the user information to be processed (or the information to be processed). For example, if the information adjustment operation is a deletion operation for user A, the information adjustment instruction carries the information deletion operation, user A, and all tag names corresponding to user A. As another example, if the information adjustment operation is a modification operation for the mobile phone number of user A, the information adjustment instruction carries the information modification operation, "131****5613", and the mobile phone number corresponding to "131****5613".

[0709] S3708, when the profile memory module receives an information adjustment instruction from the information management module, it adjusts the user information database according to the information adjustment instruction.

[0710] In some embodiments, after receiving the aforementioned information adjustment instruction, the profile memory module can directly process the user information in the user information database according to the content carried by the information adjustment instruction. This enables the adjustment of user information (i.e., adding, deleting, and modifying functions), ensuring that the user information in the database is added based on actual user needs, guaranteeing the necessity of each piece of user information, and thus improving the user experience.

[0711] In one implementation, when the aforementioned information adjustment instruction carries an information addition operation, the profile memory module can add the newly entered user information to the user information database. For example, such as... Figure 32 As shown, the mobile phone is displaying the personal information interface shown in interface (a). If the user clicks the add control 501 in the personal information interface, the mobile phone can display... Figure 32 The information editing interface shown in section (b) is shown below. Subsequently, in response to the user's input of any information control (such as the username information control 3202) in the information editing interface, the mobile phone can display the input content of that information control (e.g., Zhao Yiyi). Then, if the user clicks the "Save" control 3203 in the information editing interface, the user profile memory module in the mobile phone can add the newly entered user information to the user information database. The newly entered user information includes the username "Zhao Yiyi", the ethnicity "Han", and the mobile phone number "132****7788".

[0712] In another implementation, when the aforementioned information adjustment instruction carries an information deletion operation and the tag name corresponding to the user information of the user to whom the information deletion operation belongs, the profile memory module can delete the user information corresponding to the tag name from the user information database, that is, delete the tag name of the aforementioned information to be processed. For example, such as... Figure 39 As shown, the phone is displaying the personal information interface shown in interface (a). Responding to the user's swipe gesture on the information query control for user B, indicating that the user wants to delete all user information for user B, the phone's profile memory module can delete all user information for user B from the user information database. Afterwards, the phone can display... Figure 39 The personal information interface shown in (b) is the personal information interface that does not include the information query control belonging to user B.

[0713] In another implementation, when the aforementioned information adjustment instruction carries an information modification operation and the tag name corresponding to the user information of the user to whom the information modification operation belongs, the profile memory module can modify the user information corresponding to the tag name and replace the modified user information with the modified user information, saving it to the user information database. For example, as shown... Figure 38 As shown, in response to the user's long press operation on the student ID control in interface (c), indicating that the user wants to modify the student ID of the owner user A, the profile memory module in the phone can modify the student ID "000***147" of the owner user A based on the newly entered student ID content, and save the modified student ID to the user information database.

[0714] It should be noted that, Figure 37 The execution order of the information query and information adjustment processes shown is not limited. For example, the mobile phone can first execute steps S3701 to S3705, and then execute steps S3706 to S3708; or, the mobile phone can first execute steps S3706 to S3708, and then execute steps S3701 to S3705. Alternatively, the mobile phone can execute only steps S3701 to S3705, or only steps S3706 to S3708.

[0715] It should be understood that the corresponding steps performed by each module in the aforementioned mobile phone can also be performed by other modules in the mobile phone. For example, the process by which the profile memory module detects the label name of the name control can also be performed by the name management module in the cloud service. As another example, the process by which the profile memory module adjusts user information in the user information database can also be performed by the information management module; no specific limitations are imposed.

[0716] The above describes the process of displaying corresponding candidate user information based on triggered fill-in controls. The following will describe the specific implementation process of displaying candidate user information corresponding to all fill-in controls in the data entry interface. For example... Figure 40 As shown, the process is as follows:

[0717] S4001, the form-filling assistant in the mobile phone receives one-click filling operations from the user on the input screen.

[0718] For example, the above one-click fill operation can be performed by the user on the interface to be filled in (such as...). Figure 41 The triggering operation of the "One-click Fill" control in the interface shown in (a) can also be the user's voice command including "One-click Fill" when the mobile phone displays the input interface, and there is no specific limitation.

[0719] S4002, in response to the user's one-click entry operation on the data entry interface, the form filling assistant obtains a list of attribute information. This list includes attribute information for each name control on the data entry interface, and the attribute information for each name control includes its label name.

[0720] In one implementation, the form-filling assistant can sequentially retrieve the attribute information of the name controls in the display order on the data entry interface, and then combine the retrieved attribute information of multiple name controls into an attribute information list. In other words, the attribute information of the name controls in this attribute information list is sorted according to the display order. This provides a basis for the subsequent sequential display of recommendation information, reducing the occurrence of display issues caused by mismatches between recommendation information and tag names.

[0721] For example, such as Figure 41 As shown, the phone is displaying... Figure 41 The data entry interface shown in (a) displays the name controls in the following order: "Recipient," "Mobile Number," and "Detailed Address." Therefore, the aforementioned attribute information list can include the attribute information of the "Recipient" control, the "Mobile Number" control, and the "Detailed Address" control; that is, the attribute information controls contain the recipient, mobile number, and detailed address.

[0722] S4003, the form filling assistant sends a list of attribute information to the portrait memory module in the mobile phone.

[0723] Specifically, after obtaining the aforementioned list of attribute information, the form-filling assistant can send the list to the profile memory module, so that the profile memory module can determine the corresponding candidate user information in sequence based on the attribute information in the list, thereby improving the accuracy of the candidate user information determination.

[0724] S4004, when the portrait memory module receives the attribute information list sent by the form filling assistant, it standardizes the label name of each name control in the attribute information list to obtain the target label name of each name control.

[0725] In some embodiments, after receiving the aforementioned list of attribute information, the portrait memory module can detect the label name of each name control in the list and obtain a detection result. Then, if the detection result indicates that the label name of the name control can be standardized, the portrait memory module can standardize the label name of the name control to obtain the target label name of the name control.

[0726] The detection and standardization processes for the label names of the aforementioned name controls are similar to those described above. Figure 35 The standardization process for the label names shown is similar and will not be repeated here.

[0727] S4005, the portrait memory module divides the target label names of multiple name controls to obtain at least one set of sub-attribute information lists.

[0728] Specifically, after obtaining the target label name for each name control, the profile memory module can divide the target label names of multiple name controls to obtain at least one set of sub-attribute information lists. Each set of sub-attribute information lists includes at least one target label name for a name control.

[0729] It's understandable that, since a user information entry interface cannot require more than two entries of the same user information, meaning that the interface cannot simultaneously display at least two name controls with the same target tag name, if the interface displays a second name control with the same target tag name, the profile memory module can use that second name control's target tag name as the second user's tag name. Furthermore, generally, name controls belonging to different users' target tag names will not be displayed interchangeably in the entry interface. Therefore, after determining the second user's target tag name, the profile memory module can use the target tag names of all name controls displayed before the second name control as the first user's tag name, thus obtaining the first set of sub-attribute information lists. In other words, the target tag names of name controls included in the same set of sub-attribute information lists all belong to the same user's tag name.

[0730] After obtaining the first set of sub-attribute information list, the profile memory module can determine whether there are identical target tag names in the remaining attribute information (excluding the first sub-attribute information list). If identical target tag names exist in the remaining attribute information, it indicates that a third user's tag name may exist in the attribute information list. Therefore, the profile memory module can assemble all target tag names in the remaining attribute information that are sorted before the second identical target tag name into a second set of sub-attribute information list, until no identical target tag names exist in the remaining attribute information, thus obtaining at least one set of sub-attribute information lists. If no identical target tag names exist in the remaining attribute information, it indicates that no third user's tag name exists in the attribute information list. Therefore, the profile memory module can use the remaining attribute information as the second set of sub-attribute information lists.

[0731] In one implementation, when multiple name controls contain multiple identical target tag names, the profile memory module can divide the target tag names of the multiple name controls based on these identical target tag names, obtaining at least one set of sub-attribute information lists. Each set of sub-attribute information lists includes one of the multiple identica...

Claims

1. An information processing method characterized by comprising: The method is applied to an electronic device, and the method comprises: displaying a first interface; the first interface comprises a table and a first control, and does not comprise a second control for inputting or triggering display of first information; the table comprises a first table label and a to-be-filled control, the first table label is a table label associated with the to-be-filled control in the first interface; the first information comprises all information to be filled in the first interface; in response to a user operation on the first control, based on the first table label, filling content to be filled matching the first table label into the to-be-filled control.

2. The method of claim 1, wherein, Before the filling of the content to be filled matching the first table label into the to-be-filled control based on the first table label, the method further comprises: in response to the user operation, identifying a label text of the first table label from the first interface; based on the label text of the first table label, obtaining content to be filled matching the first table label from a user information library.

3. The method of claim 2, wherein, The filling of the content to be filled matching the first table label into the to-be-filled control based on the first table label comprises: based on a first correspondence relationship among the first table label, the to-be-filled control and the content to be filled, sequentially performing a filling operation on each to-be-filled control to fill the content to be filled into the to-be-filled control.

4. The method of claim 3, wherein, The sequentially performing of the filling operation on each to-be-filled control based on the first correspondence relationship comprises: selecting one to-be-filled control as a target filling control; based on the first correspondence relationship, filling content to be filled corresponding to the target filling control into the target filling control; returning to the step of selecting one to-be-filled control as a target filling control until the filling operation on all to-be-filled controls is completed.

5. The method of claim 4, wherein, Before the filling of the content to be filled matching the first table label into the to-be-filled control based on the first table label, the method further comprises: in response to the user operation, identifying coordinate information of the to-be-filled control from the first interface; after the obtaining of the content to be filled matching the first table label from the user information library, the method further comprises: generating a content vector corresponding to the content to be filled according to the first correspondence relationship; the content vector comprises a label text of the first table label, coordinate information of the to-be-filled control and the content to be filled; storing the content vector corresponding to each content to be filled into a preset data structure; the selecting of one to-be-filled control as a target filling control comprises: taking out a first content vector of the preset data structure, and determining a to-be-filled control corresponding to coordinate information of the to-be-filled control included in the first content vector as the target filling control; the filling of the content to be filled corresponding to the target filling control into the target filling control based on the first correspondence relationship comprises: filling the content to be filled in the first content vector into the target filling control.

6. The method according to claim 4 or 5, characterized in that, After the filling of the to-be-filled content corresponding to the target fill-in control into the target fill-in control, before the step of returning to the selection of one to-be-filled control as the target fill-in control, the method further comprises: detecting whether an abnormal situation occurs; if the abnormal situation occurs, performing an exception handling; the exception handling is used to exit the abnormal situation.

7. The method of claim 6, wherein, The abnormal situation includes that the keyboard mode is not exited. The exception handling includes performing a keyboard exit operation. The step of returning to the selection of one to-be-filled control as the target fill-in control includes, in the case that the keyboard mode has been exited, returning to the selection of one to-be-filled control as the target fill-in control.

8. The method of claim 6, wherein, The abnormal situation includes that the table is shifted. The exception handling includes: obtaining to-be-filled content matched with a second table label in the shifted table; the second table label is a table label associated with a to-be-filled control in the shifted table; obtaining a second correspondence relationship between the second table label, the to-be-filled control and the to-be-filled content; the second correspondence relationship is used to perform a fill-in operation on the to-be-filled control in the shifted table; or obtaining coordinate information of the to-be-filled control in the shifted table; updating the first correspondence relationship based on the coordinate information of the to-be-filled control in the shifted table; the updated first correspondence relationship is used to perform the fill-in operation on the to-be-filled control in the shifted table.

9. The method of claim 6, wherein, The abnormal situation includes that the keyboard mode is not exited and the table is shifted. If the abnormal situation occurs, the exception handling includes: detecting whether the keyboard mode is not exited; if the keyboard mode is not exited, performing a keyboard exit operation; after the keyboard exit operation is performed, if the keyboard mode has been exited, detecting whether the table is shifted; if the table is shifted, performing an exception handling for the table shift; the exception handling for the table shift is used to obtain new to-be-filled content, which is used to perform a fill-in operation on a to-be-filled control in the shifted table; after the keyboard exit operation is performed, if the keyboard mode is not exited, stopping the fill-in operation on the to-be-filled control.

10. The method according to any one of claims 1-9, characterized in that, The method further comprises: identifying a filled control of the table in the first interface and a table label associated with the filled control; generating a filled content vector based on the association relationship between the filled control and the table label; the filled content vector is used to update a user information library.

11. An electronic device, comprising: The electronic device comprises a display screen, a processor, a memory and a computer program stored in the memory; the display screen and the memory are coupled with the processor respectively; when the electronic device is running, the processor executes the computer program to implement the method in any one of claims 1-10.

12. A computer-readable storage medium, characterized in that, The computer program is stored in the electronic device, and when the computer program is executed by the processor of the electronic device, the method in any one of claims 1-10 is implemented.

13. A computer program product, characterised in that, A computer program comprising computer program elements, which, when executed by a processor, carry out the method according to any one of claims 1-10. A computer program comprising computer program elements, which, when executed by a processor, carry out the method according to any one of claims 1-10.

Images