Table information filling method, electronic device and readable storage medium

Abstract

The embodiment of the application is applied to the computer technology field, and provides a table information filling method, an electronic device and a readable storage medium, which are used for one-key automatic filling of multiple contents of a table. The electronic device can display an interface including at least one filling box and simultaneously display a first control. In response to input of the first control by a user, the electronic device displays a recommended first service control after analyzing the interface content. After receiving an operation of inputting the first service control, the electronic device displays a keyboard and displays at least one form filling control and information corresponding to the form filling control in an area above the keyboard. After receiving an operation of any form filling control by the user, the electronic device automatically fills at least one to-be-filled box in the table filling interface. Since the label of the form filling control prompts the subject of information to be filled into the table filling interface, the user can perform one-key form filling operation by selecting the corresponding form filling control, thereby saving cumbersome operation and meeting the use demand.

Description

[0001] This application claims priority to Chinese Patent Application No. 202411771327.9, filed on December 3, 2024, entitled "A Vision-Based Adaptive Table Filling Method for Single and Multiple Information", the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application relates to the field of computer technology, and in particular to a method for filling in form information, an electronic device, and a readable storage medium. Background Technology

[0003] In scenarios requiring the collection of personal information, such as initial account registration, users are typically required to fill in their name and mobile phone number on a form. Current technology requires users to manually enter this information item by item, which can be cumbersome, waste a significant amount of user time, and result in a poor user experience. Summary of the Invention

[0004] To address the aforementioned issues, embodiments of this application provide a method for filling out form information, an electronic device, a readable storage medium, and a program product, which can effectively and automatically fill out multiple controls of a form with one click.

[0005] To achieve the above objectives, the embodiments of this application adopt the following technical solutions: In a first aspect, the embodiments of this application provide a method for filling in form information, applied to an electronic device, comprising: displaying a first interface, the first interface including a first fill box, a second fill box, and a first control; responding to a first operation input by a user to the first control, displaying a first service control; responding to a second operation input by a user to the first service control, displaying a keyboard and displaying a first form-filling control and first information in a first area above the keyboard; responding to a third operation input to the first form-filling control, displaying the first information in the first fill box and displaying second information in the second fill box, the second information being associated with the first information and different from the first information.

[0006] This embodiment provides a method for filling in form information, where the electronic device can display an interface including at least one fill-in box, for example, displaying... Figure 2A(a) includes the first input box 12 and the second input box form filling interface 100. The electronic device can also display a first control, such as control 10, simultaneously with the form filling interface. The first control can be displayed at the bottom of the form filling interface to avoid affecting normal user operation. In response to the user's input to the first control (first operation), the electronic device can display a first service control. The first service control is a functional control recommended by the electronic device after analyzing the interface content; for example, the service control 105 displayed on the electronic device's form filling interface 100 is a functional control recommended by the electronic device after analyzing the interface content as a table. The electronic device can also recommend different functional controls based on different interfaces. The electronic device can receive input to the first service control, such as... Figure 2A In step (c), when the form filling interface 100 includes a first service control 105, the electronic device can receive input operations (second operations) onto the first service control 105. These operations may include long-press operations, click operations, etc. In response to the user's input operation onto the first service control, the electronic device displays a keyboard and displays at least one form filling control and corresponding information in the area above the keyboard (first area). For example, Figure 2A As shown in (d), the label information "Wang Wu" (first information) of the form-filling control 181 is displayed in the area above the keyboard 184. After displaying the form-filling control, the electronic device can automatically fill in at least one of the fields in the form-filling interface after receiving any operation from the user on any form-filling control. For example, in response to the user's operation on the form-filling control 181 (third operation), the electronic device fills in the name "Wang Wu" (first information) in the name field 11' and the telephone number "1234567890" (second information) in the telephone number field 12'. In addition, the information "Province A, City B, District C, Street D" can also be filled in the address field 13'. Among them, the telephone number "1234567890" and / or "Province A, City B, District C, Street D" are information associated with "Wang Wu", but are different from "Wang Wu". Since the label of the form-filling control indicates the main body of the information to be filled in the form-filling interface, the user can perform a one-click form-filling operation by selecting the corresponding form-filling control, which not only saves the user from tedious operations but also meets the user's actual usage needs.

[0007] In one possible design approach, in response to a user's input to a first control (the first operation), the electronic device can display a first service control. This includes responding to the user's input to the first control and performing a screenshot operation on the first interface to obtain a first screenshot. The electronic device can also employ a visual detection model, or a model with table frame recognition capabilities, such as the YOLO (You Only Look Once) model, to determine that the first screenshot is a form filling interface. When the first interface of the electronic device is a form filling interface, the electronic device can determine and display the first service control used for form filling. The electronic device also displays other functional controls related to the form filling interface, such as controls for exporting form files and controls for copying forms. Based on this, this application can recommend corresponding functional controls to the user based on the content of the page, improving the convenience of user operation of the page.

[0008] In one possible design approach, in response to a user's input to the first service control, the electronic device takes a screenshot (second screenshot) of the first interface (form filling interface 100). After acquiring the second screenshot, the electronic device can use a model capable of simultaneously detecting fill boxes and their text to determine the fill boxes and their labels. For example, after acquiring the second screenshot, the electronic device performs a first detection to identify text information and text boxes. This first detection can utilize Optical Character Recognition (OCR) technology to perform optical character recognition on the second screenshot, recognizing the text information and the corresponding text boxes for each piece of text. Simultaneously, the electronic device can perform a second detection of fill boxes on the second screenshot. The electronic device can use a visual detection model or a model with form box recognition capabilities, such as the YOLO (You Only LookOnce) model, to determine the fill boxes and their categories in the second screenshot. Electronic devices can use a center-point matching algorithm to match text boxes and fill-in boxes, determining whether the coordinates of the text box's center point and the fill-in box's table frame coordinates fall within the same range. After identifying the fill-in box and its corresponding text information, the electronic device queries a user information database based on the fill-in box and its corresponding text information, determining at least one user information that matches the text information. As one implementation, for example, the query might match the name "Wang Wu" (first information), the phone number "1234567890" (second information), and / or the detailed address "Province A, City B, District C, Street D".

[0009] After identifying the user information (first information, second information) that matches the text information, the electronic device can display at least one form-filling control, and each control can display corresponding label information, allowing users to distinguish between the control types. Since the name is the only unique identifier for the information to be filled in, the label information can be the full name or an abbreviation of the name. Based on this, this application can accurately identify and match the information to be filled in the form with the user information stored in the electronic device's database, enabling users to fill in their information in the form with a single click, thus improving the accuracy and convenience of the form-filling experience.

[0010] In one possible design approach, before the electronic device displays the keyboard and shows at least one form-filling control and corresponding information (first information) in the area above the keyboard (first area) in response to user input to the first service control, it further includes determining whether the keyboard is the electronic device's system keyboard. If the keyboard displayed by the electronic device is the system keyboard, the electronic device calls the system interface AndroidReflectUtil.getInputMethodHeight (first interface) to obtain the display area of ​​the system keyboard, and draws the form-filling control above or below the display area of ​​the system keyboard to ensure that candidate options are displayed outside the display area of ​​the system keyboard. Afterwards, the electronic device calls the InputMethodManager class interface (second interface) to fill in the user information (first information, second information) matching the form text information into the fill-in boxes.

[0011] When the keyboard displayed on the electronic device is not the system keyboard, the electronic device can obtain a screenshot of the first interface (third screenshot), identify the keyboard display area using a keyboard detection model, and determine the top position of the keyboard by determining the minimum absolute value of the y-coordinate of each key. It then determines the bottom position of the keyboard by determining the maximum absolute value of the y-coordinate of each key, thereby obtaining the display area of ​​the system keyboard. A form-filling control is then drawn above or below this display area. Afterwards, the electronic device can take a screenshot of the first interface (third screenshot) to detect keyboard keys. Combining the detected key information, the electronic device performs simulated clicks on the keyboard keys to fill in the user information "Wang Wu" (first information) and the phone number "1234567890" (second information) into the first fill box. Based on this, and considering the different keyboard types available for user input across different form filling interfaces, this solution can draw form-filling controls outside of different keyboard areas. Users can select the corresponding fill-in control for one-click form filling, saving users tedious operations and meeting their actual usage needs.

[0012] In one possible design, after the electronic device enters first information in a first input box and second information in a second input box, in response to the user's input in the first input box (fourth operation), the electronic device displays a second input control and its corresponding label information (third information). Figure 2B As shown in (b), in response to the user's selection of fill box 11, the electronic device displays fill control 183 and its label information "Zhang San" in the form filling interface 100. In response to the user's input on the second fill control, such as fill control 183, as shown in (b), the electronic device displays fill control 183 and its label information "Zhang San" in the form filling interface 100. Figure 2B As shown in (c), the electronic device displays window 21, and displays controls 22 and 23 within window 21. Control 22 includes a prompt to replace only the user-selected input box, and control 23 includes a prompt to replace all already filled boxes. When the user inputs to control 23, such as... Figure 2B As shown in (d), in the form entry interface 100, the electronic device replaces the name in the name field 11 with the name "Zhang San" and replaces the phone number in the phone number field 12 with the phone number "1987654320". When the electronic device receives user input on the control 22, such as... Figure 2B As shown in (e), in the form filling interface 100, the electronic device enters the name "Zhang San" in the name input box 11, while the information already entered in the phone number input box 12 and the address input box 13 remains displayed as "1234567890". Based on this, users can choose to modify the user information already entered in multiple form input boxes with one click or only modify the user information already entered in the selected form input box, which improves the convenience of automatically modifying the entered form and greatly enhances the user experience.

[0013] In one possible design, when a user interacts with a field associated with "Name," the electronic device can modify the information already entered in multiple fields. As another example, when a user interacts with a field associated with user information other than "Name," the electronic device modifies the information already entered in that field. See also Figure 2B In (a), the user can modify input fields associated with user information other than "Name," such as the "Phone" input field 12' or the "Detailed Address" input field 13'. As another example, when the user operates on an input field associated with user information other than "Name," the electronic device modifies the information already entered in that input field. For example, when the user operates on an input field associated with user information other than "Name," the electronic device modifies the information already entered in that input field. For details, please refer to [link / reference]. Figure 2BIn section (a), users can modify input fields associated with user information other than "Name," such as the "Phone" input field 12' or the "Detailed Address" input field 13'. Figure 11 As shown in (a), the electronic device displays form-filling controls 30 on the form-filling interface 100. In response to the user's selection of the form-filling controls 30, the electronic device displays the name "Wang Wu" in the name field 11' and enters the phone number "1987654320" in the phone number field 12'. Based on this, the user's interaction on the form-filling interface can be greatly reduced, improving the user experience.

[0014] In one possible design, the information to be entered into the form, such as the first piece of information, originates from user information stored in the electronic device. The electronic device then displays an interface (second interface) containing the user information, for example... Figure 15 The image display interface 1500 shown in Figure (a) includes an ID card image 1501. A first control 10 is displayed at the bottom of the image display interface 1500. The ID card image 1501 includes the user information "Name": "Wang Wu" (first information). In response to the user's input on control 10, the electronic device displays a second service control, as shown in the figure, the first service control 1502. In response to the user's operation on the second service control 1502, the electronic device extracts and saves the user information "Name": "Wang Wu" from the ID card image 1501 to the user information database. Based on this, this application provides a quick way to save users' personal information used for form filling, reducing the tedious process of users manually entering information and improving the convenience of user information saving.

[0015] In a second aspect, embodiments of this application provide an electronic device, including: a memory and a processor; the memory is used to store computer execution instructions, and when the electronic device is running, the processor executes the computer execution instructions stored in the memory to cause the electronic device to perform the device connection method as described in any of the first aspects above.

[0016] Thirdly, embodiments of this application provide a readable medium storing instructions that, when executed on an electronic device, enable the electronic device to perform the form information filling method described in any of the first aspects above.

[0017] Fourthly, embodiments of this application provide a program product that includes instructions that, when executed by an electronic device, enable the electronic device to perform the form information filling method described in any of the first aspects above.

[0018] The beneficial effects of the second to fourth aspects mentioned above can be referred to the relevant descriptions in the first aspect and various possible implementations of the first aspect, which will not be repeated here. Attached Figure Description

[0019] Figure 1 This is a schematic diagram of a form-filling interface according to an embodiment of this application;

[0020] Figure 2A This is a schematic diagram of a form-filling interface according to an embodiment of this application;

[0021] Figure 2B This is a schematic diagram of a form-filling interface according to an embodiment of this application;

[0022] Figure 3 This is a schematic diagram of a form-filling interface according to an embodiment of this application;

[0023] Figure 4A This is a schematic diagram of a process involved in an embodiment of this application;

[0024] Figure 4B This is a schematic diagram of a process involved in an embodiment of this application;

[0025] Figure 5 This is a schematic diagram of a process involved in an embodiment of this application;

[0026] Figure 6 This is a schematic diagram of a process involved in an embodiment of this application;

[0027] Figure 7 This is a schematic diagram of a form-filling interface according to an embodiment of this application;

[0028] Figure 8 This is a schematic diagram of a process involved in an embodiment of this application;

[0029] Figure 9 This is a schematic diagram of a process involved in an embodiment of this application;

[0030] Figure 10 This is a schematic diagram of a process involved in an embodiment of this application;

[0031] Figure 11 This is a schematic diagram of a form-filling interface according to an embodiment of this application;

[0032] Figure 12 This is a schematic diagram of a form-filling interface according to an embodiment of this application;

[0033] Figure 13 This is a schematic diagram of an information interface according to an embodiment of this application;

[0034] Figure 14 This is a schematic diagram of an information interface according to an embodiment of this application;

[0035] Figure 15This is a schematic diagram of an information interface according to an embodiment of this application;

[0036] Figure 16 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of this application;

[0037] Figure 17 A schematic diagram of the software structure of an electronic device provided in an embodiment of this application; Detailed Implementation

[0038] The technical solutions of the embodiments of this application are described below with reference to the accompanying drawings. In the description of the embodiments of this application, the terminology used in the following embodiments is for the purpose of describing specific embodiments only and is not intended to limit the application. As used in the specification and appended claims of this application, the singular expressions "a," "the," "the," "the," and "this" are intended to also include expressions such as "one or more," unless the context clearly indicates otherwise. It should also be understood that in the following embodiments of this application, "at least one" and "one or more" refer to one or more (including two). The term "and / or" is used to describe the relationship between related objects, indicating that three relationships can exist; for example, A and / or B can represent: A alone, A and B simultaneously, or B alone, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship.

[0039] References to "one embodiment" or "some embodiments" in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized. The term "connection" includes direct connections and indirect connections, unless otherwise stated. "First" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated.

[0040] In the embodiments of this application, the words "exemplarily" or "for example" are used to indicate examples, illustrations, or explanations. Any embodiment or design described as "exemplarily" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design solutions. Specifically, the use of the words "exemplarily" or "for example" is intended to present the relevant concepts in a specific manner.

[0041] Currently, electronic devices offer various application interfaces for displaying information, including form-filling interfaces for entering user information, such as courier address entry interfaces and account registration interfaces. These form-filling interfaces typically consist of multiple input boxes and corresponding label information.

[0042] For example, such as Figure 1 As shown in (a), the electronic device can display a desktop interface 01, which contains at least one application icon, such as icon 02 for a first application for shopping, icon 03 for a second application for time booking, etc. When the user interacts with icon 02, such as... Figure 1 As shown in (b), the electronic device will display the shopping application's interface 04. Interface 04 includes control 05, which triggers the display of a form-filling interface. For example, in response to a user's action on control 05, such as... Figure 1 As shown in (c), the electronic device displays a form filling interface 100 for filling in delivery address information. This interface 100 includes multiple form filling boxes for collecting user information, such as filling boxes 11, 12, and 13, and a save control 14. Each filling box includes a filling area and a label display area. For example, filling box 11 includes a label display area 15 and a filling area 16. The label display area 15 displays the prompt "Name," instructing the user to enter their name in filling box 11. The user can select filling box 11 by clicking and input information using the keyboard displayed on the electronic device. After completing filling box 11, the user can continue to select the next filling box to input information. After completing filling boxes 11 through 13, the user can click the save control 14 to save the entered information. Therefore, in the prior art, when users input information in a form filling interface, they need to click each box one by one and then type in the information sequentially. In scenarios involving multiple forms, this operation is cumbersome, resulting in a poor user experience.

[0043] Furthermore, the information required for user input may be the same when electronic devices display different form-filling interfaces. For example, in response to a user's action on the appointment application icon 03, such as... Figure 1As shown in (d), the electronic device will display a form filling interface 07 for account registration. Form filling interface 07 contains multiple fill-in boxes, where fill-in boxes 08 and 09 require the same information as shown in the image. Figure 1 The form filling interface 100 shown in (c) has identical filling boxes 11 and 12. Therefore, when filling out different forms, even if the required information is the same, users still need to tediously repeat the information using the keyboard, further reducing the user experience.

[0044] Based on this, this application proposes a method for filling out form information and an electronic device. The electronic device can respond to user touch operations and trigger an automatic form-filling function. The electronic device will automatically identify the fill-in boxes in the form and search for user information stored in the device that matches the fill-in boxes, thereby achieving automatic filling. This method can significantly reduce user interaction on the form-filling interface and improve the overall user experience.

[0045] For example, the electronic device can be a mobile phone, tablet computer, personal computer (PC), smart screen, desktop, laptop, handheld computer, notebook computer, ultra-mobile personal computer (UMPC), netbook, wearable device such as a smartwatch, artificial intelligence (AI) speaker, and in-vehicle device. It can also be various teaching aids (such as learning machines and early education machines), smart toys, portable robots, personal digital assistants (PDAs), augmented reality (AR) / virtual reality (VR) devices, media players, etc. Furthermore, it can be a device with mobile office functions, a device with smart home functions, a device with audio-visual entertainment functions, or a device supporting smart travel. This application embodiment does not impose special limitations on the specific form of the electronic device. The structure of the electronic device will be described in detail in the later embodiments.

[0046] Figures 2A to 6 This is one specific implementation method of the form information filling method in this scheme. The following will combine... Figures 2A to 6 This section explains the implementation scheme of the form information filling method provided in this embodiment.

[0047] S1. Display the function control that triggers automatic form filling on the first interface.

[0048] In the embodiments provided in this solution, the electronic device can recommend and display functional controls that trigger the automatic form filling process based on the interface content. When the user selects the functional control, the electronic device enters the form filling process, and can automatically fill in the saved user information in the form filling interface.

[0049] Specifically, such as Figure 4A As shown, steps S100-S104 are the process of displaying the functional control used to trigger the automatic filling of the form in this solution.

[0050] S100: The electronic device displays a first interface, which contains a first control.

[0051] The first interface is the currently displayed interface shown by the display module of the electronic device. Optionally, the first interface can be a form filling interface, and the first control is displayed in the first area of ​​the form filling interface.

[0052] like Figure 2A As shown in (a) above, when the electronic device displays the form filling interface 100, a first control 10 is also displayed in the bottom area 101 of the form filling interface 100. The form filling interface 100 includes at least one fill box, such as fill box 11. Referring to the foregoing, fill box 11 includes label information 15 and a corresponding fill area 16, which are displayed on the same row. Optionally, the label information 15 is "Name," used to prompt the user to enter their name in fill box 11 by operating the corresponding fill area 16. Optionally, when the first interface 100 includes multiple fill boxes, the label box and fill box of each table frame group in the form filling interface 100 are usually located in the same row or column of the table. (Continue referring to...) Figure 2A In (a) of the first interface 100, there are also fill-in boxes 12 and 13, which are displayed in the same column. Fill-in box 12 and its label information "Telephone" are displayed in the same row, and fill-in box 13 and its label information "Detailed Address" are displayed in the same row. Furthermore, the first interface 100 may also include a save control 14 for saving the content entered in the fill-in boxes. In this solution, the interface with a similar page layout will be referred to as the form filling interface below.

[0053] In this configuration, the first control 10 remains displayed even when the electronic device displays different application interfaces. For example, when the interface switches to another interface, the first control 10 is displayed as follows: Figure 3 When the other form filling interface 800 is displayed, or as shown below... Figure 15 The image display interface 1500 shown here still displays the first control 10. The first control 10 can be operated by the user, thereby triggering the electronic settings to perform subsequent operations.

[0054] S101: Receive user operations on the first control.

[0055] Optionally, the user's operation on the first control can be a long press operation. For example, a swipe operation, a multiple click operation, a custom gesture operation, etc. This application embodiment is only used as an example to illustrate the method of long pressing the first control, and this application does not limit it.

[0056] In one implementation, such as Figure 2A As shown in (b), to enhance the user experience, after receiving the user's operation on the first control 10 in step S101, an animation effect can be displayed to indicate that the electronic device is currently analyzing the currently displayed interface. Optionally, the electronic device can enlarge the first control 10 and display an animation effect 17 on the first interface for a preset duration. The animation effect can be a flashing effect; this embodiment does not limit the type of animation effect. The preset duration of the animation effect can be, for example, 1-N seconds, and specific settings will be described later. Optionally, during the user's long-press operation on the first control 10, the electronic device can provide vibration feedback through a motor or play a prompt sound through a speaker to enhance the user's perception experience.

[0057] S102: Take a screenshot of the first interface to obtain the first screenshot.

[0058] In response to the user's operation on the first control, the electronic device performs a screenshot operation on the first interface to obtain the first screenshot.

[0059] For example, the electronic device screenshot module responds to the user's operation on the first control 10, such as... Figure 2A The form filling interface 100 shown in (a) generates the first screenshot. Optionally, the first screenshot may not include elements such as the status bar and the first control. Optionally, the first screenshot may not be displayed on the electronic device and / or saved to the electronic device's photo library to avoid interfering with the user's normal use.

[0060] S103: Based on the first screenshot, determine the recommended features.

[0061] After obtaining the first screenshot on the electronic device, the recommended functions can be determined based on the analysis of the first screenshot.

[0062] As an example, the recommendation module of an electronic device can determine the category of a first screenshot by analyzing one or more features such as page layout, content characteristics, color features, and shape features, and then determine the recommended function based on the category of the first screenshot. As another example, an image classification model can be trained using labeled image data to determine the category of the first screenshot. As yet another example, the electronic device can also determine the category of the first screenshot by inputting it into a large model for discrimination. As yet another example, the electronic device can also use a visual detection model, or a model with table frame recognition capabilities, such as the YOLO (You Only Look Once) model, to determine that the first screenshot is a form filling interface.

[0063] When the electronic device determines that screenshot A is categorized as a table, the first interface can be considered a table filling interface, and the electronic device's recommendation module can determine that the recommended function is for filling out a table. In other embodiments, the electronic device may also recommend functions such as exporting table files and copying tables, etc., which are not limited here.

[0064] Optionally, the duration of steps S102-S103 performed by the electronic device is related to the display of the aforementioned steps. Figure 2A The preset duration of the animation shown in (b) is the same, which can prompt the user that intelligent perception analysis is currently being performed, thereby improving the user's operation feedback experience.

[0065] S104: Display the first service control based on the recommended function type.

[0066] As an example, after the electronic device determines the recommended function according to step S103, the display module can draw and display the first service control.

[0067] When the recommended function of an electronic device is for filling out forms, the device displays a service control as a quick entry point to launch the form-filling function. For example, ... Figure 2A As shown in (c), the electronic device displays the following on interface 100: Figure 2A The first service control 105 and the first window 102 are shown in (c). The first service control 105 displays the prompt message "Smart Form Filling," prompting the user to fill out the form. The first window 102 may also include a command input box 103 and / or a voice control 104 for receiving user input commands and performing corresponding operations. For example, the command input box 103 displays text commands entered by the user via a keyboard, and the voice control 104 receives user voice input commands. After receiving user input commands, the electronic device can provide a corresponding response and / or automatically execute the task corresponding to the command.

[0068] In one implementation, since the electronic device stores user information for multiple users, and users need to select information to fill in when filling out a form, the first service control 105 can be used to trigger the electronic device to display at least one form-filling control. The form-filling control includes label information to prompt the user to select information that the electronic device will automatically fill in. That is, after receiving a trigger operation on the first service control 105, the electronic device can execute steps S106-S108.

[0069] Based on this, electronic devices can intelligently analyze the interface content to recommend functional controls. Before the user actually clicks the form filling box, the controls related to the form filling function are displayed first, so that the user can complete the form filling function through subsequent operation of the controls.

[0070] Furthermore, electronic devices can recommend other related service controls based on the content currently displayed on the interface. For example, ... Figure 15 As shown in (a), when the interface 1500 currently displayed by the electronic device includes an image, and the image content includes card-related information, in response to the user's operation on control 10, the electronic device can display service control 1502. Service control 1502 is used to extract user information from the card and save it to the user information database. Optionally, as an example, when the displayed interface includes an address, the electronic device can display service controls that provide operations such as navigation, hailing a taxi, viewing an address, viewing a distance, and adding items to favorites. As another example, when the page currently displayed by the electronic device includes text content, and the text content includes a language other than the native language, the electronic device can also display service controls that provide translation. As another example, when the interface displayed by the electronic device includes image content, and the image content includes products, the electronic device can display service controls that provide shopping and adding items to favorites; when the interface displayed by the electronic device includes a face image, the electronic device can display service controls for face-swapping detection. The examples listed here are merely examples, and this solution does not specifically display the service controls that the electronic device can display.

[0071] S2. On the first interface, display at least one form-filling control and the corresponding label information for the form-filling control.

[0072] After executing step S1 and displaying the function control that triggers automatic form filling, as mentioned earlier, since the electronic device may have already saved user information for multiple users, the user can select the user information to be filled in before the electronic device automatically fills in the user information into the form filling interface. Based on this, in the embodiment provided in this solution, the electronic device can display at least one form filling control and corresponding label information. The form filling control is used to trigger the electronic device to fill in user information in the form filling interface, and the label information indicates the user information to be filled into the corresponding filling box when the form filling control is triggered.

[0073] For details, please refer to [link / reference]. Figure 4A Steps S105-S108 are the process of displaying at least one form-filling control and the label information corresponding to the form-filling control in this solution.

[0074] S105 receives user operations on the first interface.

[0075] In one specific implementation, the user's operation on the first interface in step S105 can be an input operation on the first functional control in the first interface. Specifically, such as... Figure 2A As shown in (c), when the form filling interface 100 includes a first service control 105, the electronic device can receive input operations on the first service control 105. These operations may include long-press operations, click operations, etc., and this application does not limit the type of operation.

[0076] In another specific embodiment, the user's operation on the first interface in step S105 can be an operation on the fill box on the first interface. Specifically, such as... Figure 2A As shown in (a), the electronic device can receive user input on any of the unfilled fill boxes 11, 12, and 13. Alternatively, as... Figure 2B As shown in (a), when all the fields in the form filling interface have been filled with information, if the user wants to modify the information that has been filled, the electronic device can also receive the user's input on any of the fields that have been filled in, such as field 11', field 12', and field 13'.

[0077] In another embodiment provided by this solution, in response to the operation of the first control, the electronic device may not display the function control that triggers automatic form filling, but directly display the form filling control and the label information corresponding to the form filling control. That is, steps S104-S105 are optional steps, and the electronic device may directly execute step S106 after executing step S103.

[0078] S106, Detect the input box and the corresponding text information based on the screenshot.

[0079] In one specific implementation, after the electronic device receives the user's operation on the first interface, the visual detection module of the electronic device detects the screenshot of the first interface obtained in step S102 to identify the fill boxes and the corresponding text information in the screenshot, thereby avoiding the increase in power consumption caused by the electronic device repeatedly taking screenshots. In this implementation, although the electronic device does not take a new screenshot, for ease of description, the screenshot detected in step S106 is referred to as the "second screenshot".

[0080] In another specific implementation, considering that the content of the interface displayed by the electronic device may change after step S102 is executed, therefore, after the electronic device receives the user's operation on the first interface, executing step S106, which detects the fill box and the text information corresponding to the fill box based on the screenshot, may include: executing as follows Figure 5 The steps S1060-S1063 are shown. Specifically:

[0081] S1060, in response to the user's operation on the first interface, takes a screenshot of the first interface and obtains a second screenshot.

[0082] In response to the user's operation on the first interface in step S105, the electronic device takes a screenshot of the first interface (form filling interface 100) as the screenshot detected in step S106 (the second screenshot). Similarly, the second screenshot may not include elements such as the status bar and the first control. Optionally, the second screenshot may not be displayed by the electronic device and / or saved to the electronic device's photo library.

[0083] S1061, Perform the first detection on the second screenshot to identify the text boxes and text information in the second screenshot.

[0084] After obtaining the second screenshot, the electronic device can use a model that can detect both the input box and the text within it to detect the second screenshot and thus determine the input box and the corresponding text information.

[0085] In one specific implementation, see Figure 5 As shown, after acquiring the second screenshot, the electronic device performs a first detection on the second screenshot to detect text information and text boxes. This first detection can utilize Optical Character Recognition (OCR) technology to perform optical character recognition on the second screenshot, identifying the text information and the corresponding text box for each piece of text. For example... Figure 5As shown, by detecting the second screenshot, the electronic device can identify text information 1000 "Name", text information 1002 "Phone Number", and text information 1004 "Detailed Address", as well as the text boxes 1001 (corresponding to text information 1000), 1003 (corresponding to text information 1002), and 1005 (corresponding to text information 1004). The center point of text box 1001 is O1(XO1, YO1), the center point of text box 1003 is O2(XO2, YO2), and the center point of text box 1005 is O3(XO3, YO3).

[0086] After executing step S1061, or while executing step S1061, the electronic device may execute step S1062.

[0087] S1062, perform a second detection on the second screenshot to identify the fill box and fill box type in the second screenshot.

[0088] In one specific implementation, see further. Figure 5 As shown, the electronic device can perform a second detection of fill boxes on the second screenshot. The electronic device can utilize a visual detection model, or a model with frame recognition capabilities, such as the YOLO (You Only Look Once) model, to determine the fill boxes and their categories in the second screenshot. Specifically, as an optional implementation, the output of the visual detection model may also include: Where N is the total number of detected boxes, x i ,y i w represents the center coordinates of the i-th fill box. i ,h i This indicates the width and height of the input box; s i This indicates the category of the fill-in box. The categories include: 1. Filled dropdown, 2. Unfilled dropdown, 3. Filled fill-in box, 4. Unfilled fill-in box, 5. Filled fill-in box, 6. Unfilled fill-in box, 7. Table label box, etc. Based on the visual detection model output, the coordinates (XA, YA), (XB, YB), (XC, YC), and (XD, YD) of the four vertices ABCD of the unfilled fill-in box 1006 in the second screenshot can be obtained. Similarly, the electronic device can obtain the coordinates (XA1, YA1), (XB1, YB1), (XC1, YC1), and (XD1, YD1) of the four vertices A1B1C1D1 of fill-in box 1007, and the coordinates (XA2, YA2), (XB2, YB2), (XC2, YC2), and (XD2, YD2) of the four vertices A2B2C2D2 of fill-in box 1007.

[0089] Optionally, after performing steps S1061-S1062, in order to improve data processing efficiency, the electronic device may also perform step S1063.

[0090] S1063, Match the text box with the fill box, and match the text information with the fill box.

[0091] In one example, as an implementation, the electronic device can use a center point matching algorithm to match the text box and the fill box. For example, the electronic device can determine whether the coordinates of the center point O1 of the text box and the coordinate range of the table frame of the fill box 1006 satisfy one of the following conditions based on the coordinates (XA, YA) of the top-left vertex A and the coordinates (XD, YD) of the bottom-right vertex D of the fill box 1006: (XA≤XO1)&(XD≥XO1)&(YA≥YO1)&(YD≤YO1), or (XA≤XO1≤XD)&(YA≥YO1≥YD). As a matching representation method, if the electronic device determines that the coordinates of the center point O1 of the text box and the coordinate range of the table frame of the fill box 1006 satisfy the above conditions, it determines that the text box 1001 matches the table frame 1006, and then forms a first key-value pair with the text information 1000 "Name" corresponding to the text box 1001 and the fill box 1006. In this context, the text information 1000 "Name" is used as the key (K), and the identification information in the fill box 1006 is used as the value (V).

[0092] If the electronic device determines that the coordinates of the center point O1 of text box 1001 and the coordinate range of the table frame of fill box 1006 do not meet the above conditions, then it determines that text box 1001 and fill box 1006 do not match, and continues to check the matching status of text box 1001 with other fill boxes until a match is successful or all fill boxes have been traversed. It should be noted that the above example uses the top-left and bottom-right vertices of the text outer box as factors in calculating the table frame coordinate range. In other cases, the bottom-left and top-right vertices of the text outer box can also be used as the calculated coordinate values ​​of the table frame, or all four vertices of each text outer box can be used as factors in calculating the table frame coordinate range; there is no limitation.

[0093] In another specific implementation, the electronic device may employ a matching algorithm between table label text information and fill boxes based on two-dimensional Gaussian distribution modeling.

[0094] The electronic device has a pre-stored two-dimensional Gaussian distribution model p(d) ij ),

[0095]

[0096] Where μ represents the mean, ε represents the covariance matrix, exp represents the natural exponential function, and T represents the transpose matrix. The Gaussian distribution model is obtained by statistically analyzing the relative positional distribution of the center coordinates of text boxes and fill-in boxes in a large number of table samples. For each fill-in box I in the electronic device... j Calculate the center distance from other text boxes, and find the text T whose center distance is within a predefined first threshold r. i Forming candidate set T j T j =T i ‖‖d ij |≤r. For example, the first threshold r can be less than or equal to 100 pixels. Then, the electronic device selects each candidate text box T within the candidate set. i Calculate the probability I that it matches the fill box. j The calculation formula is:

[0097]

[0098] The electronic device obtains the probability of each candidate text box relative to the input box in the candidate set, and selects the text box T with the highest probability. i * As fill box I j The paired text boxes, where, For example, electronic devices can determine that the text box with the highest probability of filling in box 1006 is 1001 by constructing a Gaussian model. They then match box 1006 with text box 1001, achieving a result with less computation and higher accuracy. Electronic devices can also use other methods to detect fill boxes and their corresponding text boxes.

[0099] After the electronic device can match each input field and text field according to the above method, it can obtain the first key-value pair consisting of text information 1000 "Name" and input field 1006, the second key-value pair consisting of text information 1002 "Telephone" and input field 1007, and the third key-value pair consisting of text information 1004 "Detailed Address" and input field 1008.

[0100] S107, Determine the matching user information in the user information database based on the fill box and the text information corresponding to the fill box.

[0101] After the electronic device executes step S106 to obtain the fill box and its corresponding text information, it can query the user information database based on the fill box and its corresponding text information to determine at least one user information that matches the text information. As mentioned above, the electronic device has a user information database that stores user information. The user information can come from user input, recognition of ID photo information in the user's image library, information from forms filled out by the user, SMS messages, etc. The specific storage process of user information is detailed below.

[0102] In one example, as an implementation method, since the name can uniquely associate the information subject to be filled in, when the electronic device performs step S107, it can use the name as an identifier to view the user information stored in the user information database on the personal information interface.

[0103] Specifically, for example, after executing step S106, the electronic device detects the input boxes and their corresponding text information based on the screenshot, it obtains a first key-value pair consisting of text information 1000 "Name" and input box 1006, a second key-value pair consisting of text information 1002 "Phone Number" and input box 1007, and a third key-value pair consisting of text information 1004 "Name" and input box 1008. The electronic device determines matching information in the user information database based on the keys (i.e., text information) in these key-value pairs. For example, for the first key-value pair, the electronic device will determine text information 1000 "Name" as the (key, K) key to determine matching user information in the user information database. Similarly, the electronic device can determine the matching information for text information 1002 "Phone Number" and the matching information for text information 1004 "Detailed Address". This solution can use existing retrieval and matching methods to determine information matching the keys in the user information database.

[0104] The rules for determining matching information in the user information database will be further explained below.

[0105] In some embodiments, when there is only one user information in the user information database that matches the text, the electronic device's user information database will return the uniquely matched user information to the display module for use in subsequent automatic form filling steps. For example, when the user information database only contains the information "Li Si" that matches the text information 1000 "Name", the user information database will send the information "Li Si" as the matching information to the text information 1000 "Name" to the display module for display as label information of the form filling control.

[0106] In some embodiments, when there are multiple user information entries matching the tag in the user information database, the electronic device can determine the user information matching the fill-in box based on the order of information update time, the number of information entries, or the number of historical entries. Specifically, for example, the electronic device can sort multiple candidate user information entries from most recent to oldest based on their update time, resulting in a candidate information list. For example, the candidate user information entries are user A "Wang Wu", user B "Li Si", and user C "Zhang San", with user A's information update time being T1, user B's being T2, and user C's being T3. T1 is earlier than T2, and T2 is earlier than T3. Therefore, the electronic device can place user C "Zhang San" first in the candidate user information list, user B "Li Si" second, and user A "Wang Wu" third. In other words, the candidate user information list is arranged in the order of "Zhang San", "Li Si", and "Wang Wu".

[0107] In another implementation, the electronic device can count the number of times user information is entered. It can then sort multiple candidate user information entries in descending order of the number of entries, resulting in a candidate information list. For example, the same user "Wang Wu" has multiple addresses. The address "Province A, City B, District C, Street D" has been entered 10 times, the address "Province E, City F, District G, Street H" has been queried 5 times, and the address "Province I, City J, District K, Street L" has been queried once. Therefore, the electronic device can place "Province A, City B, District C, Street D" first in the candidate address information list for user "Wang Wu," "Province E, City F, District G, Street H" second, and "Province I, City J, District K, Street L" third. In other words, the order used in the candidate address information list for "Wang Wu" is "Province A, City B, District C, Street D," "Province E, City F, District G, Street H," and "Province I, City J, District K, Street L."

[0108] In another implementation, the electronic device can sort multiple candidate user information based on the source of the user information. For example, the accuracy of information filled in by the user on a form can be considered greater than the accuracy of information identified from the user's identification document. For instance, user "Wang Wu" has multiple phone numbers. The first number, "1234567890," comes from information previously filled in by the user on a form, while the second number, "0010-123456," comes from identification of the user's identification document. The electronic device can place "1234567890" first in the candidate list of user "Wang Wu's" phone numbers and "0010-123456" second. That is, the order of "Wang Wu's" candidate phone numbers is "1234567890" followed by "0010-123456."

[0109] Therefore, electronic devices can determine the sorting of different users and the sorting of user information in each different dimension according to the above process. For example, the sorting order used to form the candidate list of user information can be "Zhang San", "Li Si", and "Wang Wu". Among them, the sorting order of the candidate list of phone numbers for "Zhang San" is "1987654320"; the sorting order of the candidate list of phone numbers for "Li Si" is "1666654321" and "1322254321"; the sorting order of the candidate list of phone numbers for "Wang Wu" is "1234567890" and "0010-123456"; the information in other dimensions is similar and will not be elaborated here.

[0110] Furthermore, optionally, in some embodiments, when there are multiple user information entries matching the tags in the user information database, the different information entries for different users can be sorted first by name, and then sorted by different information entries for the same name. The electronic device can sort the different user entries first, and then sort the multiple matching entries for the same user. This application can use other methods to determine the matching user information in the user information database, and no limitation is made here. Based on this, the electronic device determines the matching user information from the user information database according to the fill-in box and the tag corresponding to the fill-in box, providing a foundation for accurate form filling subsequently.

[0111] It should be understood that if the electronic device fails to execute step S107 after executing step 106, it is considered that the electronic device has not saved user information that can be filled in. At this time, the electronic device can display a prompt message to inform the user that the electronic device cannot provide an automatic form filling function and the user needs to switch to manual form filling.

[0112] S108, based on user information, display the form filling control.

[0113] After step S107, the electronic device can display the form-filling controls. Specifically, when there is one user information matching the fill-in box in the user information database, the electronic device can display one form-filling control and display the user information as a label on the control within the form. When there are multiple user information matching the fill-in box in the user information database, the electronic device can display multiple form-filling controls, each displaying its corresponding label information, allowing the user to distinguish between the controls using different labels. As an example, since "name" can distinguish different users, taking three user information entries returned from the user information database as an example, if the candidate list determined in step S107 uses the names "Zhang San," "Li Si," and "Wang Wu," then... Figure 2AAs shown in (d), the electronic device can simultaneously display three different form-filling controls. Optionally, the form-filling controls are also displayed from left to right according to the order used in the information candidate list: form-filling control 183 and its label information "Zhang San", form-filling control 182 and its label information "Li Si", and form-filling control 181 and its label information "Wang Wu". Additionally, the label information of the form-filling controls can optionally include surname abbreviations; for example, the label information of form-filling control 181 also includes "Wang", the label information of form-filling control 182 also includes "Li", and the label information of form-filling control 183 also includes "Zhang". It should be understood that the electronic device can also use different icons, colors, textures, shapes, animations, and other features to distinguish the form-filling controls. For ease of description, this type of control with label information prompting the content to be filled is called a "form-filling control". The function of the form-filling controls will be described in detail later, and will not be repeated here.

[0114] Optionally, form-filling controls are displayed above the keyboard. For example, Figure 2A As shown in (d) of the diagram, keyboard 184 is the system keyboard of the electronic device. Form filling controls 181, 182, and 183 are all displayed in the area above keyboard 184. Or as... Figure 3 As shown in (a) in the figure, Figure 3 The keyboard 80 shown in (a) is a third-party keyboard, and form filling controls 81, 82, and 83 are all displayed above the keyboard 80. The keyboard can be the system keyboard of an electronic device or a third-party keyboard provided by an app, mini-program, or webpage.

[0115] As an example, if the keyboard displayed on the electronic device is the system keyboard, the device can obtain the display area of ​​the system keyboard by calling a system interface, such as a third-party system interface. Then, the device can draw form-filling controls in a specific area above or below the system keyboard's display area to ensure that candidate options are displayed outside the system keyboard's display area. However, if the keyboard displayed on the electronic device is a third-party keyboard, the device may not be able to obtain the third-party keyboard's display area by calling a system interface. Therefore, depending on the keyboard type of the form-filling interface, such as... Figure 6 As shown, the electronic device may include performing the following steps when executing step S108:

[0116] S1081, Determine if the system keyboard exists on the first screen.

[0117] Electronic devices can detect whether the first interface includes the system keyboard, either by calling a system interface, specifically the `AndroidReflectUtil.getInputMethodHeight` interface (hereinafter referred to as the first interface). If the phone obtains the system keyboard information by calling the first interface, it means that the first interface includes the system keyboard. Conversely, if the phone does not obtain the system keyboard information by calling the first interface, it means that the first interface does not include the system keyboard.

[0118] If the result of step S1081 is yes, then step S1082 is executed. If the result of step S1081 is no, then step S1083 is executed.

[0119] S1082, call the system interface to obtain the system keyboard display area.

[0120] Specifically, the electronic device can access the display area of ​​the system keyboard by calling an interface. Then, step S1804 is executed.

[0121] S1083, based on the screenshot of the first interface, detect the display area of ​​the keyboard.

[0122] When the first interface does not include the system keyboard, the electronic device can acquire a screenshot of the first interface and use a keyboard detection model to identify the keyboard display area. The screenshot of the first interface can be the one taken by the electronic device in step S102, or it can be a screenshot obtained by the electronic device from a new screenshot. Taking the third screenshot P as the input to the keyboard detection model as an example, the keyboard detection model M... keyboard The result of recognizing the third screenshot P can be expressed as:

[0123]

[0124] Where, x j and y j Let w represent the x and y coordinates of the center of the j-th button, respectively. j and h j c represents the width and height of the j-th button frame, respectively. j This represents the key press content of the j-th key. M is the total number of keys detected on the keyboard.

[0125] Specifically, this can include: electronic devices using keyboard detection models to detect key information on the keyboard, such as the key position and key content, and then analyzing and determining the keyboard's display area based on the key positions. Specifically, the electronic device can determine the top position of the keyboard based on the key positions closest to the top of the page. Similarly, it can determine the bottom position of the keyboard based on the key positions closest to the bottom of the page. Then, the electronic device can determine the keyboard's display area based on the top and bottom positions of a third-party keyboard.

[0126] Alternatively, electronic devices can determine the top position of the keyboard simply by analyzing the key positions of each key in the results output by the visual model.

[0127] To identify the keyboard using the first visual model M keyboard For example, determining the position of the display area of ​​the disk can include the following steps 1-2. The coordinates used in these steps are illustrated with the top-left corner of the screen as the origin (0, 0).

[0128] Step 1: Collect the set of Y-coordinates of the key presses:

[0129]

[0130] Among them, y j This represents the y-coordinate of the j-th key. M is the total number of keys detected on the keyboard.

[0131] Step 2: Determine the keyboard display area based on Gaussian distribution modeling. Specifically, this includes the following process:

[0132] Step 2.1: Calculate the mean and standard deviation:

[0133]

[0134] Where μ represents the mean of the vertical axis of the button, and σ represents the standard deviation of the vertical axis of the button.

[0135] Step 2.2, Remove outliers: Set a threshold k, and remove outliers that meet the following conditions. j Treat them as outliers and remove them:

[0136] |y j -μ|>kσ

[0137] The threshold k can be set according to the actual situation; for example, k = 2 or k = 3. Afterwards, the phone can obtain the set of button y-coordinates Y` after removing outliers, where |Y`| = M`.

[0138] Step 2.3: Determine the top position of the keyboard:

[0139] Y top =minY`

[0140] The electronic device determines the top position of the keyboard by using the minimum absolute value of y for each key according to the above formula. Similarly, the bottom position of the keyboard can be determined by using the maximum absolute value of y for each key.

[0141] After determining the top and / or bottom position of the keyboard, the keyboard display area can be determined.

[0142] S1084, Display form-filling controls according to the keyboard display area.

[0143] Once the keyboard display area is determined, the form-filling controls can be displayed outside of that area, such as at the top or bottom. Specifically, the form-filling controls can be displayed at position 'd' within the keyboard's display area. The position of the form-filling controls is Y. hint It can be represented as: Y hint =Y top -d.

[0144] For example, such as Figure 2A As shown in (d), the electronic device is located at the top position 181 of the keyboard 184. Displaying the form-filling controls at the top or bottom of the keyboard avoids conflicts between the keyboard and the form-filling controls.

[0145] Optionally, while displaying the form-filling controls on the electronic device, the device can also simulate clicking the "Name" field and display a cursor prompt in the Name field to begin automatically filling in the initial field. For example, as... Figure 2A As shown in (d), while the electronic device displays the form filling controls 181 to 183 in form filling 1, it also displays the cursor 18 in the filling area 16 of the filling box 11 with the label information 15 as "Name", to prompt the user that the electronic device will automatically fill in the form starting from the filling box 11.

[0146] After receiving the operation on the form filling control, S3 fills in at least one of the fill-in boxes on the first interface.

[0147] After displaying the form-filling controls, the electronic device can automatically fill in at least one of the fields in the form-filling interface after receiving the user's input of the form-filling controls.

[0148] One implementation method involves an electronic device automatically filling in multiple fields in a form after receiving user input from the form's input controls; this is known as "one-click fill." It's important to note that "one-click fill" here refers to a scheme where the electronic device automatically fills in the corresponding text in multiple fields. Other names may exist, such as "one-click full fill," without limitation. The following will primarily focus on... Figure 4B Steps S2001-S2008 introduce the automatic filling scheme for the "one-click fill" mode form provided in this solution.

[0149] S2001: Receives the operation that triggers one-click form filling.

[0150] In one implementation, the user triggers the one-click form filling operation by clicking on the form filling control displayed on the electronic device. Specifically, the electronic device can complete the form filling in a scenario where the system keyboard is displayed. Optionally, long press, swipe, or other methods can also be used to input data into the form filling control. This application embodiment does not limit the method of inputting data into the form filling control.

[0151] As an example, electronic devices can automatically fill out forms through methods such as injection or simulated keyboard clicks. When the input keyboard on the form interface is the system keyboard, the electronic device can use injection to automatically fill out the form. The injection method can be referenced from existing technologies that call system interfaces, and will not be elaborated here. When the input keyboard is a third-party keyboard, injection may fail to fill out the form; therefore, the electronic device can use simulated clicks to automatically fill out the form. For example... Figure 2A As shown in (d), when a user wants to fill in information for "Wang Wu" in the form filling interface, they can select form filling control 181.

[0152] See also Figure 4B Steps S2002-S2008 are the process of automatically filling in the multiple fields in the first interface after receiving the operation on the form filling control.

[0153] S2002, determine whether the system keyboard exists on the first screen.

[0154] After receiving the user's one-click form filling operation, as mentioned above, the electronic device can detect whether the first interface includes the system keyboard. The detection of the system keyboard can also be achieved through the aforementioned first interface (AndroidReflectUtil.getInputMethodHeight).

[0155] Optionally, as one implementation, if the judgment result of step S2002 is yes, that is, a system keyboard exists, then the electronic device executes step S2003. If the judgment result of step S2002 is no, that is, a system keyboard does not exist. Since third-party keyboards typically cannot inject user information through the system interface, the electronic device executes step S2004 and then executes step S2006.

[0156] Optionally, as another implementation, if the judgment result of step S2002 is negative, the electronic device can also first attempt to inject user information through the system interface, i.e., execute steps S2003-S2007. Optionally, before executing steps S2003-S2007, the electronic device can also determine whether the first interface belongs to the whitelist. Specifically, the electronic device can pre-collect and store some common activities that do not support text injection. If it is determined from the activity that the table on the first page belongs to the preset whitelist, it is assumed that the first input box may not support the injection input method. Then, steps S2003-S2007 can be executed to try whether it can be executed successfully.

[0157] S2003, Call the system interface and fill in the information in the input box.

[0158] Specifically, the electronic device can fill in the information to be filled in the form box by calling the second interface. The second interface can be the InputMethodManager class interface that supports text injection; for details, please refer to the existing technology description of text injection. Specifically, in response to the user's input on the form control 181, the electronic device calls the system interface to fill in the form based on the label information "Wang Wu" of the form control 181 and the matching "Wang Wu" user information queried in step S107, which includes the phone number "1234567890" and the detailed address "Province A, City B, District C, Street D". For example... Figure 2B As shown in (a), the name field 11' in the form filling interface 100 is filled with the name "Wang Wu", the phone number field 12' is filled with the phone number "1234567890", and the address field 13' is filled with the information "Province A, City B, District C, Street D". Optionally, the electronic device can also display the information 20 "Information queried on the current interface has been filled" on the form filling interface 100 to prompt the user that the information queried on the current interface has been filled. Based on this, low-power form filling can be achieved through injection.

[0159] After completing step S2003, optionally, as one implementation, the electronic device may end the filling process. Alternatively, to determine whether the text injection was successful, steps S2004-S2005 are executed after step S2003 is completed by the electronic device.

[0160] S2004, screenshot of the first interface.

[0161] Specifically, the electronic device can take a screenshot of the first interface to obtain screenshot C (the fourth screenshot), which is used for detection and analysis. Optionally, screenshot C may only include the keyboard area.

[0162] S2005, check whether the input box was filled in successfully based on the screenshot.

[0163] Specifically, the electronic device can use the aforementioned visual detection model to detect fill boxes in the fourth screenshot. In the output of the visual model, if the fill box category is "4 Unfilled Fill Box," it indicates that filling the fill box failed, i.e., the judgment result of S2005 is negative, and the electronic device executes S2006-S2007. If the fill box category is "3 Filled Fill Box," i.e., the judgment result of S2005 is positive, the electronic device can end the filling process.

[0164] S2006, detects keyboard key information.

[0165] Specifically, if the judgment result of S2005 is negative, it is considered that the filling in of the input box by injection has failed. Or, if it is determined in S2002 that the system keyboard does not exist, the electronic device can obtain a screenshot of the first interface obtained in step S2004, and use the keyboard detection model to detect the key positions and key contents of each key on the keyboard. Let the screenshot be the fourth screenshot K, then the first visual model M... keyboard The result of recognizing the fourth screenshot K can be expressed as:

[0166]

[0167] Where, x j and y j Let w represent the x and y coordinates of the center of the j-th button, respectively. j and h j c represents the width and height of the j-th button frame, respectively. j This represents the key press content of the j-th key. M is the total number of keys detected on the keyboard.

[0168] Optionally, as another implementation, since the keyboard detection model has already identified the key information of the keyboard in step S1083, the key information detected in step S1083 can be directly reused in step S2006.

[0169] S2007, based on button information and user information, performs simulated clicks to fill in user information.

[0170] In one implementation, the electronic device can simulate clicking and filling in multiple input boxes on the first interface at once. Specifically, for example, in... Figure 3 In example (a), the electronic device displays a form filling interface 800, which includes a "Name" fill-in box 84, an "Email" fill-in box 85, a "Student ID" fill-in box 86, a form filling control 81 labeled "Wang Wu", a form filling control 82 labeled "Li Si", and a form filling control 83 labeled "Zhang San", as well as a three-way keyboard 80. In response to the user's input on the form filling control 81, the electronic device, combined with the detected key information from the three-way keyboard obtained in step S2006, determines the key coordinates of the corresponding key on the keyboard 80 for the content to be filled. Then, the electronic device performs simulated click operations on the corresponding key coordinates of "w", "u", "a", "n", "g", "w", and "u" on the keyboard 401 in the character order of the content to be filled, such as "Wang Wu", to fill in the content "Wang Wu" into the fill-in box, resulting in fill-in box 84'. Similarly, the electronic device can complete the filling of fill boxes 85 and 86 in the order of the characters "wu@123.com" and "1015205208".

[0171] S2008, simulates clicking the next fill box.

[0172] After the electronic device has filled in the first input box, it can simulate clicking the next unfilled input box by using the coordinates of the input box belonging to "4 Unfilled Input Box" detected in step S2005, so as to switch the input object.

[0173] In one implementation, after the electronic device simulates clicking on the fill-in area, the layout of the form fill-in interface may change, causing the coordinates of the form buttons to change. Continuing with... Figure 3 The example of a three-way keyboard shown in (a) allows a user to select form-filling control 81 when they wish to enter information about "Wang Wu" into a form. In response to user input on form-filling control 81, such as... Figure 3 As shown in (b), the electronic device enters the name "Wang Wu" in the "Name" field 84' by simulating a click, and then displays the keyboard 80'. Because in Figure 3 In (b) of the code, the form-filling controls are not displayed, and the 80' key is positioned higher than the keypad. Figure 3 Keyboard 80 in (a) of the text. That is, in Figure 3 In (a), fill in the previous keypad 80 and in the... Figure 3 In (b), the position of keyboard 80' after filling in the form is different.

[0174] For another example, in another three - party keyboard scenario, when the electronic device performs automatic filling in the filling box based on the analog trigger method, the keyboard displayed on the page may also be updated, resulting in changes in button coordinates. For example Figure 3 In another three - party keyboard example in (c) of Figure 3 , the electronic device displays a form filling interface 801 for a license plate. The form filling interface 801 includes a form filling control 893, form filling areas 891 and 892, and a keyboard 87. Among them, the form filling control 893 includes label information "Jing E18881" to prompt the form information to be filled in. The form filling area 891 is a Chinese character filling area, the form filling area 892 is an alphabet and number filling area, and the keyboard 87 is a three - party keyboard provided by the application corresponding to the form filling interface 801. In response to the operation input by the user to the form filling control 893, after the electronic device executes steps S2001 - S2007, the electronic device inputs the province abbreviation "Jing" in the first filling box 891. After filling in the first filling box 891, the electronic device switches from displaying the three - party keyboard 87 to displaying the three - party keyboard 88. Since the page layout information may change after completing the filling of one filling box by simulating a click, before simulating a click on the keyboard to fill in each filling box, the electronic device can run a keyboard detection model to detect key information and obtain the correct key coordinates, thereby improving the accuracy of filling in by the electronic device through simulated clicks.

[0175] Specifically, after the first execution of S2008 and simulating a click on the next filling box, the electronic device can repeat steps S2002 - S2008 for each filling box until all filling boxes are filled. For example Figure 3 As shown in (b) of Figure 3 , the electronic device fills in the name "Wang Wu" in the "Name" filling box 84' by simulating a click, and then repeats steps S2002 - S2008 to fill in the phone number wu@123.com in the "Email" filling box 85'. Similarly, repeating steps S2002 - S2008 again fills in the number "1015205208" in the "Student ID" filling box 86'. Or Figure 3 As shown in (d) of Figure 3 , the electronic device fills in the Chinese character information "Jing" in the form filling area 891 by simulating a click, and then repeats steps S2002 - S2008 to complete filling in the alphabet and number "E18881" in the form filling area 892.

[0176] Based on this, this application provides a solution for automatically filling out forms using a form-filling control. In this solution, after the user selects a form-filling control via its label, the electronic device can automatically fill in multiple fields on the form-filling interface. Since the labels of the form-filling controls indicate the main information to be entered into the form, the user can perform a one-click form-filling operation by selecting the corresponding control, which not only saves the user from tedious operations but also meets the user's actual usage needs.

[0177] As another example, an electronic device can automatically fill in the text corresponding to a selected field based on the user's input through the form control; this is known as "select-and-fill" mode. It should be noted that "select-and-fill" here only refers to the scheme where the electronic device automatically fills in the text corresponding to the currently selected form control. In other cases, "select-and-fill" may have other names, such as form filling, sequential form filling, single-frame form filling, etc., without limitation.

[0178] The following combination Figure 7 This section introduces the "select and fill" mode.

[0179] Specifically, such as Figure 8 As shown, steps S3001-S3007 are the process of automatically filling in the input box selected by the user in the first interface after receiving the operation of the user selecting the input box.

[0180] S3001 receives user input from selecting fields.

[0181] In one alternative implementation, the action that triggers the filling in of the form can be the user selecting an action to fill in the form. For example... Figure 7 As shown in (a), in response to a user's input in the form filling box 12 of the form filling interface 100, the electronic device... Figure 7 As shown in (b), the form filling interface 100 displays a form filling control 53, which includes the label information "Zhang 1987654320" to indicate the information to be entered in the fill box 12. Optionally, the form filling interface 500 may also display other form filling controls, such as form filling control 54, which includes part of the label information "Li 1889...". When the user wants to enter "1987654320" in the fill box 11, they can select form filling control 53.

[0182] After the electronic device executes step S3001 to receive the user's selection of the fill control, it can execute steps S3002-S3007 to fill in the form fill box 12. Steps S2002-S2007 can be referred to above and will not be repeated here.

[0183] After completing steps S3002-S3007, as follows Figure 7 As shown in (c), “1987654320” was entered into the form filling interface by the electronic device, resulting in the “Telephone” field 12’ being filled in, while the “Name” field 11 and the “Detailed Address” field 13 remained unfilled.

[0184] As can be seen from the above solutions, the one-click fill mode eliminates the need for users to select each fill box individually, while the select-and-fill mode allows users to manually select the information to be filled in. Furthermore, this application embodiment also provides another solution for automatically filling out forms. The electronic device determines whether to enter one-click fill mode or select-and-fill mode based on the user's current selection.

[0185] Combination Figure 9 As shown, before performing steps S2002-S2008 or steps S3002-S3007, the following steps are also included:

[0186] S4001 receives user input from selecting fields.

[0187] Specifically, in one implementation, the electronic device determines the user-selected input box by detecting the cursor position. For example... Figure 7 In response to the user's operation on the fill box 12, the electronic device displays the cursor 52 in the fill area of ​​the fill box 12. The electronic device can perform feature extraction, extracting possible features corresponding to the cursor, including color features, shape features, texture features, etc., to obtain the pixel coordinates of the cursor 52, and then determine that the fill box selected by the user is the fill box 12. For example, the electronic device can determine the fill box selected by the user based on the fill box information obtained in step S106, where the coordinate range of the fill box includes the coordinate range of the cursor 52. Alternatively, the electronic device can also determine the fill box 12 selected by the user where the difference between the starting coordinates of the fill box and the coordinates of the cursor 52 is within a preset range.

[0188] In another example, the electronic device can also obtain the cursor position by extracting parameters associated with the cursor in the first interface, and then determine the first fill-in box to be filled in on the first interface based on the cursor position. For example, for input elements and textarea elements, the electronic device can use the selectionStart and selectionEnd properties to obtain the cursor position. selectionStart returns the starting position of the cursor in the element, while selectionEnd returns the ending position of the cursor in the element. In specific implementations, the electronic device can also determine the fill-in box selected by the user through other methods.

[0189] S4002, determine the filling mode based on the filling box selected by the user.

[0190] If the filling mode is determined to be one-click filling, then execute steps S2002-S2008; if the filling mode is determined to be side-by-side filling, then execute steps S3002-S3007. See above for details.

[0191] The filling mode can be determined based on the label information of the selected fill box. For example, since the name is a key element associated with the information to be filled in, and electronic devices can use names as identifiers to store user information in a user database, if the label information of the user's fill box belongs to the "Name" category, such as "First Name" or "Personal Name," then the device enters the "One-Click Fill" mode. In this mode, the electronic device automatically fills in multiple user information fields in response to the user's interaction with the fill control. If the label information of the user's fill box does not belong to the "Name" category, then the device enters the "Select and Fill" mode. In this mode, the electronic device fills in user information from the selected fill box in response to the user's interaction with the fill control.

[0192] Optionally, the electronic device can compare the label of the input box determined in step S106 with "Name" to determine if they are similar. One comparison method is that the electronic device has a pre-stored list of synonyms or near-synonyms for "Name," and can determine whether the label of the input box is in the list to determine if it is associated with "Name." Another optional comparison method is that the electronic device converts the label of the input box and "Name" into vectors, and determines whether the label of the input box is associated with "Name" by comparing the vector similarity. Optionally, since "Phone number" can be associated with the information subject, when the label information of the input box selected by the user is associated with "Phone number," such as "Mobile number," "Contact information," etc., it can also be determined to enter the one-click fill mode. If the label information of the input box selected by the user is not associated with "Phone number" and / or "Name," it is determined to enter the "select and fill" mode.

[0193] In another implementation, if a user needs to modify the information already entered, they can select the content already entered in the input box to make changes. This application embodiment provides a method for modifying entered information in a form input interface.

[0194] S4. Receive input from any input field and display the form filling control.

[0195] As an example, after the electronic device performs step S3 and fills in at least one input box on the first interface, if the user needs to modify an already filled input box, they can operate on the input box to be modified. For example, as... Figure 2BAs shown in (a), the user's selection of fill box 11' triggers the electronic device to execute steps S105-S108. That is, the electronic device takes another screenshot of the first interface to obtain a fifth screenshot. By detecting the text box and text information in the fifth screenshot, the text box is matched with the fill box, and the text information is matched with the fill box. Based on the fill box and its corresponding text information, the matching user information is determined in the user information database, and the form-filling control is displayed according to the user information. The difference lies in this embodiment: when the electronic device executes S106, which detects the fill box and its corresponding text information based on the screenshot, it further includes: matching the user text information already filled in the fill box with the text information of the fill box and the form label.

[0196] For example, Figure 10 As shown, after obtaining the fifth screenshot, the electronic device performs a first detection on the fifth screenshot, obtaining recognizable text information 1000 "Name", text information 1002 "Phone Number", and text information 1004 "Detailed Address", as well as text boxes 1001, 1003, and 1005 corresponding to text information 1000, text box 1002, and text box 1005 corresponding to text information 1004; similarly, it can obtain text information 1200 "Wang Wu", text information 1202 "1234567890", and text information 1004 "Province A, City B, District C, Street D", as well as text boxes 1201, 1203, and 1205 corresponding to text information 1204. At the same time, we can obtain the center point of text box 1001 as O1, the center point of text box 1003 as O2, the center point of text box 1005 as O3, the center point of text box 1201 as O4, the center point of text box 1203 as O5, and the center point of text box 1205 as O6. As mentioned above, after the electronic device performs the second detection on the fifth screenshot, including the fill boxes 1206, 1207, and 1208 and their categories in the fifth screenshot, the electronic device can refer to the aforementioned matching method to match the user text information already filled in the fill boxes with the text information of the fill boxes and the table label, to obtain, for example, the fourth key-value pair consisting of the table label text information 1000 "Name" and the filled text information 1200 "Wang Wu" and the fill box 1006; the fifth key-value pair consisting of the table label text information 1002 "Telephone" and the filled text information 1202 "1234567890" and the fill box 1007; and the sixth key-value pair consisting of the table label text information 1004 "Name" and the filled text information 1204 "A Province B City C District D Street" and the fill box 1008.

[0197] After information matching is completed, when the electronic device executes S107 to determine the matching user information in the user information database based on the fill-in boxes and the corresponding text information, it may also include excluding already filled-in text information and determining the information candidate list. For example, when the original information candidate list is "Zhang San", "Li Si", and "Wang Wu", since the text information "Wang Wu" has already been filled in, the new information candidate list is "Zhang San" and "Li Si", and no longer includes "Wang Wu". As another example, when the original telephone information candidate list for "Wang Wu" is arranged in the order of "1234567890" and "0010-123456", since the text information "1234567890" has already been filled in, the new telephone information candidate list for "Wang Wu" is arranged in the order of "0010-123456". Similarly, the new address information candidate list for "Wang Wu" uses the order of "E Province, F City, G District, H Street" and "I Province, J City, K District, L Street".

[0198] Therefore, after the electronic device has determined the user information as described above, when it executes step S108 to display the form-filling controls, it will no longer display the form-filling controls corresponding to the already filled user information. For example... Figure 2B As shown in (b), in response to the user's selection of fill box 11, the electronic device displays fill control 182 and fill control 183 in the form filling interface 100. Since the user information "Wang Wu" has already been filled in in the form filling interface 101, fill control 181 labeled "Wang Wu" is not displayed. The fill controls are also displayed from left to right according to the sorting order used in the information candidate list: fill control 183 and its label information "Zhang San", and fill control 182 and its label information "Li Si".

[0199] S5. Receive input from the form fill control and modify the information already entered in the fill box.

[0200] Furthermore, users can simultaneously modify information entered in multiple fields by manipulating the form control. As one implementation method, for example, continuing to refer to... Figure 2B In (b) of the above, when the user selects to operate the form filling control 183, in response to the user's input on the form filling control 183, in an optional implementation, such as Figure 2B As shown in (d), the electronic device enters the name "Zhang San" in the name input box 11', the telephone number "1987654320" in the telephone number input box 12', and the information "Y Province, Z City, X District, W Street" in the address input box 13'. Specifically, the electronic device enters information in multiple input boxes in the manner described in step S3 above.

[0201] In another alternative implementation, in response to a user's input on the form-filling control 183, such as Figure 2BAs shown in (c), the electronic device displays window 21, and displays controls 22 and 23 within window 21. Control 22 includes a prompt to replace only the user-selected input box, and control 23 includes a prompt to replace all already filled boxes. Optionally, window 21 also includes a prompt for the user to replace the user information, such as "replace with 'Zhang San's' information." The user can modify the information in the already filled boxes by selecting different controls. Specifically, when the user wants to replace all and operates control 23, in response to the user's input on control 23, such as... Figure 2B As shown in (d), in the form filling interface 100, the electronic device replaces the name "Zhang San" in the name field 11', replaces the phone number "1987654320" in the "Phone" field 12', and replaces the information "Y Province, Z City, X District, W Street" in the "Detailed Address" field 13'. When the user wants to replace only the information already entered in field 11', they can manipulate control 22. Responding to the user's input on control 22, such as... Figure 2B As shown in (e), in the form filling interface 100, the electronic device enters the name "Zhang San" in the name filling box 11', and the information already entered in the telephone number filling box 12' and address filling box 13' remains displayed as "1234567890" and "Province A, City B, District C, Street D". Based on this, this application embodiment provides a method for users to choose to modify the user information already entered in multiple form filling boxes with one click or to modify only the user information already entered in the selected form filling box, which improves the convenience of automatically filling in forms and greatly enhances the user experience.

[0202] As an example, referring to the aforementioned S3 step, electronic devices can automatically fill in information through injection, simulated keyboard clicks, etc. When the input keyboard of the form filling interface is the system keyboard, the electronic device can automatically fill in information by calling an interface to select all the already filled information, deleting it, and then re-injecting it. Specifically, it can use `inputtext.model.KeyEventUtil#selectAll` to select all the already filled information, use the `InputMethodExecutor#cleanAndReceivedText` interface to delete the selected information, and then inject new information. Alternatively, it can use the `InputMethodExecutor#onRemoveContentBothSide` interface to delete the information, and then use the `InputMethodExecutor#onReceivedInputContent` interface to inject new information. The injection method can refer to the injection method of calling system interfaces in existing technologies, and will not be elaborated here. When the input keyboard is a third-party keyboard, the above-mentioned interface calling method can also be used to modify the already filled information. Optionally, for some third-party keyboards, the method of calling the interface may fail to fill in the information. Therefore, electronic devices can use simulated clicks to delete the already filled information and then automatically fill it in for some third-party keyboards that cannot be filled in by calling the interface. For details of the deletion and filling process, please refer to the steps in S3 above, which will not be repeated here.

[0203] Optionally, as one implementation, the electronic device determines whether to enter "one-click modification" mode or "select-and-modify" mode based on the currently selected filled-in field. In one optional implementation, when the user operates on the field associated with "name," the electronic device enters "one-click modification" mode, meaning the electronic device can modify information already entered in multiple fields. For example, ... Figure 2B As shown in (c), in response to the user's selection of the name field 11', the electronic device displays form-filling controls 182 and 183 in the form-filling interface 100. In response to the user's input on form-filling control 183, in one optional implementation, as... Figure 2B As shown in (d), the electronic device enters the name "Zhang San" in the name input box 11', the telephone number "1987654320" in the telephone number input box 12', and the information "Y Province, Z City, X District, W Street" in the address input box 13'. In another optional implementation, in response to the user's input on the form filling control 183, such as Figure 2B As shown in (c), the electronic device displays window 21 as described above, where the user can select how to modify the filled-in boxes. The specific implementation method is as described above and will not be repeated here.

[0204] As another example, when the user interacts with a field related to user information other than "name," the electronic device enters a "select-as-you-go" mode, meaning the device can modify the information already entered in the field. For details, please refer to [link / reference]. Figure 2B In (a), the user can interact with input fields associated with user information other than "Name," such as the "Phone" input field 12' or the "Detailed Address" input field 13'. For example, when the user interacts with the "Phone" input field 12', in response to the user's input in the "Phone" input field 12', the electronic device is triggered to perform actions such as... Figure 4A Steps S105-S108 are shown below, specifically as described above. After the electronic device executes steps S105-S108, as follows... Figure 11 As shown in (a), the electronic device displays form-filling controls 30 on the form-filling interface 100. In response to the user's selection of form-filling controls 30, the electronic device maintains the display of the name "Wang Wu" in the name field 11' and "Province A, City B, District C, Street D" in the "Detailed Address" field 13', while the phone number "1987654320" is entered in the phone number field 12'. Alternatively, when the user selects the "Detailed Address" field 13', as shown... Figure 11 As shown in (c), the electronic device displays a form filling control 32 with the label information "Wang Province, F City, G District, H Street" in the form filling interface 100, which is used to fill in the address information "E Province, F City, G District, H Street" into the filling box 13'.

[0205] Optionally, when a user modifies the information already entered in the form by interacting with the input box, the electronic device can display multiple form-filling controls. Figure 2B When filling in the "Telephone" field 12' as shown in (a) above, if Figure 11 As shown in (a), the electronic device displays both form-filling controls 31 and 32. Form-filling control 31 includes the identifier "Wang + E Province F City G District H Street". Due to limited display space on the electronic device, form-filling control 32 includes partial identifier "Wang + I Province J City K…". Optionally, the user can manipulate the display area of ​​the form-filling controls to allow the electronic device to display more form-filling controls and their identifier information. For example, as… Figure 11 As shown in (d), in response to the user's sliding operation of the form filling control 31, the electronic device displays the form filling control 31 including the complete identification information "I Province J City K District L Street". As another example, when the user operates... Figure 2B When filling in the "Detailed Address" field as shown in (a) 13', if... Figure 12As shown in (a), the electronic device displays a form-filling control 32, which includes the label information "Wang + E Province F City G District H Street". The electronic device can also simultaneously display control 61, allowing the user to manipulate control 61 to display more user information. For example, in response to user input on control 61, the electronic device displays window 63, which shows address control 64 and its label information "Wang I Province J City K District L Street". The user can replace the original information in the form-filling box 13' with its corresponding label information by manipulating address control 64. In response to user input on address control 64, such as... Figure 12 As shown in (c), the electronic device replaces the "Detailed Address" field 13' with the entered information "Province J City K District L Street". Additionally, the electronic device can display control 65 in window 63, prompting the user to manage personalized information. After receiving user input on control 65, the electronic device can display the relevant interface for managing personal information. Specifically, the interface for managing personal information can be found below. Figure 13 The description will not be repeated here.

[0206] Based on this, embodiments of this application provide a method for filling out form information. The electronic device responds to the user's touch operation on the form filling controls, triggering automatic form filling, and allows modification of the already filled form through touch operations on the form filling controls. This greatly reduces user interaction on the form filling interface, improving the user experience.

[0207] The information entered into the form in the above embodiments comes from information stored in the user's personal information database by the electronic device. Therefore, this application further provides a method for viewing and saving the user's personal information entered into the form.

[0208] Electronic devices can display a personal information interface for viewing saved user information. Users can view this information in a settings application, for example... Figure 13 As shown in (a), the electronic device can display a personal information page 700 in the settings application. Alternatively, the user can view previously saved personal information while filling out a form, in response to... Figure 12 When an operation is performed on control 65 as shown in Figure (b), the electronic device displays a personal information page 700. The user personal information interface 700 includes information about different users identified by their names. For example, the personal information interface 700 includes multiple names displayed in information bar 71, such as "Zhang San" 710, "Li Si" 711, "Wang Wu" 712, "Ke Liu" 713, and "Yu Qi" 714. Optionally, the personal information interface 700 may also include an information addition control 72.

[0209] Users can view more user information under the name displayed on the personal information page 700. For example, in response to a user's action on any name in the information bar 71, such as "Wang Wu" 712, such as... Figure 7 As shown in (b), the electronic device can display a personal information interface 701 under the name "Wang Wu". The personal information interface 701 includes at least one contact number for the name "Wang Wu", such as the first contact number 731 "1234567890" and the second number 732 "0010-123456". The personal information interface 701 may also include an email address 74 for the name "Wang Wu" "wu@123.com", and at least one detailed address, such as one or more of the following: the first detailed address 751: "District C, District D, City B, Province A", the second detailed address 752: "District G, District H, City F, Province E", and the third detailed address 753: "District K, District L, City J, Province I". It should be understood that the electronic device may store more or less personal information; the examples provided are only a few.

[0210] In one implementation, the user information in the aforementioned user information database can be pre-filled by the user in the information editing interface of the settings application. Specifically, upon detecting a user's information entry event, the electronic device can store the user-entered information as new user information in the user information database. For example, optionally, the personal information interface 700 may also include an add information control 72. In response to the user's operation on the add information control 72, the electronic device can add and save more personal information (not shown in the figure).

[0211] In another implementation, the electronic device can automatically collect the user's personal information and store it on the device side. See also... Figure 14The electronic device displays an automatic data collection interface 1400. This interface includes an automatic data collection switch 1401A, a switch 1402 for the SMS application, a switch 1403 for the Gallery application, and a switch 1404 for the Notes application. If the user enables the automatic data collection switch 1401A in the data collection settings interface, it means the user allows the electronic device to extract user information from the aforementioned applications. If the user does not enable the automatic data collection switch 1401A in the data collection settings interface, but clicks the switch 1402 for the SMS application in the 1400 settings interface, it means the user allows the phone to extract user information from the SMS application. The SMS messages saved in the SMS application can be received messages or edited messages by the user. Similarly, enabling the switch 1403 for the Gallery application means the electronic device can extract user information from images saved in the Gallery application. If enabling the switch 1404 for the Notes application, the electronic device can extract user information from notes saved in the Notes application.

[0212] In another implementation, the electronic device can proactively recommend that the user save user information and provide a quick method for retrieving and saving user information. For example, such as... Figure 15 As shown, exemplarily, the electronic device displays Figure 15 The image display interface 1500 shown in (a) includes an ID card image 1501, and a first control 10 is displayed at the bottom of the image display interface 1500. The ID card image 1501 includes user information such as "Name": "Wang Wu", "Gender": "Female", "Ethnicity": "Han", and "ID Number": "210106********6123". In response to the user's input to control 10, the electronic device executes steps S102-S104. Figure 15As shown in (b), after the electronic device executes steps S102-S104, it displays the second service control 1502 and can also display the first window 102. The specific functions of the first window 102 are described above. In response to the user's operation on the second service control 1502, the electronic device extracts the user information from the ID card image 1501 and saves it to the user information database. Optionally, specifically, the electronic device can extract user information from the image 1501, such as "Name": "Wang Wu", "Gender": "Female", "Ethnicity": "Han", and "ID Number": "210106********6123", using OCR to automatically fill in the table as shown in this embodiment. In one implementation, the electronic device needs to authenticate the user before extracting user information. Only if the authentication is successful (i.e., the user is the owner of the device) can the phone extract information from the displayed interface. The phone can authenticate the user through facial recognition, fingerprint recognition, or password input, etc., without specific limitations. If the user's identity is successfully authenticated, the mobile phone can save the user information in the ID card image 1501 to the user information database.

[0213] Based on this, this application provides a quick way to save users' personal information used for filling out forms, reducing the tedious process of manually entering information and improving the convenience of saving user information. The saving methods involved in the embodiments of this application are not limited to the methods illustrated in this application.

[0214] like Figure 16 The diagram shown is a structural schematic of an electronic device 1600 provided in an embodiment of this application. Figure 16 This is a schematic diagram of the hardware structure of the electronic device 1600 provided in this application embodiment, as shown below. Figure 16 As shown, the electronic device 1600 may include a processor 1610, an external memory interface 1620, an internal memory 1621, a universal serial bus (USB) interface 1630, a charging management module 1640, a power management module 1641, a battery 1642, antenna 1, antenna 2, a mobile communication module 1650, a wireless communication module 1660, an audio module 1670, a speaker 1670A, a receiver 1670B, a microphone 1670C, a headphone jack 1670D, a sensor 1680, a button 1690, a motor 1691, an indicator 1692, a camera 1693, a display screen 1694, and a subscriber identification module (SIM) card interface 1695, etc.

[0215] It is understood that the structures illustrated in the embodiments of this application do not constitute a specific limitation on the electronic device 1600. In other embodiments of this application, the electronic device 1600 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0216] Processor 1610 may include one or more processing units, such as an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural network processing unit (NPU). These different processing units may be independent devices or integrated into one or more processors.

[0217] In some embodiments, the electronic device 1600 may use the processor 1610 to perform the information entry method provided in this application.

[0218] The wireless communication function of electronic device 1600 can be implemented through antenna 1, antenna 2, mobile communication module 1650, wireless communication module 1660, modem processor and baseband processor, etc.

[0219] Electronic device 1600 implements display functions through a GPU, a display screen 1694, and an application processor. The GPU is a microprocessor for image processing, connecting the display screen 1694 and the application processor. The GPU performs mathematical and geometric calculations for graphics rendering. Processor 1610 may include one or more GPUs, which execute program instructions to generate or modify display information.

[0220] Display screen 1694 is used to display images, videos, etc. In some embodiments, electronic device 1600 can utilize display screen 1694 as... Figures 1 to 15 The diagram shows the relevant form filling interface, as well as display elements such as form filling controls and label information displayed in the form filling interface.

[0221] Electronic device 1600 can achieve shooting function through ISP, camera 1693, video codec, GPU, display 1694 and application processor.

[0222] Electronic device 1600 can implement audio functions, such as music playback and recording, through an audio module 1670, a speaker 1670A, a receiver 1670B, a microphone 1670C, a headphone jack 1670D, and an application processor. In some embodiments, electronic device 1600 can acquire voice commands input by the user through the microphone 1670C. Then, electronic device 1600 can determine the user's intent based on the voice command and display it on a screen such as... Figure 2A When window 102 (c) is in the middle, it receives the user's instructions and responds accordingly and / or automatically executes the tasks corresponding to the instructions.

[0223] Button 1690 may include a power button, volume buttons, etc. Button 1690 may be a mechanical button or a touch button. Electronic device 1600 can receive button input and generate key signal inputs related to user settings and function control of the electronic device. Motor 1691 can generate vibration feedback. Motor 1691 can provide vibration feedback during a long press operation of the first control 10 by the user. It can also be used for touch vibration feedback. Indicator 1692 may be an indicator light, which can be used to indicate charging status, power changes, messages, missed calls, notifications, etc. SIM card interface 1695 is used to connect a SIM card. The SIM card can be inserted into or removed from the SIM card interface 1695 to achieve contact and separation with the electronic device.

[0224] For example, the software system of the aforementioned electronic device 1600 can adopt a layered architecture, event-driven architecture, microkernel architecture, microservice architecture, or cloud architecture. This application embodiment uses a layered Android system as an example to illustrate the software structure of the electronic device 1600.

[0225] Other embodiments of this application provide an electronic device (such as a mobile phone). The electronic device may include a screen, a memory, and one or more processors. The memory is coupled to the processors. The memory is also used to store computer program code, which includes computer instructions. When the processor executes the computer instructions, the electronic device can perform various functions or steps performed by the mobile phone in the above method embodiments. The structure of the electronic device can be referred to... Figure 16 The structure of the electronic device 1600 shown.

[0226] An operating system runs on top of the aforementioned components. The operating system of the electronic device 1600 can adopt a layered architecture, event-driven architecture, microkernel architecture, microservice architecture, or cloud architecture. This application embodiment uses the layered architecture Android system as an example to illustrate the software structure of the electronic device 1600.

[0227] See Figure 17This is a software structure block diagram of the electronic device 1600 provided in this application embodiment. The layered architecture of the electronic device 1600 divides the software into several layers, each with a clear role and division of labor. The layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into four layers, from top to bottom: the application layer, the application framework layer, the Android Runtime and system libraries, and the kernel layer.

[0228] The application layer can include a series of application packages, such as Figure 17 As shown, the application package may include applications such as Gallery, SMS, Notes, Smart Assistant, and Settings. For example, the Smart Assistant application provides the functionality described in this application, recommending functional controls based on the currently displayed interface of the electronic device after receiving user input to the first control. The Smart Assistant application also provides an auto-fill function for forms. Furthermore, the Smart Assistant application can extract user information from the Gallery, SMS, and Notes and store this information in the user's personal information database within the Settings application. The Smart Assistant application includes applications such as... Figure 4A and Figure 4B The diagram shows a screenshot module, a recommendation module, and a visual inspection module. The screenshot module captures screenshots of the interface displayed on the electronic device. The recommendation module determines recommended functions based on the screenshots. The visual inspection module detects whether the screenshot is a form entry interface and detects relevant features of the input form fields.

[0229] The application framework layer provides application programming interfaces (APIs) and a programming framework for applications in the application layer, including various components and services to support Android development. The application framework layer includes some predefined functions. For example... Figure 17 As shown, the application framework layer may include a window manager, content provider, notification manager, resource manager, and visual graphics module, etc.

[0230] The window manager is used to manage windowed applications. It can retrieve screen size, determine if a status bar is present, lock the screen, and capture screenshots, among other things.

[0231] Content providers store and retrieve data, making that data accessible to applications. This data can include videos, images, audio, phone calls made and received, browsing history and bookmarks, phone books, and more.

[0232] File Explorer can provide applications with various resources, such as localized strings, icons, images, layout files, video files, and so on.

[0233] The notification manager allows applications to display notification information in the status bar. It can be used to convey informational messages and can disappear automatically after a short pause without user interaction. For example, the notification manager can be used to inform users of... Figure 2A The table information shown in (a) has been filled in, and message reminders are also included. The notification manager can also display notifications in the form of charts or scroll bar text in the system's top status bar, such as notifications from background applications, or notifications that appear on the screen as dialog windows. For example, it can display text messages in the status bar, emit alert sounds, provide vibration alerts during long-press operations on the first control 10 via a motor, and flashing indicator lights.

[0234] The system runtime library layer includes the Android Runtime and system libraries. The Android Runtime comprises core libraries and the virtual machine. The Android Runtime is responsible for the scheduling and management of the Android system.

[0235] The core library consists of two parts: one part is the functionalities that need to be called by the Java language, and the other part is the Android core library.

[0236] The application layer and application framework layer run in a virtual machine. The virtual machine executes the Java files of the application layer and application framework layer as binary files. The virtual machine is used to perform functions such as object lifecycle management, stack management, thread management, security and exception management, and garbage collection.

[0237] System libraries can include multiple functional modules. For example: surface manager, media libraries, 3D graphics processing libraries (e.g., OpenGL ES), 2D graphics engines (e.g., SGL), etc.

[0238] The Surface Manager is used to manage the display subsystem and provides the blending of 2D and 3D layers for multiple applications.

[0239] The media library supports playback and recording of various common audio and video formats, as well as still image files. It supports multiple audio and video encoding formats, such as MPEG4, H.264, MP3, AAC, AMR, JPG, and PNG.

[0240] The 3D graphics processing library is used to implement 3D graphics drawing, image rendering, compositing, and layer processing.

[0241] A 2D graphics engine is a graphics engine for 2D drawing.

[0242] The kernel layer is the layer between hardware and software. The kernel layer contains at least the display driver, camera driver, audio driver, and sensor driver.

[0243] Understandable, Figure 17 The components included in the system framework layer, system library, and runtime layer shown do not constitute a specific limitation on the electronic device 1600. In other embodiments of this application, the electronic device 1600 may include more or fewer components than shown, or combine some components, or split some components, or have different component arrangements. Exemplarily, an embodiment of this application provides an electronic device including: a processor; a memory for storing program instructions; and a processor for calling the program instructions in the memory to cause the electronic device to perform the methods described in the preceding embodiments.

[0244] For example, this application provides an electronic device, including: a memory and a processor; the memory is used to store program instructions; the processor is used to invoke the program instructions in the memory to cause the electronic device to execute the method in the preceding embodiments.

[0245] For example, this application provides an electronic device, including: a display screen, a memory, and a processor; the memory is used to store program instructions; the processor is used to invoke the program instructions in the memory to cause the electronic device to execute the method in the preceding embodiments.

[0246] For example, this application provides a chip system applied to an electronic device including a memory, a display screen, and a sensor; the chip system includes a processor; when the processor executes computer instructions stored in the memory, the electronic device executes the method described in the preceding embodiments.

[0247] For example, this application provides a chip including: an input interface, an output interface, a processor, and a memory. The input interface, the output interface, the processor, and the memory are connected through an internal connection path. The processor is used to execute code in the memory. When the code is executed, the processor is used to execute the method in the preceding embodiments.

[0248] For example, embodiments of this application provide a computer-readable storage medium having a computer program stored thereon, the computer program being processed by a processor to cause an electronic device to implement the methods described in the preceding embodiments.

[0249] For example, this application provides a computer program product including: execution instructions stored in a readable storage medium, at least one processor of an electronic device can read the execution instructions from the readable storage medium, and the at least one processor executes the execution instructions to cause the electronic device to implement the method in the foregoing embodiments.

[0250] In the above embodiments, all or part of the functions can be implemented by software, hardware, or a combination of software and hardware. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state disk (SSD)).

[0251] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. This program can be stored in a computer-readable storage medium, and when executed, it can include the processes described in the above method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as ROM or random access memory (RAM), magnetic disks, or optical disks.

Claims

1. A method for filling in form information, characterized in that, Applied to electronic devices, the method includes: The first interface is displayed, which includes a first input box, a second input box, and a first control; In response to a first operation input by the user onto the first control, the first service control is displayed; In response to a second operation input by the user to the first service control, a keyboard is displayed and a first form-filling control and first information are displayed in a first area above the keyboard; In response to a third operation input to the first form filling control, the first information is displayed in the first fill box, and the second information is displayed in the second fill box, wherein the second information is associated with the first information and is different from the first information.

2. The method according to claim 1, characterized in that, In response to a first operation input by the user to the first control, displaying the first service control includes: in response to the first operation, performing a screenshot operation on the first interface to obtain a first screenshot; Based on the first screenshot, the first service control is determined to be displayed.

3. The method according to claim 1, characterized in that, The second operation in response to user input on the first service control, displaying the keyboard and showing the first form-filling control and first information in a first area above the keyboard, includes: The text information corresponding to the first input box and the text information corresponding to the second input box are determined based on the second screenshot, where the second screenshot is a screenshot of the first interface; The first information and the second information are determined based on the text information corresponding to the first input box and the text information corresponding to the second input box; Display the first form-filling control and the first information.

4. The method according to claim 3, characterized in that, Based on the second screenshot, the information corresponding to the first input box and the information corresponding to the second input box include: Perform a first detection on the second screenshot to identify the text information in the second screenshot; Perform a second detection on the second screenshot to identify the fill boxes in the second screenshot; The text information is matched with the input boxes to determine the text information corresponding to the first input box and the text information corresponding to the second input box.

5. The method according to claim 1, characterized in that, Also includes: Before displaying the keyboard and showing the first form-filling control and the first information in the first area of ​​the keyboard, it is determined whether the keyboard is a system keyboard; When the keyboard is a system keyboard, the display area of ​​the keyboard is determined through the first interface; When the keyboard is not the system keyboard, the display area of ​​the keyboard is determined according to the third screenshot, which is a screenshot of the first interface; The first area above the display area of ​​the keyboard displays the first form filling control and the first information.

6. The method according to claim 1, characterized in that, Also includes: The first interface is used to determine whether the keyboard is a system keyboard; When the keyboard is the system keyboard, the first information is entered in the first input box through the second interface, and the second information is entered in the second input box; When the keyboard is not the system keyboard, a screenshot operation is performed to obtain a third screenshot; The key information of the keyboard is determined based on the third screenshot; Based on the key information, the first information, and the second information, simulate clicking the keyboard, fill in the first information in the first input box, and fill in the second information in the second input box.

7. The method according to any one of claims 1-6, characterized in that, Also includes: After the third operation in response to input to the first form filling control, displaying the first information in the first fill box and the second information in the second fill box, In response to a fourth operation by the user on the first fill-in box, a second fill-in control and third information are displayed, the third information being different from the first information; In response to a fifth operation input by the user on the second form-filling control, a first window is displayed, the first window including a first control and a second control; When a sixth operation is received on the first control, the third information is displayed in the first input box, and the fourth information is displayed in the second input box. The fourth information is associated with the third information, and the third information is different from the fourth information. When a seventh operation is received on the second control, the second information is displayed in the second input box, and the third information is displayed in the first input box.

8. The method according to claim 7, characterized in that, The first input field is related to the name, while the second input field is not related to the name. The method further includes: In response to the third operation input to the first form-filling control, after displaying the first information in the first fill-in box and the second information in the second fill-in box, In response to the user's eighth action on the second fill box, the third form control and the fifth information are displayed; In response to a ninth operation that displays input to a third form control, the first information is displayed in the first form input box, and the fifth information is displayed in the second form input box.

9. The method according to claim 1, characterized in that, Also includes: A second interface is displayed, which includes the first control and the first information. The second interface is different from the first interface. In response to the tenth operation input by the user to the first control, the second service control is displayed; In response to the eleventh operation input to the second service control, the first information is saved.

10. An electronic device, characterized in that, The electronic device includes: one or more processors, and memory; The memory is coupled to the one or more processors, the memory being used to store computer program code, the computer program code including computer instructions, the one or more processors invoking the computer instructions to cause the electronic device to perform the method as described in any one of claims 1 to 9.

11. A readable medium, characterized in that, The readable medium stores instructions that, when executed on an electronic device, cause the electronic device to perform the method of any one of claims 1 to 9.

12. A program product, characterized in that, The program product includes instructions that, when executed on an electronic device, cause the electronic device to perform any one of 1 to 9.

Images