Information input method and electronic equipment

Abstract

The embodiment of the invention is applied to the technical field of computers, and provides an information input method and electronic equipment. The electronic device displays a to-be-input interface, and the to-be-input interface comprises a plurality of filling controls and name controls corresponding to the filling controls. And then, in response to a trigger operation of a user for any filling control in the to-be-input interface, the electronic equipment obtains a tag name of a name control corresponding to the triggered filling control. Then, the electronic equipment selects candidate user information corresponding to the tag name of the name control from a user information base; wherein the user information base comprises a plurality of pieces of user information, and the user information is obtained by the electronic equipment through information collection from a plurality of application programs. And then, the electronic equipment displays the candidate user information in a to-be-input interface. According to the method and the device, the input process of the user information can be simplified, so that the information filling efficiency is improved.

Description

TECHNICAL FIELD

[0001] Embodiments of the present application relate to the technical field of computer, and particularly relate to an information input method and an electronic device. BACKGROUND

[0002] With the development of computer technology, more and more application programs (APPs) are developed and installed on electronic devices to perfect the functions and individualization of the electronic devices and provide users with more abundant use experience.

[0003] However, for some information filling scenarios (such as first registration of an application program), the user usually needs to fill in relevant information (such as name, mobile phone number, etc.), which makes the user need to input the same information on different application programs, and this may lead to a relatively cumbersome input process of user information, waste a long filling time of the user, and make the user have a poor use experience. SUMMARY

[0004] Embodiments of the present application provide an information input method and an electronic device, which are used to simplify the input process of user information and improve the filling efficiency of information.

[0005] To achieve the above object, embodiments of the present application adopt the following technical solutions:

[0006] In a first aspect, an information input method is provided, which is applied to an electronic device, and the electronic device is installed with multiple application programs. In the method, the electronic device displays a to-be-input interface, wherein the to-be-input interface includes multiple filling controls and name controls corresponding to the filling controls. Then, in response to a triggering operation of a user on any filling control in the to-be-input interface, the electronic device acquires a label name of the name control corresponding to the triggered filling control. Then, the electronic device selects candidate user information corresponding to the label name of the name control from a user information library. The user information library includes multiple pieces of user information, which are obtained by the electronic device through information collection from the multiple application programs. Then, the electronic device displays the candidate user information in the to-be-input interface.

[0007] In the present application, after detecting a triggering operation of a user on any fill-in control in the to-be-entered interface, the electronic device can select corresponding candidate user information from the user information library according to the label name of the triggered fill-in control, and display the candidate user information in the to-be-entered interface. In this way, intelligent filling of information can be realized, and the user only needs to select and fill in the corresponding user information according to the actual needs, without manually filling in the user information, simplifying the information filling process, improving the filling efficiency of the user information, and thus improving the user experience. In addition, since the candidate user information is user information collected by the electronic device from multiple application programs, that is, the electronic device can select the corresponding candidate user information from all previously collected user information, and is not limited to user information collected by the application to which the to-be-entered interface belongs, the comprehensive display of user information can be realized, the situation of repeatedly filling in the same user information in different applications by the user can be reduced, and a convenient condition for the user to quickly enter information is provided.

[0008] In a possible implementation manner of the first aspect, the process of selecting the candidate user information by the electronic device can specifically include: the electronic device performs standardization processing on the label name of the name control to obtain a target label name of the name control. The target label name is any label name existing in the user information library. Then, the electronic device selects the candidate user information corresponding to the target label name from the user information library.

[0009] In the present application, after obtaining the label name of the name control, the electronic device can perform standardization processing on the label name of the name control, so that the processed label name (i.e., the target label name) is any label name in the user information library, facilitating the electronic device to select the corresponding candidate user information from the user information library, and providing a basis for subsequent accurate display of the candidate user information.

[0010] In a possible implementation manner of the first aspect, the process of the electronic device performing standardization processing on the label name of the name control can specifically include: in a case where the attribute information of the name control satisfies any mapping relationship in the preset mapping rule, and / or, the similarity between the label name existing in the user information library and the label name of the name control is greater than a preset similarity, the electronic device performs standardization processing on the label name of the name control to obtain the target label name. The attribute information of the name control includes at least one of the label name of the name control, the application program of the interface to which the name control belongs, and the title name of the interface to which the name control belongs, and the preset mapping rule includes mapping relationships between a plurality of label names and the target label name.

[0011] In the present application, by judging whether the attribute information of the name control meets any mapping relationship in the preset mapping rule and / or whether there is a label name in the user information library that has a similarity greater than a preset similarity with the label name of the name control, it is judged whether the label name of the name control can be standardized. If the label name of the name control can be standardized, the electronic device can standardize the label name of the name control to obtain a target label name. In this way, multi-angle standardization of the label name can be realized, that is, the label name can be standardized in different ways, which provides a basis for subsequent determination of candidate user information.

[0012] In a possible implementation form of the first aspect, the process of standardizing the label name of the name control by the electronic device can specifically include: in a case where the attribute information of the name control meets any mapping relationship in the preset mapping rule, the electronic device takes the target label name included in the mapping relationship met by the attribute information as the target label name of the name control.

[0013] In the present application, if the attribute information of the name control meets any mapping relationship in the preset mapping rule, it means that the label name of the name control has been standardized, therefore, the electronic device can directly take the target label name in the mapping relationship met by the attribute information of the name control as the target label name of the name control. In this way, not only can the situation of energy loss caused by repeated standardization be reduced to improve the utilization rate of mobile phone resources, but also the standardization efficiency of the label name can be improved, thereby speeding up the efficiency of information display and providing convenient conditions for subsequent selection of corresponding information by the user.

[0014] In a possible implementation form of the first aspect, the process of standardizing the label name of the name control by the electronic device can specifically include: in a case where the attribute information of the name control does not meet any mapping relationship in the preset mapping rule, and there is a label name in the user information library that has a similarity greater than a preset similarity with the label name of the name control, the electronic device takes the label name with the similarity greater than the preset similarity as the target label name of the name control.

[0015] In the present application, if the attribute information of the name control does not satisfy any mapping relationship in the preset mapping rule, it indicates that the label name of the name control has not been standardized. Therefore, in order to further determine whether the label name of the name control can be standardized, the electronic device can continue to determine whether there is a label name in the user information library that has a similarity greater than a preset similarity with the label name of the name control. If there is a label name in the user information library that has a similarity greater than a preset similarity with the label name of the name control, it indicates that the label name in the user information library is similar to the label name of the name control. Therefore, the electronic device can directly use the label name with the similarity greater than the preset similarity as the target label name of the name control. In this way, the target label name can be quickly and accurately determined, which provides a basis for subsequent determination of candidate user information.

[0016] In a possible implementation manner of the first aspect, the method further includes: in a case where the attribute information of the name control does not satisfy any mapping relationship in the preset mapping rule, and there is no label name in the user information library that has a similarity greater than a preset similarity with the label name of the name control, the electronic device sends the label name of the name control to a server; wherein the server is configured to perform standardized learning on the label name of the name control. Subsequently, in response to an information input operation of the user on the triggered fill-in control, the electronic device displays the input user information in the triggered fill-in control; wherein the input user information is obtained based on the information input operation of the user.

[0017] In the present application, if the attribute information of the name control does not satisfy any mapping relationship in the preset mapping rule, and there is no label name in the user information library that has a similarity greater than a preset similarity with the label name of the name control, it indicates that the label name of the name control cannot be standardized, that is, the electronic device cannot query the corresponding user information from the user information library. Therefore, the electronic device can send the label name of the name control to the server, so as to subsequently perform standardized learning on the label name of the name control and obtain the target label name. In this way, the accuracy of determining the target label name is improved, and a basis is provided for subsequently determining the target label name of the name control.

[0018] In a possible implementation manner of the first aspect, after the electronic device selects the candidate user information, the method further includes: in a case where the information quantity of the candidate user information is multiple, the electronic device sorts the multiple candidate user information to obtain an information candidate list; wherein the information candidate list is configured to represent the arrangement order of the multiple candidate user information. Subsequently, the electronic device displays the candidate user information in the to-be-input interface according to the arrangement order of the multiple candidate user information.

[0019] In the present application, if the information quantity of the candidate user information is multiple, the electronic device can sort the multiple candidate user information, and display the corresponding candidate user information in the to-be-entered interface according to the arrangement order of the multiple candidate user information. In this way, the user can be facilitated to select the corresponding user information, and the user is provided with a convenient condition for quickly filling in the information, and thus the use experience of the user is improved.

[0020] In a possible implementation manner of the first aspect, the process that the electronic device sorts the multiple candidate user information can specifically include: the electronic device sorts the multiple candidate user information according to a preset display rule to obtain an information candidate list. The preset display rule includes at least one of the following: a chronological order of information update time, a descending order of information entry quantity, and an association with historical user information. The information update time is the time when the candidate user information is added to the user information library. The information entry quantity is the number of times that the candidate user information is selected by the user. The historical user information is the user information that has been entered in the to-be-entered interface.

[0021] In the present application, the electronic device sorts the multiple candidate user information from the aspects of information update time, information entry quantity, and association degree with historical user information, so that the candidate user information displayed in the to-be-entered interface is displayed in the arrangement order, the user is facilitated to select the corresponding user information, the user is provided with a convenient condition for quickly filling in the information, and thus the use experience of the user is improved.

[0022] In a possible implementation manner of the first aspect, before the electronic device displays the to-be-entered interface, the method further includes: in response to an information filling operation of the user on an information editing interface, the electronic device updates the filled user information to the user information library. The information editing interface is a display interface in a settings application of the electronic device.

[0023] In the present application, the user information in the user information library can be actively filled in by the user in the display interface of the settings application, so that the user information called by the electronic device subsequently is manually filled in by the user, the accuracy of the user information is ensured, and thus a basis for subsequently improving the information filling efficiency is provided.

[0024] In a possible implementation manner of the first aspect, before the electronic device displays the to-be-entered interface, the method further includes: the electronic device acquires new user information from a preset collection application, and saves the new user information to the user information library. The preset collection application is an application program with an information collection function in multiple application programs.

[0025] In the present application, the user information in the user information library can be obtained from a preset collection application with information collection function. In this way, automatic extraction of information can be realized without the need for the user to perform a corresponding trigger operation, simplifying the operation process and thereby providing a basis for subsequent improvement of information filling efficiency.

[0026] In a possible implementation of the first aspect, the process of obtaining the new user information by the electronic device can specifically include: in response to an information extraction operation of the user on a display interface in the preset collection application, the electronic device performs identity authentication on the user to obtain an identity authentication result. The display interface includes at least one piece of user information, and the identity authentication result is used to indicate whether the user is the owner user. Then, in a case where the identity authentication result indicates that the user is the owner user, the electronic device performs information extraction on the display interface to obtain the new user information.

[0027] In the present application, the phone can only perform information extraction on the display interface in a case where the identity authentication is successful (i.e., the user is the owner user). In this way, the occurrence of the situation where the non-owner user extracts user information can be reduced, the security of information extraction is improved, and the user experience is improved. In addition, since the new user information is extracted based on the information extraction operation of the user, that is, the user can be confirmed before information extraction, the occurrence of the situation where the user has information crisis due to excessive extraction of user information can be reduced, the security of the extraction environment is ensured, and convenient conditions are provided for subsequent display of candidate user information.

[0028] In a possible implementation of the first aspect, the process of saving the new user information by the electronic device can specifically include: the electronic device obtains a plurality of user information corresponding to a user to which the new user information belongs from the user information library. Then, in a case where there is no user information with the same label name as the new user information in the plurality of user information, the electronic device adds the new user information to the user information library.

[0029] In the present application, if there is no user information with the same label name as the new user information in the plurality of user information, it indicates that the user information library does not store the user information with the label name, and therefore the electronic device can save the new user information to the user information library. In this way, accurate saving of user information can be realized, the occurrence of the situation where user information is missed or incorrectly stored can be reduced, and convenient conditions are provided for subsequent timely acquisition of user information.

[0030] In a possible implementation manner of the first aspect, the method further includes: in a case where the plurality of user information includes user information that is identical to the label name of the new user information, the electronic device takes the user information that is identical to the label name of the new user information in the plurality of user information as original user information. Then, in a case where the original user information is not identical to the new user information, the electronic device displays an information update interface. Then, in response to a triggering operation of a user on an update control in the information update interface, the electronic device replaces the original user information with the new user information and updates the user information in the user information library.

[0031] In the present application, if the plurality of user information includes user information that is identical to the label name of the new user information, it indicates that the user information library already stores the user information of the label name, therefore, in order to improve the accuracy of information storage in the user information library, the electronic device can take the user information of the label name in the plurality of user information as original user information, and determine whether the original user information is identical to the new user information. If the original user information is not identical to the new user information, it indicates that there is error information in the original user information and the new user information, therefore, the electronic device can display an information update interface for the user to determine whether the user information needs to be updated. In this way, the accurate update of user information can be realized, the occurrence of user information update errors can be reduced, and the user experience is improved.

[0032] In a possible implementation manner of the first aspect, the method further includes: in response to a closing operation of an information collection function of a preset collection application, the electronic device deletes all user information associated with the preset collection application with the closed information collection function from the user information library.

[0033] In the present application, if the information collection function of the preset collection application is closed, it indicates that the user does not want to disclose the user information entered by the application, therefore, the electronic device can directly delete all user information associated with the application from the user information library. In this way, the information privacy of the user can be protected, and security assurance is provided for the user.

[0034] In a possible implementation manner of the first aspect, the preset collection application includes at least one of a short message application, a gallery application, and a note application.

[0035] In the present application, if the preset collection application is a short message application, the electronic device can extract user information from the short messages saved by the short message application. If the preset collection application is a gallery application, the electronic device can extract user information from the pictures saved by the gallery application. If the preset collection application is a note application, the electronic device can extract user information from the notes saved by the note application. In this way, the user information can be extracted in multiple aspects, that is, the user information in the user information library is not limited to the user information manually filled by the user, which provides a convenient condition for subsequent determination of candidate user information.

[0036] In a possible implementation manner of the first aspect, the method further includes: in response to the information adjustment operation of the user, the electronic device adjusts the user information library.

[0037] In the present application, after detecting the adjustment operation of the user, the electronic device can adjust the user information in the user information library according to the information adjustment operation. In this way, the adjustment function of the user information can be realized, so that the user information in the user information library is adjusted based on the actual needs of the user, ensuring the necessity of each piece of user information, and further improving the user experience.

[0038] In a possible implementation manner of the first aspect, the adjustment process of the electronic device to the user information library can specifically include: in response to an information deletion operation of the user on any user information in the user information library, the electronic device deletes the user information from the user information library; and / or, in response to an information modification operation of the user on any user information in the user information library, the electronic device modifies the user information to obtain modified user information, and updates the modified user information to the user information library; and / or, in response to an information addition operation of the user on the user information library, the electronic device adds the newly added user information to the user information library.

[0039] In the present application, if the information deletion operation of the user on any user information in the user information library is detected, the electronic device can delete the user information from the user information library to realize the deletion function of the user information. If the information modification operation of the user on any user information in the user information library is detected, the electronic device modifies the user information and updates the modified user information to the user information library to realize the modification function of the user information. If the information addition operation of the user on the user information library is detected, the electronic device adds the newly added user information to the user information library to realize the addition function of the user information. So that the user information in the user information library is adjusted based on the actual needs of the user, ensuring the necessity of each piece of user information, and further improving the user experience.

[0040] In a possible implementation manner of the first aspect, the method further includes: in response to the selection operation of the user on the candidate user information, the electronic device displays the selected candidate user information in the triggered filling control.

[0041] In the present application, after displaying the candidate user information in the interface to be filled in, if any candidate user information is selected by the user, the form filling assistant can display the selected candidate user information in the triggered filling control. In this way, intelligent filling of user information can be realized. The user only needs to select the corresponding user information for filling according to the actual demand, without manually filling in the user information, simplifying the information filling process, improving the filling efficiency of the user information, and thus improving the user experience.

[0042] In a possible implementation of the first aspect, after the electronic device displays the interface to be filled in, the method further includes: in response to a one-key filling operation of the user on the interface to be filled in, the electronic device obtains the label name of each name control in the interface to be filled in. Then, the electronic device performs standardization processing on the label name of each name control to obtain target label names of the plurality of name controls. Then, the electronic device divides the target label names of the plurality of name controls to obtain at least one group of sub-attribute information lists. The sub-attribute information list includes the target label name of at least one name control, and the target label names of the at least one name control are the label names of the same user. Then, for each group of sub-attribute information lists, the electronic device selects at least one candidate user information group matching the sub-attribute information list from the user information library. Then, the electronic device displays at least one candidate user information group corresponding to each group of sub-attribute information lists in the interface to be filled in.

[0043] In the present application, after detecting the one-key filling operation of the user on the interface to be filled in, the electronic device can perform standardization processing on the label name of each name control to obtain target label names of the plurality of name controls. By dividing the target label names of the plurality of name controls, the label names belonging to the same user are divided into a group of sub-attribute information lists, so as to display the candidate list of each group of sub-attribute information lists in the interface to be filled in. In this way, it provides a convenient condition for subsequent one-key filling of user information, simplifies the information filling process, greatly improves the filling efficiency of user information, and thus improves the user experience.

[0044] In a possible implementation of the first aspect, the process of dividing the target label names of the plurality of name controls by the electronic device can include: in the case that the target label names of the plurality of name controls include a plurality of same target label names, the electronic device divides the target label names of the plurality of name controls according to the plurality of same target label names to obtain at least one group of sub-attribute information lists. Each group of sub-attribute information lists includes one target label name of the plurality of same target label names.

[0045] In the present application, since it is impossible to need to input the same user information twice in one to-be-input interface, that is, at least two name controls with the same target label name cannot be displayed simultaneously in the to-be-input interface, if the target label names of the plurality of name controls include a plurality of same target label names, it indicates that the to-be-input interface can need to input the user information of different users, and therefore, the electronic device can divide the target label names of the plurality of name controls according to the plurality of same target label names to obtain at least one group of sub attribute information list. In this way, the target label names in each group of sub attribute information list are the label names of the same user, which provides a basis for subsequent accurate determination of the candidate user information group, and further improves the accuracy of determination of the candidate user information group.

[0046] In a possible implementation manner of the first aspect, the method further includes: in response to a selection operation of a user on any candidate user information group corresponding to the sub attribute information list, the electronic device determines at least one fill-in control corresponding to the selected candidate user information group from the plurality of fill-in controls to which the sub attribute information list belongs. Then, the electronic device displays the candidate user information in the at least one fill-in control.

[0047] In the present application, if any candidate user information group corresponding to the sub attribute information list is selected by the user, the electronic device can determine at least one fill-in control corresponding to the selected candidate user information group from the plurality of fill-in controls to which the sub attribute information list belongs, and display the candidate user information in the at least one fill-in control. In this way, one-key filling of user information can be realized, and the user only needs to select the corresponding user information group for filling according to the actual demand, without the need for the user to fill in the user information in the fill-in control one by one, which simplifies the information filling process, greatly improves the filling efficiency of the user information, and further improves the user experience.

[0048] In the second aspect, the present application provides an electronic device, which includes a display screen, one or more processors, and one or more memories; the one or more processors are coupled with the display screen and the one or more memories; the display screen is used to display a to-be-input interface, and the one or more memories are used to store computer program codes including computer instructions, which, when executed by the one or more processors, cause the electronic device to perform the method as described above.

[0049] In the third aspect, the present application provides a computer readable storage medium, which includes computer instructions, which, when executed on an electronic device, cause the electronic device to perform the method as described above.

[0050] In a fourth aspect, the present application provides a computer program product, which, when running on an electronic device, causes the electronic device to perform the method described above.

[0051] In a fifth aspect, a chip is provided, comprising: an input interface, an output interface, a processor and a memory, the input interface, the output interface, the processor and the memory are connected through internal connection paths, the processor is configured to execute the code in the memory, when the code is executed, the processor is configured to perform the method described above.

[0052] The beneficial effects that can be achieved by the electronic device of the second aspect, the computer readable storage medium of the third aspect, the computer program product of the fourth aspect and the chip of the fifth aspect provided above can refer to the beneficial effects in the first aspect and any possible design of the first aspect, which will not be repeated here. BRIEF DESCRIPTION OF DRAWINGS

[0053] Figure 1 A schematic diagram of a table related to an embodiment of the present application;

[0054] Figure 2 An interface and operation schematic diagram of a form filling interface related to an information processing method provided by an embodiment of the present application;

[0055] Figure 3 A flowchart of a side selection and form filling process related to an information processing method provided by an embodiment of the present application;

[0056] Figure 4 An interface and operation schematic diagram of a side selection and form filling process related to an information processing method provided by an embodiment of the present application;

[0057] Figure 5 A table frame relative position schematic diagram related to an information processing method provided by an embodiment of the present application;

[0058] Figure 6 An element matching schematic diagram related to an information processing method provided by an embodiment of the present application;

[0059] Figure 7 An automatic form filling flowchart related to an information processing method provided by an embodiment of the present application;

[0060] Figure 8 A screenshot schematic diagram of a first interface related to an information processing method provided by an embodiment of the present application;

[0061] Figure 9 A table frame matching schematic diagram related to an information processing method provided by an embodiment of the present application;

[0062] Figure 10The relative position diagram of the filling box and the matched label box involved in the information processing method provided by the embodiment of the present application;

[0063] Figure 11 The table box coordinate diagram involved in the information processing method provided by the embodiment of the present application;

[0064] Figure 12 The flow diagram of the information processing method provided by the embodiment of the present application;

[0065] Figure 13 The flow diagram of the information processing method provided by the embodiment of the present application;

[0066] Figure 14 The display diagram of the mobile phone provided by the embodiment of the present application;

[0067] Figure 15 The display diagram of the mobile phone provided by the embodiment of the present application;

[0068] Figure 16 The diagram of the state machine provided by the embodiment of the present application;

[0069] Figure 17 The diagram of the state machine provided by the embodiment of the present application;

[0070] Figure 18 The diagram of the state machine provided by the embodiment of the present application;

[0071] Figure 19 The diagram of the state machine provided by the embodiment of the present application;

[0072] Figure 20 The diagram of the state machine provided by the embodiment of the present application;

[0073] Figure 21 The diagram of the state machine provided by the embodiment of the present application;

[0074] Figure 22 The diagram of the state machine provided by the embodiment of the present application;

[0075] Figure 23A The diagram of the part style of the table provided by the embodiment of the present application;

[0076] Figure 23B The diagram of the five target categories of the table provided by the embodiment of the present application;

[0077] Figure 24 The diagram of the part software structure of the electronic device provided by the embodiment of the present application;

[0078] Figure 25A flowchart of a table visual inspection method provided by an embodiment of the present application is shown in FIG. 1;

[0079] Figure 26A A flowchart of a table visual inspection method provided by an embodiment of the present application is shown in FIG. 1;

[0080] Figure 26B A flowchart of a table visual inspection method provided by an embodiment of the present application is shown in FIG. 1;

[0081] Figure 27A A flowchart of a table visual inspection method provided by an embodiment of the present application is shown in FIG. 1;

[0082] Figure 27B A flowchart of a table visual inspection method provided by an embodiment of the present application is shown in FIG. 1;

[0083] Figure 28 A hardware structure diagram of an electronic device provided by an embodiment of the present application is shown in FIG. 1;

[0084] Figure 29 A software structure diagram of an electronic device provided by an embodiment of the present application is shown in FIG. 1;

[0085] Figure 30 A flowchart of an information entry method provided by an embodiment of the present application is shown in FIG. 1;

[0086] Figure 31 An interface diagram for filling in user information provided by an embodiment of the present application is shown in FIG. 1;

[0087] Figure 32 An interface diagram for adding user information provided by an embodiment of the present application is shown in FIG. 1;

[0088] Figure 33 An interface diagram for setting a preset collection application provided by an embodiment of the present application is shown in FIG. 1;

[0089] Figure 34 An interface diagram for extracting user information provided by an embodiment of the present application is shown in FIG. 1;

[0090] Figure 35 A method flowchart for label name standardization provided by an embodiment of the present application is shown in FIG. 1;

[0091] Figure 36 A method flowchart for updating a user information library provided by an embodiment of the present application is shown in FIG. 1;

[0092] Figure 37 A method flowchart for querying and adjusting user information provided by an embodiment of the present application is shown in FIG. 1;

[0093] Figure 38 An interface diagram for querying user information provided by an embodiment of the present application is shown in FIG. 1;

[0094] Figure 39 A user information deletion interface provided by an embodiment of the present application;

[0095] Figure 40 A flowchart of an information entry method provided by an embodiment of the present application;

[0096] Figure 41 A user information selection interface provided by an embodiment of the present application. DETAILED DESCRIPTION

[0097] The technical solutions in the embodiments of the present application will be described below with reference to the drawings in the embodiments of the present application. In the description of the embodiments of the present application, the terms used in the following embodiments are only for the purpose of describing the specific embodiments and are not intended to be limiting on the present application. As used in the specification and the appended claims of the present application, the singular forms “a,” “an,” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will also be understood that “at least one” and “one or more” as used in the embodiments herein indicates one or two or more (including two). The term “and / or” is used to describe the association relationship of the associated objects, which means that there can be three relationships; for example, A and / or B can mean that A exists alone, A and B exist together, and B exists alone, where A and B can be singular or plural. The character “ / ” generally represents an “or” relationship between the associated objects.

[0098] In the present specification, the reference to “one embodiment” or “some embodiments” or the like means that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment of the present application. Thus, the appearance of the phrases “in one embodiment”, “in some embodiments”, “in other embodiments”, “in additional embodiments” and the like in various places in the specification are not necessarily all referring to the same embodiment, but mean that “one or more but not all embodiments”, unless otherwise specifically noted. The terms “comprise”, “include”, “have” and their conjugates mean “including but not limited to”, unless otherwise specifically noted. The term “connected” includes direct and indirect connections, unless otherwise specified. “First”, “second”, etc. are only for the purpose of description and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features.

[0099] In the embodiments of the present application, the word "exemplarily" or "for example" is used to represent an example, illustration or description. Any embodiment or design scheme described as "exemplarily" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or design schemes. In fact, the word "exemplarily" or "for example" is used to present the relevant concept in a specific manner.

[0100] It should be noted that the user information used in the technical solutions of the present application is limited to information that has obtained individual consent of the user, including but not limited to informing and reminding the user to read the relevant user agreement (notification) before the user uses the information collection function, and signing the agreement (authorization) including authorization of the relevant user information.

[0101] A table is a kind of data organization information, used for structured display of information. The table includes a plurality of table boxes, and the table box can include content text. The table displayed by the electronic device can include a plurality of table box groups, and each table box group includes a label and a fill-in box. The label includes a label box and a label name, and the label name is in the label box.

[0102] The electronic device provides a plurality of application programs (APPlications, APPs), and the application interface of the application program can include a form filling interface. For example, the electronic device provides a shopping application related to a delivery address filling interface, an account registration interface of an application program, and a starting address filling interface of a car-hailing application. The electronic device displays a table on such interfaces, and the table includes a label box and a fill-in control, and the fill-in control includes a fill-in box, which can have filled-in text or can have no filled-in text.

[0103] It should be noted that the table displayed by the electronic device on the form filling interface can be a table in various forms. For example, the table displayed by the electronic device includes only label text and a fill-in box, or the table includes a label box and a fill-in box. The information processing method related to the embodiments of the present application is applicable to the information processing process of the electronic device on the information filling interface of the application program, and is not applicable to the information processing process of the table document in the conventional Excel format.

[0104] As shown in FIG. 1, it is a schematic diagram of a table related to the embodiments of the present application. Figure 1 As shown in FIG. 2, it is a schematic diagram of the interface and operation of the form filling interface related to the information processing method provided by the embodiments of the present application. Figure 2

[0105] ​The filling control includes a filling box and filling content, and the filling content is in the filling box. In the case that there is filling content in the filling box of the filling control, the filling content can be divided into prompt text and content text. In the case that there is no filling content in the filling box of the filling control, the filling content can be a null value, and the electronic device can fill the filling box with content in response to a user operation.

[0106] Each table can include a plurality of table frame groups, and the label box and the filling box of each table frame group are relatively close in position in the table. For example, the label box and the filling box of each table frame group are usually located in the same row and adjacent columns in the table, or the label box and the filling box of each table frame group are usually located in the same column and adjacent rows in the table.

[0107] As shown in Figure 1 , it is a schematic diagram of a table. Figure 1 In the table, the table includes a plurality of table frames, and each table frame can fill in text. As shown in Figure 1 , the table includes a label 101 and a filling control 102, the label 101 includes a label box 1011 and a label name 1012, and the filling control includes a filling box 1021 and content text 1022.

[0108] As shown in Figure 1 , the table can include a plurality of label boxes 1011, and different label boxes 1011 include different label names 1012, such as name, mobile phone number, ID number, email, contact address, etc. Each label box 1011 has a matching filling box 1021, and the filling box 1021 is used to fill in the content text 1022 matching the label name. The content text 1022 in different filling boxes 1021 is different.

[0109] According to the matching relationship between the label 101 and the filling control 102, Figure 1 the table in the table can also be divided into a plurality of table frame groups 103, and the table frame group 103 includes a label 101 and a filling control 102, and the label 101 and the filling control 102 are located in the same row and adjacent columns of the table. In other cases, the label 101 and the filling control 102 of a table frame group 103 can also be located in adjacent rows in the table.

[0110] As shown in Figure 1 , there is content text "177****6666" in the filling box 1021A, there is prompt text "Please enter ID number here" in the filling box 1021B, there is no content text in the filling box 1021C, and there is prompt content "Please enter contact address here" in the filling box 1021D.

[0111] Figure 2 As shown in the table, the border of the table frame can be present and displayed. As shown in Figure 2As shown in FIG. 1, a schematic diagram of a form filling interface involved in an information processing method provided by an embodiment of the present application is shown. As shown in FIG. 1, the form filling interface provided by the electronic device is a form filling interface of a form filling application. Figure 2 As shown in (1) of FIG. 1, the form filling interface displays a form, and the edges of the form frame can also be non-existent or hidden and not displayed.

[0112] The form filling interface provided by the electronic device includes a plurality of labels and filling boxes. The label and the label name in the label box are usually already existent and fixed. The content text in the plurality of filling boxes needs to be manually input by the user. The form filling interface of the electronic device in various scenarios all needs the user to manually fill in the text in the form, which is relatively cumbersome to operate. In addition, the content text involved in the various form filling interfaces provided by the electronic device can be the same, and the electronic device needs the user to repeatedly input multiple times, which further increases the degree of cumbersome operation of the user, and the user experience of form filling is poor.

[0113] Therefore, an information processing method is provided in the embodiments of the present application. The electronic device triggers automatic form filling in response to a touch operation of the user. The electronic device automatically identifies a filling box to be filled in the form, finds a label matched with the filling box, and automatically matches content text for the filling box according to the label, so as to realize automatic form filling. In this way, the interactive operation of the user in the form filling interface can be greatly reduced, and the user experience is improved.

[0114] For example, the electronic device can be a mobile phone, a tablet computer, a personal computer (PC), a smart screen, a desktop computer, a laptop computer, a notebook computer, an ultra-mobile personal computer (UMPC), a netbook, a smart watch, an artificial intelligence (AI) sound box, a vehicle-mounted device, various teaching aids (such as a learning machine and an early education machine), a smart toy, a portable robot, a personal digital assistant (PDA), an augmented reality (AR) \ virtual reality (VR) device, a media player, a device with mobile office function, a device with smart home function, a device with audio-visual entertainment function, a device supporting smart travel, and the like. The specific form of the electronic device is not specially limited in the embodiments of the present application. The structure of the electronic device will be described in detail in the following embodiments.

[0115] The implementation scheme of triggering automatic form filling and the implementation scheme of various automatic form filling scenarios involved in the information processing method provided by the embodiments of the present application will be described in turn with reference to several embodiments.

[0116] In Example 1, the electronic device triggers automatic form filling in response to a first operation.

[0117] The electronic device triggers the execution of the information processing method provided in this embodiment in response to a touch operation of a user, and automatically fills in the form in the form interface.

[0118] In one example, the electronic device triggers automatic form filling in response to a touch operation of a user in the form interface.

[0119] As shown in Figure 2 , the interface and operation schematic diagram for triggering automatic form filling involved in the information processing method provided in this embodiment. As shown in (1) of Figure 2 , the electronic device displays a form interface 2101, which can be a form filling interface for receiving information. The form displayed in the form interface 2101 includes label boxes and filling controls. The label names included in the label boxes include name, phone number, region, and detailed address, etc. The filling controls include or are used to fill in the content texts corresponding to the label names in the label boxes, such as the content text corresponding to the name, the content text corresponding to the phone number, the content text corresponding to the region, and the content text corresponding to the detailed address, etc.

[0120] As shown in the form interface in (1) of Figure 2 , the plurality of filling controls included in the form are blank form boxes, i.e., the corresponding content texts have not been filled in the filling controls. The electronic device can trigger an automatic form filling process in response to a touch operation of a user, and automatically fill in the corresponding content texts in the plurality of filling controls in the form interface.

[0121] According to different filling methods of the filling controls, the filling controls can be divided into input form boxes and drop-down form boxes. For example, in the case of input form boxes, the electronic device can add corresponding content texts in the input form boxes in response to an input operation of a user. The input form boxes are suitable for cases where the types of content texts are more or cannot be recommended for selection. As shown in (1) of Figure 2 , the filling control corresponding to the name, the filling control corresponding to the phone number, and the filling control corresponding to the detailed address are all input form boxes.

[0122] For another example, the input operation of a user can be a text input operation or a voice input operation of a user on the form interface. For another example, in the case of drop-down form boxes, the electronic device can add corresponding content texts in the drop-down form boxes in response to a drop-down selection operation of a user. The drop-down form boxes are suitable for cases where the types of content texts are relatively less or can be recommended for selection. As shown in (1) of Figure 2 , the filling control corresponding to the region is a drop-down form box.

[0123] The electronic device can trigger automatic form filling in response to a touch operation of a user, automatically match or fill in corresponding content text for an input form box and / or a drop-down form box in a form filling interface displayed by the electronic device.

[0124] In an example, the electronic device displays a first control in the form filling interface, and the first control is triggered by the user to trigger automatic form filling. The electronic device can trigger automatic form filling in the form filling interface in response to a touch operation of the user on the first control.

[0125] As shown in (1) of FIG. 1, the electronic device displays a first control 202 in a form filling interface 201, and the first control 202 can include a gesture prompt bar, a control in the form of a table, a control in the form of artificial intelligence (AI), a control in the form of an input pen, etc., without limitation. Figure 2 As shown in (1) of FIG. 1, the electronic device can receive a touch operation of the user on the first control 202, such as a single click, a double click, or a long press, to trigger an automatic form filling process.

[0126] Figure 2 In a case, the electronic device directly triggers the automatic form filling process and starts to perform automatic form filling in the form filling interface in response to a touch operation of the user on the first control 202.

[0127] In another case, the electronic device can also display automatic form filling options corresponding to different form filling modes in response to a touch operation of the user on the first control 202. As shown in (2) of FIG. 1, the electronic device displays automatic form filling options, including a first mode control 203 corresponding to a first form filling mode and a second mode control 204 corresponding to a second form filling mode.

[0128] The first form filling mode corresponds to a side selection and side filling mode. The electronic device automatically fills in content text of one writing control currently selected by the user. The electronic device automatically fills in content text corresponding to another writing control selected by the user next time. Figure 2 The second form filling mode corresponds to a one-key form filling mode. After the electronic device triggers automatic form filling, the electronic device automatically fills in content text corresponding to multiple writing controls according to all the writing controls in the form filling interface.

[0129] It should be noted that the side selection and side filling described herein is only used to indicate a scheme in which the electronic device automatically fills in content text of one writing control currently selected by the user. In other cases, the side selection and side filling can also have other names, such as form filling, sequential form filling, single-box form filling, etc., without limitation.

[0130] The second form filling mode corresponds to a one-key form filling mode. After the electronic device triggers automatic form filling, the electronic device automatically fills in content text corresponding to multiple writing controls according to all the writing controls in the form filling interface.

[0131] The second form filling mode corresponds to a one-key form filling mode. After the electronic device triggers automatic form filling, the electronic device automatically fills in content text corresponding to multiple writing controls according to all the writing controls in the form filling interface.

[0132] ​It should be noted that the one-key form filling described herein is only used to indicate that the electronic device fills multiple writing controls of the form filling interface at one time. In other cases, one-key form filling can also have other names, such as quick form filling, one-key filling, etc., which are not limited.

[0133] As shown in (2) of Figure 2 , the electronic device displays the first mode control 203 and the second mode control 204, which can be displayed in the form of text on the first interface 201. For example, the electronic device displays "form filling" to mark the first mode control 203, and displays "one-key writing" to mark the second mode control 204. In other cases, the electronic device can also mark the first mode control and the second mode control in the form of a graphical control, which is not limited.

[0134] As shown in (2) of Figure 3 , the electronic device receives a click operation of the user acting on the first mode control 203, triggers the edge selection and filling, as shown in (3) of Figure 4 , the specific implementation scheme of the edge selection and filling can be referred to the related description of subsequent embodiment 2. Or as shown in (4) and (5) of Figure 4 , the electronic device can receive a click operation of the user acting on the second mode control 204, trigger one-key form filling, and the specific implementation scheme of one-key form filling can be referred to the related description of subsequent embodiment 3.

[0135] The information processing method provided in this example, the electronic device displays the first control in the form filling interface, and the electronic device can display the option control corresponding to different form filling modes in response to the touch operation of the user acting on the first control, for example, the first mode control corresponding to the edge selection and filling and the second mode control corresponding to the one-key form filling. The electronic device can trigger the automatic form filling operation of the corresponding mode in response to the click operation of the user acting on the first mode control or the second mode control. The way of triggering the form filling operation provided by the electronic device is relatively simple, saves user operation, and improves user experience.

[0136] In another example, the electronic device triggers automatic form filling after displaying the form filling interface in response to the touch operation of the user on the smart application.

[0137] For example, the electronic device can automatically start the automatic form filling process in response to the touch operation of the user on the smart application. The electronic device can display the automatic form filling option in the form filling interface when it is detected that the currently displayed application interface is the form filling interface.

[0138] For example, electronic devices can also add an autofill on / off control within the settings interface of smart applications. The electronic device can receive user clicks on the autofill control and automatically start the autofill process. When the electronic device detects that the currently displayed application interface is a form-filling interface, it can display the autofill option on the form-filling interface.

[0139] In other examples, electronic devices can also respond to a user's voice commands, triggering automatic form filling.

[0140] For example, if an electronic device receives voice information from a user that includes keywords indicating automatic form filling, and the electronic device detects that the currently displayed application interface is a form filling interface, it will initiate the automatic form filling process. The keywords indicating automatic form filling mentioned here can include at least one of the following: form filling, selection-based form filling, one-click form filling, automatic form filling, and AI form filling, without limitation.

[0141] Electronic devices can also trigger automatic form filling in other ways. Other schemes that can trigger automatic form filling can be applied to the information processing method provided in the embodiments of this application, and will not be elaborated further.

[0142] Example 2: The automatic form-filling mode executed by the electronic device is "select and fill simultaneously".

[0143] The electronic device responds to user input, triggering a form-filling interface where users select and fill in information simultaneously. The device identifies the currently selected field and determines the matching label. Based on the label's name, the device fills in the corresponding text for that field.

[0144] It should be noted that, in the information processing method provided in this application embodiment, the electronic device fills in matching text in the input box. This can be done by directly displaying the acquired text in the input box, or by displaying the acquired text in the input box or an adjacent area for the user to select. If text already exists in the displayed input box, the electronic device can also respond to user operations by adding, modifying, or deleting text in the input box.

[0145] like Figure 4 The diagram shown is a schematic flowchart of the information processing method provided in this application, involving selection-while-filling. The electronic device performs selection-while-filling, mainly including the following steps:

[0146] S301: The electronic device displays a first interface, which is a form filling interface. The first interface includes a first fill box and multiple label boxes. Each label box has a corresponding label name, and different label boxes have different label names.

[0147] The information processing method provided in the embodiments of the present application includes: an electronic device displays a first interface. The first interface can be the form filling interface in the foregoing examples, and the first interface will be used to represent the form filling interface hereinafter. The electronic device triggers automatic form filling in response to a touch operation of a user on the first interface. The touch operation to which the electronic device responds can include a click operation, a single-click operation, a double-click operation, a sliding operation, etc., and is not limited in this regard.

[0148] As shown in FIG. 4A(1), the electronic device displays a first interface 401, and the first interface 401 includes a form. The form includes a plurality of labels 402, such as 402A, 402B, 402C, 402D, etc. Different labels have different label names. For example, the label name of 402A is name, the label name of 402B is mobile phone number, the label name of 402C is region, and the label name of 402D is detailed address. Figure 4

[0149] In addition, the form also includes a plurality of fill-in boxes 403, such as 403A, 403B, 403C, 403D, etc. Among the plurality of fill-in boxes displayed by the first interface, some fill-in boxes can have filled-in content text. For example, the filled-in content text in 403A is Zhang San. Among the plurality of fill-in boxes displayed by the first interface, some fill-in boxes can also have no filled-in content text. For example, 403B, 403C, and 403D have no corresponding filled-in content text.

[0150] For ease of description, the fill-in box with filled-in content text in the form can be referred to as a filled-in form box, and the fill-in box with no filled-in content text can be referred to as an unfilled form box.

[0151] The electronic device determines the fill-in box to be filled in currently in response to a touch operation of a user on the unfilled form box. For ease of description, the fill-in box to be filled in, which is indicated by the touch operation of the current user, is referred to as a first fill-in box. As shown in FIG. 4A(1), the electronic device receives a click operation of a user on the fill-in box 403C and determines that 403C is the first fill-in box to be filled in currently. Figure 3

[0152] The electronic device can also move the cursor into the first fill-in box in response to the touch operation of the user on the first fill-in box. The electronic device can determine the first fill-in box to be filled in by detecting the position of the current cursor according to the position of the current cursor.

[0153] As shown in FIG. 4A(1), the electronic device receives a touch operation of a user on the fill-in box 403C and displays a cursor 404 in the fill-in box 403C, as shown in FIG. 4A(2). Figure 4 Figure 4

[0154] ​​​​In an example, the electronic device obtains a first screenshot of the first interface, obtains the current cursor position by analyzing the first screenshot, and then determines the first fill-in box in the first interface.

[0155] In implementation, as shown in Figure 5 S302, the electronic device determines the first label box matched with the first fill-in box.

[0156] S3011: The electronic device obtains a first screenshot of the first interface.

[0157] S3012: The electronic device performs feature extraction on the first screenshot to identify the first cursor.

[0158] As shown in (2) of Figure 5 , the electronic device takes a screenshot of the first interface to obtain the corresponding first screenshot, as shown in (3) of Figure 5 .

[0159] The electronic device performs feature extraction on the first screenshot to extract possible features of the corresponding cursor, including color features, shape features, texture features, etc. The electronic device can use pattern recognition or machine learning algorithms to identify the cursor in the first screenshot, which is referred to as the first cursor.

[0160] S3013: The electronic device determines the first fill-in box according to the pixel coordinates of the first cursor in the first screenshot.

[0161] The electronic device identifies the first cursor in the first screenshot to obtain the pixel coordinates of the first cursor in the first screenshot, and then determines the first fill-in box. For example, the electronic device determines the fill-in box whose coordinate range includes the coordinate range of the first cursor as the first fill-in box, or the electronic device can determine the fill-in box whose starting coordinate and the coordinate difference of the first cursor are within a preset range as the first fill-in box.

[0162] In another example, the electronic device can also obtain the cursor position by extracting the parameters associated with the cursor in the first interface, and then determine the first fill-in box to be filled in the first interface according to the cursor position.

[0163] For example, for interface input and text area textarea elements, the electronic device can use the selectionStart and selectionEnd attributes to obtain the position of the cursor. selectionStart returns the starting position of the cursor in the element, and selectionEnd returns the ending position of the cursor in the element.

[0164] In a specific implementation, the electronic device can also obtain the first cursor in the first interface in other manners, and then determine the first fill-in box to be filled in according to the first cursor.

[0165] In S302, the electronic device determines, from the plurality of label boxes, a first label box matched with the first fill-in box according to the first operation on the first fill-in box in the first interface. The first label box has a first label name.

[0166] In the information processing method provided in this embodiment, the electronic device identifies a plurality of table boxes in the first interface, including a plurality of label boxes and a plurality of fill-in boxes, and the plurality of fill-in boxes include a first fill-in box to be filled in. The table in the first interface includes a plurality of table box groups, and each table box group includes a matched label box and a fill-in box. The positions of the fill-in box and the label box in each table box group are generally fixed. For example, the label box and the fill-in box in each table box group are located in the same row and adjacent columns, or the label box and the fill-in box in each table box group are generally located in the same column and adjacent rows in the table.

[0167] As shown in (1) of FIG. 5, the table includes two table box groups, where the first table box group 501 includes a label box 5011 and a fill-in box 5012, and the second table box group 502 includes a label box 5021 and a fill-in box 5022. Figure 5 Figure 4 The label box and the fill-in box in each table box group are located in the same row and adjacent columns. For example, the label box 5011 and the fill-in box 5012 are located in the same row, and the label box 5011 is located in the left adjacent column of the fill-in box 5012. The label box 5021 and the fill-in box 5022 are located in the same row, and the label box 5021 is located in the left adjacent column of the fill-in box 5022.

[0168] As shown in (2) of FIG. 5, the table includes two table box groups, namely, an A table box group and a B table box group. The label box 5031 of the A table box group 503 is located in the upper adjacent column of the fill-in box 5032, and the label box 5041 of the B table box group 504 is located in the upper adjacent column of the fill-in box 5042.

[0169] As shown in (2) of FIG. 5, the table includes two table box groups, namely, an A table box group and a B table box group. The label box 5031 of the A table box group 503 is located in the upper adjacent column of the fill-in box 5032, and the label box 5041 of the B table box group 504 is located in the upper adjacent column of the fill-in box 5042. Figure 4

[0170] The two relative positions of the table boxes shown in FIG. 5 can include the relative positions of the label boxes and the fill-in boxes matched with each other in most tables. Based on this, after determining the first fill-in box to be filled in, the electronic device can determine, from the plurality of label boxes, a label box matched with the first fill-in box according to the position of the first fill-in box and the relative positions of the plurality of label boxes. For ease of description, the electronic device can mark the label box matched with the first fill-in box in the plurality of label boxes as a first label box. Figure 4 ​​

[0171] S303: The electronic device displays the first content text in the associated region of the first fill-in box; the label attribute of the first content text matches the first label name.

[0172] The electronic device obtains the first label matched by the first fill-in box, extracts the label name from the first label box of the first label, and records the label name as the first label name. The electronic device matches the first content text for the first fill-in box according to the first label name, and the label attribute of the first content text matches the first label name.

[0173] As shown in (3) of FIG. 4B, the electronic device determines that the first fill-in box is 403C, the first label box matched by 403C is 402C, and the first label name in 402C is region. The electronic device matches the first content text with the label attribute of region, such as "A province B city C district", for the first fill-in box based on the first label name "region". Figure 3 In other cases, the electronic device determines that the first label name filled in the first label box is "gender", and the electronic device matches the first content text with the label attribute of gender, such as "female" or "male", for the first fill-in box.

[0174] In one specific implementation, the electronic device matches the content text for the first fill-in box, and the number of content texts can be one or more.

[0175] In one case, the electronic device matches one content text, and the electronic device can directly fill the matched content text into the first fill-in box as the first content text.

[0176] In another case, the electronic device matches at least one content text for the first fill-in box, and the electronic device can display the at least one content text in the associated region of the first fill-in box. The electronic device can fill the content text receiving the touch operation of the user into the first information table as the first content text.

[0177] As shown in (4) of FIG. 4B, the electronic device matches the content text "A province B city C district" for the first fill-in box 403C, and the electronic device displays the content text 405 in the adjacent column of 403C. If the electronic device receives the touch operation acting on the content text 405, the electronic device fills "A province B city C district" into the first fill-in box 403C as the first content text, as shown in (5) of FIG. 4B.

[0178] Figure 4 Figure 4

[0179] ​​​On the basis of the foregoing embodiments, after the electronic device matches the corresponding first content text for the first fill-in box, the electronic device can further detect that the user triggers another fill-in box, such as the second fill-in box, and match the corresponding content text, such as the second content text, for the second fill-in box. As shown in Figure 4 the electronic device can further perform the following steps:

[0180] S304: The electronic device determines, from the plurality of label boxes, a second label box matched with the second fill-in box in response to a second operation on the second fill-in box in the first interface; the second label box is different from the first label box, and the second label box has a second label name.

[0181] As shown in Figure 6 (5) in FIG. (4), after the electronic device matches the first content text for the first fill-in box 403C, the electronic device can further receive a click operation of the user on 403D and determine that 403D is a second fill-in box to be filled in.

[0182] As shown in (5) in FIG. (4), the electronic device determines the matched second label box 402D for the second fill-in box 403D.

[0183] S305: The electronic device displays the second content text in the associated area of the second fill-in box; the label attribute of the second content text matches the second label name.

[0184] Figure 6 As shown in Figure 7 (5) in FIG. (4), the electronic device extracts the second label name filled in the second label box 402D as “detailed address”, and determines the matched second content text for the second fill-in box 403D according to the “detailed address”, and the label attribute of the second content text is “detailed address”. As shown in (6) in FIG. (4), the content text 406 matched by the electronic device for the second fill-in box is “D Street E Community F Unit”. If the electronic device receives a touch operation on the content text 406, the electronic device fills the content text 406 into the second fill-in box 403D as the second content text.

[0185] It should be noted that the electronic device displaying the fill-in box can be in the form of a control, and the control can respond to the touch operation of the user. After the electronic device fills in the second fill-in box, the electronic device can further receive a touch operation of the user on the second fill-in box, and add, modify, or delete the second content text filled in the second fill-in box.

[0186] The electronic device performs S301 and S302 to determine the first fill-in box to be filled in and the first label box matched with the first fill-in box, and the electronic device performs S304 and S305 to determine the second fill-in box to be filled in and the second label box matched with the second fill-in box, both of which involve the recognition and matching operations of the electronic device on the table boxes and texts in the table. The specific implementation schemes will be explained in detail through several embodiments.

[0187] As shown in FIG. 6, an element matching schematic diagram involved in the information processing method provided by the embodiment of the present application. The electronic device recognizes the cursor position (X, Y) to determine the first fill-in box 601 to be filled in. The electronic device includes an information extraction module and a content matching module, wherein the information extraction module extracts information of the first interface, such as scene information, filled information and to-be-filled information, and the content matching module is configured to match the corresponding content text for the to-be-filled information. Figure 8 For example, as shown in FIG. 6, the information extraction module of the electronic device extracts the title, title and other information of the first interface to obtain the scene information of the table, such as “XX address”. The electronic device can also extract the filled information, such as “Zhang San”, and the label attribute of the filled information “Zhang San” is “name”. The electronic device extracts the to-be-filled information, and the label attribute of the to-be-filled information is “telephone”.

[0188] Figure 8 Based on this, the content matching module of the electronic device matches the content text with the label attribute “telephone” for the to-be-filled information, such as “177****6666”. The electronic device displays the determined content text 602 in the associated area of the first fill-in box 601 to be filled in.

[0189] It should be noted that the content text determined by the electronic device for the first fill-in box to be filled in can be determined directly based on the label attribute of the content text, for example, according to the label attribute “telephone”, it is determined that the matched content text is “177****6666”. The electronic device can also determine the matched content text based on the label attribute of the element and the filled information. For example, the electronic device determines the content text “177****6666” matched with “Zhang San” and “telephone” according to the label attribute “telephone” and the filled information “Zhang San”.

[0190] In one specific embodiment, the electronic device can recognize a plurality of table boxes and filled elements in the first interface through a pre-trained neural network model, determine the first fill-in box to be filled in through the cursor position, and find the first label box matched with the first fill-in box from a plurality of label boxes through a pre-configured first algorithm.

[0191] As shown in FIG. 6, an element matching schematic diagram involved in the information processing method provided by the embodiment of the present application. The electronic device recognizes the cursor position (X, Y) to determine the first fill-in box 601 to be filled in. The electronic device includes an information extraction module and a content matching module, wherein the information extraction module extracts information of the first interface, such as scene information, filled information and to-be-filled information, and the content matching module is configured to match the corresponding content text for the to-be-filled information.

[0192] As shown in FIG. 6, an element matching schematic diagram involved in the information processing method provided by the embodiment of the present application. The electronic device recognizes the cursor position (X, Y) to determine the first fill-in box 601 to be filled in. The electronic device includes an information extraction module and a content matching module, wherein the information extraction module extracts information of the first interface, such as scene information, filled information and to-be-filled information, and the content matching module is configured to match the corresponding content text for the to-be-filled information. Figure 8 ​As shown, the automatic form filling process involved in the information processing method provided by the embodiment of the present application is shown. The automatic form filling process performed by the electronic device mainly includes the following steps:

[0193] S701: The electronic device obtains a first screenshot of a first interface.

[0194] As shown in (1) of Figure 9 The electronic device obtains a screenshot of the first interface, denoted as a first screenshot. The first screenshot can be a screenshot of the entire interface of the first interface.

[0195] In one case, the electronic device displays the first interface, displays a form in a part of the interface region on the first interface, or newly adds a pop-up window to display the form on the first interface. The electronic device can also take a screenshot of only the part of the interface region or the pop-up window displaying the form as the first screenshot.

[0196] S702: The electronic device obtains the cursor position.

[0197] The electronic device can recognize the cursor feature in the first screenshot by performing image feature extraction on the first screenshot to obtain the cursor position.

[0198] S703: The electronic device crops the first screenshot to obtain a second screenshot according to the cursor position.

[0199] In the automatic form filling process of the edge-selecting and edge-filling scenario, the electronic device needs to analyze and match the current fill-in box to be filled in, and the filled-in fill-in box will be referred to in the analysis and matching process. In the first interface, the filled-in fill-in box is usually located above the unfilled fill-in box. The electronic device can only retain the table above the cursor position, which can obtain the relevant information required for automatic form filling and reduce unnecessary calculation.

[0200] As shown in (1) of Figure 9 The electronic device obtains the first screenshot, and retains the image features above the cursor position in the first screenshot after obtaining the cursor position, and crops the first screenshot to obtain a second screenshot, as shown in (2) of Figure 8 .

[0201] S704: The electronic device performs first type detection on the second screenshot to recognize the text bounding box and text information in the second screenshot.

[0202] S705: The electronic device performs second type detection on the second screenshot to recognize the table box and table box type in the second screenshot.

[0203] The electronic device performs first type detection and second type detection on the second screenshot, the first type detection is used to detect text information in the table, and determines a text bounding box (or a text box) corresponding to each piece of text information. The second type detection is used to detect a table box in the table. The order in which the electronic device performs the first detection and the second detection can not be limited, for example, the electronic device can simultaneously perform the first type detection and the second type detection.

[0204] The electronic device performs S704, and performs first type detection on the second screenshot. The first type detection can be optical character recognition (OCR) on the second screenshot by using an OCR technology.

[0205] The OCR technology converts the analyzed text information into a format recognizable by a computer by scanning, processing, and analyzing the text information in the image. The basic steps of the OCR technology include image preprocessing, feature extraction, character recognition, and post-processing. Preprocessing usually involves operations such as denoising, binarization, image enhancement, etc., to improve image quality and clarity. Feature extraction is to extract text features such as shape, size, inclination, etc. through image analysis. Character recognition uses these feature information to recognize and classify characters, and post-processing includes operations such as correcting text position and correcting wrong characters.

[0206] The electronic device performs first type detection on the second screenshot, and can also recognize a text box corresponding to each piece of text information. As shown in Figure 9 The table includes a table box 901 and a table box 902, and text information 903 and text information 904. The electronic device can recognize the text information 903 and the text information 904, and a text bounding box 905 corresponding to the text information 903 and a text bounding box 906 corresponding to the text information 904 by the first type detection.

[0207] The electronic device performs S705, and performs second type detection on the second screenshot, which can be implemented by using a first model. The first model can be a visual detection model or a model with table box recognition capability, for example, a yolo model. The first model has the capability of recognizing a table box from an input image, obtaining coordinates of each table box in the input image, and obtaining a table box type.

[0208] The electronic device inputs the second screenshot into the first model to obtain coordinates of each table box in the second screenshot. For example Figure 9 As shown in the figure, the electronic device inputs the second screenshot into the first model, recognizes the table box 901 and the table box 902, and obtains coordinates of four vertices (A, B, C, and D) of the table box 901 in the second screenshot and coordinates of four vertices (a, b, c, and d) of the table box 902 in the second screenshot.

[0209] And the electronic device inputs the second screenshot into the first model, identifies a plurality of table frames present in the second screenshot, and determines a table frame type of each table frame according to a presence state or an input mode of an element in each table frame. The table frame types obtained by the electronic device using the first model can include a label frame and a fill-in frame, and the fill-in frame can be further divided into an unfilled input box, an unfilled drop-down option box, a filled content text input box, and a filled drop-down option box.

[0210] The label frame refers to a table frame filled with a label name, and the label frame can also be defined as a prompt box. The unfilled input box is a fill-in frame without content text and with an input mode, the unfilled drop-down option box is a fill-in frame without content text and with a drop-down option mode, the filled input box is a fill-in frame with content text and with an input mode, and the filled drop-down option box is a fill-in frame with content text and with an input mode. In some cases, a fill-in frame can support an input and a drop-down option, and in this case, the fill-in frame can be confirmed as an input box.

[0211] As shown in the second screenshot in (2) of FIG. 9, Figure 5 As shown in the second screenshot in (2) of FIG. 9,

[0212] S706: The electronic device matches the text bounding box with the table frame and matches the text information with the table frame.

[0213] As shown in FIG. 9, Figure 10 As shown in FIG. 9,

[0214] In an example, the electronic device can use a center point matching algorithm to determine the text bounding box matched with each table frame.

[0215] Specifically, the electronic device obtains a text box center point of each text bounding box, finds a table frame in the plurality of table frames, and determines the table frame as a table frame matched with the text bounding box, if a table frame coordinate range of the table frame includes the text box center point coordinate. In a specific implementation, the electronic device can determine the table frame coordinate range by using coordinates of two opposite vertices of each table frame, and determine whether the table frame coordinate range includes the text box center point coordinate.

[0216] For example, the electronic device determines the table frame coordinate range according to coordinates of a top-left vertex and a bottom-left vertex of the table frame. As shown in FIG. 10, the table frame coordinate range of the table frame 1001 includes the text box center point 1002, and the table frame coordinate range of the table frame 1003 does not include the text box center point 1002.Figure 11 As shown, the electronic device obtains the text box center point O1 (X O1 , Y O1 ) of the text bounding box 905, the coordinates of the four vertices ABCD of the table box 901, and the coordinates of the four vertices abcd of the table box 902.

[0217] In one case, the electronic device obtains the coordinates (X A , Y A ) of the top-left vertex A of the table box 901 and the coordinates (X D , Y D ) of the bottom-right vertex D, and determines whether the coordinates of the text box center point O1 and the table box coordinate range of the table box 901 satisfy one of the following conditions:

[0218] (X A ≤X O1 )&(X D ≥X O1 )&(Y A ≥Y O1 )&(Y D ≤Y O1 ), or (X A ≤X O1 ≤X D )&(Y A ≥Y O1 ≥Y D ).

[0219] In a case where it is determined that the coordinates of the text box center point O1 and the table box coordinate range of the table box 901 satisfy the above conditions, the electronic device determines that the text bounding box 905 matches the table box 901, that is, the text information in the table box 901 is “name”.

[0220] In another case, the electronic device obtains the coordinates (X a , Y a ) of the top-left vertex a of the table box 902 and the coordinates (X d , Y d ) of the bottom-right vertex D, and determines whether the coordinates of the text box center point O1 and the table box coordinate range of the table box 902 satisfy one of the following conditions:

[0221] (X a ≤X O1 )&(X d ≥X O1 )&(Y a ≥Y O1 )&(Y d ≤Y O1 ), or (X a ≤X O1 ≤X d )&(Ya ≥Y O1 ≥Y d ).

[0222] If the coordinates of the center point O1 of the text box and the coordinate range of the table frame 902 do not meet the above conditions, the electronic device determines that the text outer box 905 and the table frame 902 do not match, that is, the text information in the table frame 901 is not "name".

[0223] In another scenario, the electronic device can obtain the coordinates (X) of the top-left vertex a of frame 902. a The coordinates (X,Ya) and the coordinates of the lower right vertex D. d (Yd), determine whether the coordinates of the center point O2 of the text box of the text outer box 906 and the coordinate range of the table frame of the table frame 902 satisfy one of the following conditions:

[0224] (X a ≤X O2 )&(X d ≥X O2 )&(Y a ≥Y O2 )&(Y d ≤Y O2 ), or (X a ≤X O2 ≤X d )&(Y a ≥Y O2 ≥Y d ).

[0225] When the electronic device determines that the coordinates of the center point O2 of the text box and the coordinate range of the table frame 902 meet the above conditions, it determines that the text outer box 906 and the table frame 902 match, that is, the text information in the table frame 901 is "Please enter your name".

[0226] It should be noted that the above example uses the top-left and bottom-right vertices of the text outer box as factors in calculating the table frame coordinate range. In other cases, the bottom-left and top-right vertices of the text outer box can also be used as the calculated coordinate values ​​of the table frame, or all four vertices of each text outer box can be used as factors in calculating the table frame coordinate range; there are no restrictions.

[0227] In another specific implementation, the electronic device may employ a table label and fill box pairing algorithm based on two-dimensional Gaussian distribution modeling.

[0228] like Figure 11The relative position relationship of the two table frame groups shown can cover the relative position relationship of the label frame and the fill-in frame in a large number of forms. The electronic device can obtain a large number of form samples, according to the relative position relationship between the label frame and the fill-in frame in the plurality of table frame groups in the form samples, use two-dimensional Gaussian distribution modeling, count the relative distribution between the center point coordinates of the fill-in frame and the center point coordinates of the label frame, and calculate the probability of matching between the fill-in frame and each label frame in the plurality of label frames based on the relative distribution. For example Figure 12 As shown, the relative position of the fill-in frame and the matching label frame is shown, the X-axis coordinate difference of the label frame and the fill-in frame is negative, the Y-axis coordinate difference is 0 or positive, and the greater the probability of matching between the label frame and the fill-in frame; on the contrary, the X-axis coordinate of the label frame and the fill-in frame is positive, or the Y-axis coordinate is negative, and the smaller the probability of matching between the label frame and the fill-in frame.

[0229] In this way, the electronic device can realize high precision and low latency for matching the fill-in frame to the label frame.

[0230] The electronic device based on Gaussian distribution modeling, the scheme for matching the fill-in frame to the table frame mainly includes the following steps:

[0231] Step 1, the electronic device pre-processes the second screenshot.

[0232] The electronic device identifies the label frame and the fill-in frame in the second screenshot using the first model, and records the center point coordinates of each label frame and each fill-in frame.

[0233] Step 2, the electronic device constructs a two-dimensional Gaussian distribution statistical model

[0234] The electronic device counts the relative position distribution of the center point coordinates of the label frame and the center point coordinates of the fill-in frame from a large number of form samples, and defines the relative position vector d ij = C Ij -C Ti . Wherein C Ti is the center point coordinates of the label frame, C Ti = (x Ti , y Ti ), C Ij represents the center point coordinates of the fill-in frame, C Ij = (x Ij , y Ij ). Based on this, the relative position vector d ij = (x Ij -x Ti , y Ij -y Ti ), the value corresponding to the relative position vector is the center distance between the label frame and the fill-in frame, that is, the distance between the center point of the matching label frame and the center point of the matching fill-in frame.

[0235] The electronic device constructs a two-dimensional Gaussian distribution model p(d ij ) based on the distribution result of the relative position vectors of a large number of table samples.

[0236]

[0237] wherein μ represents the mean, ε represents the covariance matrix, exp represents the natural exponential function, and T represents the transpose matrix.

[0238] Step 3: The electronic device constructs a first candidate set, and the first candidate set includes candidate label boxes with a center distance from the center of the first fill-in box within a first threshold.

[0239] The electronic device calculates the center distance from other candidate boxes for each fill-in box I j , finds candidate label boxes T i with a center distance within a pre-defined first threshold r, forms a candidate set T j , and T j = T i ||d ij |≤r. For example, the first threshold r can be less than or equal to 100 pixels.

[0240] Step 4: The electronic device determines the probability of matching each candidate label box to the fill-in box.

[0241] The electronic device calculates the probability of matching each candidate label box T i in the candidate set to the fill-in box I j , and the calculation formula is:

[0242] The electronic device obtains the probability of each candidate label box in the candidate set relative to the fill-in box, and selects the label box with the maximum probability as the paired label box of the fill-in box I j , wherein

[0243] Based on this, the electronic device can determine the matched first label box for the first fill-in box indicated by the user. The electronic device constructs a Gaussian model by analyzing the relative position distribution of the matched fill-in box and label box in a large number of table samples, and finds the first label box matched with the first fill-in box according to the Gaussian model. The electronic device finds the matched first label box for the first fill-in box by constructing a Gaussian model, which has less calculation and higher accuracy.

[0244] S77: The electronic device obtains the first label name matched with the first fill-in box corresponding to the cursor position.

[0245] S78: The electronic device outputs the first content text matched with the first label name.

[0246] The electronic device matches the content text to the first fill-in box according to the first label name of the first label box.

[0247] In a specific implementation, the electronic device can maintain a table information library, which stores content text and corresponding label names for filling in a table. The electronic device determines the first label name corresponding to the first fill-in box, finds the content text matched with the first label name from the table information library, and takes the content text as the first content text matched with the first fill-in box. The electronic device fills the first content text into the first fill-in box of the first interface. In this way, the operation of manually inputting the content for filling in a table can be saved, and the user experience of filling in a table can be optimized.

[0248] On the basis of the above-mentioned embodiments, the electronic device can also detect whether the current scenario is a table filling scenario by recognizing the cursor trajectory before executing the automatic table filling process, and execute the subsequent automatic table filling process in the case of determining that the current scenario is a table filling scenario.

[0249] As shown in Figure 2 , a table box coordinate schematic diagram involved in the information processing method provided by the embodiments of the present application. The table includes fill-in box 1101, fill-in box 1102, and fill-in box 1103. The vertical coordinate of fill-in box 1101 is y1, the vertical coordinate of fill-in box 1102 is y2, and the vertical coordinate of fill-in box 1103 is y3. The electronic device can detect the region of the current input operation of the user based on the cursor coordinate.

[0250] In one case, the user touch operation received by the electronic device is a horizontal input operation or a line change operation in a fill-in box, and the y value of the cursor changes little.

[0251] In another case, the user touch operation received by the electronic device is switching from one fill-in box to another fill-in box, and the y value of the cursor changes greatly. As shown in Figure 2 , the electronic device switches from fill-in box 1101 to fill-in box 1102, and the y value changes by Δy1, Δy1=y1-y2. The electronic device switches from fill-in box 1102 to fill-in box 1103, and the y value changes by Δy2, Δy2=y2-y3. Based on this, the electronic device can set a second threshold value, and the second threshold value ∈ y , the electronic device confirms the line change table filling in the case where the y value change exceeds the second threshold value.

[0252] In one example, the electronic device provides a low-power listening strategy to listen whether it is in a table filling scenario.

[0253] For example, the electronic device listens to the cursor coordinate change and calculates the Y value change of the cursor coordinate within the initial listening time T init . When |Δy i | = |yi+1 -y i |>∈ y When the absolute value of the change of the Y value is greater than the error threshold, the electronic device confirms that the line is filled in and enters a continuous listening state.

[0254] The electronic device listens to the cursor coordinates and counts the cursor trajectory features. The cursor trajectory of the electronic device is processed in segments, and the cursor state is determined according to the number of segments.

[0255] The electronic device segments the sequence of cursor movement according to the change in the Y value, and each segment is considered as a line of input process. The average value of the change in the X value in each segment is counted, and the average value of the Y value of each segment is calculated, and then the average value of the change in the Y value between each segment is calculated.

[0256] The electronic device first segments the sequence of cursor movement.

[0257] The electronic device counts the trajectory of the cursor in the time sequence T as where i represents the time step, and n represents the total number of time steps. After determining the start of continuous listening according to the change in the Y value, the electronic device continuously listens and determines whether the change in the Y value is greater than the error threshold y When the absolute value of the change of the Y value is greater than the error threshold, the electronic device confirms that the line is filled in and enters a continuous listening state. i |>∈ y When the absolute value of the change of the Y value is greater than the error threshold, the electronic device confirms that the line is filled in and enters a continuous listening state. where k is the segment number.

[0258] When the number of segments K is greater than a specified value N segments , the electronic device calculates the average value of the change in the X value in each segment and the average value of the change in the Y value between each segment

[0259] The electronic device calculates the average value of the change in the X value in each segment The formula includes:

[0260] The electronic device calculates the average value of the Y value in each segment The formula includes:

[0261] The electronic device calculates the average value of the change in the Y value between each segment based on the average value of the change in the X value in each segment and the average value of the Y value in each segment y The calculation formula includes:

[0262] The electronic device obtains the average value of the change in the X value in each segment If the average value of the change in the X value in all segments is , it indicates that the cursor moves from left to right in each segment, that is, the electronic device responds to user operations to input content text in the fill-in box from left to right in sequence.

[0263] The electronic device obtains a mean value Δμ of Y value changes between each segment y , Δμ y > ΔY threshold , where ΔY threshold is a preset Y-axis interval threshold value, it indicates that the Y-axis interval of the input box of each segment is large, which is different from inputting long text.

[0264] If the electronic device determines that the following conditions are met in each segment and Δμ y > ΔY threshold , it is determined as a form filling scene. In the case that the electronic device determines that the current scene is a form filling scene by listening to the cursor trajectory, it can trigger automatic form filling, saving the user's operation of triggering automatic form filling, and further improving the convenience of the user's form filling operation.

[0265] The information processing method provided by the above embodiments of the present application can identify a form filling scene based on a cursor, and the electronic device automatically triggers automatic form filling. The electronic device can also trigger automatic form filling based on the user's touch operation. The electronic device starts the automatic form filling process, determines the filling box to be filled, matches the label box for the filling box, matches the content text for the filling box according to the label name of the label box, saves the user's operation of manually inputting the content text, and improves the convenience of the user's form filling operation.

[0266] In embodiment 3, the automatic form filling mode executed by the electronic device is one-key filling.

[0267] The present application provides an information processing method, which supports automatic filling of all contents of a form. The method includes: displaying a first interface; the first interface includes a form and a first control, and does not include a second control for inputting or triggering display of first information; the form includes a first form label and a to-be-filled control, and the first form label is a form label associated with the to-be-filled control in the first interface. The first information includes all information to be filled in the first interface. In response to a user operation on the first control, the to-be-filled content matched with the first form label is filled into the to-be-filled control based on the first form label. In this way, the electronic device can realize one-key automatic filling of the form in the first interface in response to one user operation, which can greatly improve the efficiency of filling the form.

[0268] In some embodiments of the present application, during the execution of the form automatic filling, the electronic device can display prompt information for prompting the user that the current is automatically filling a form.

[0269] According to different filling manners, the filling controls in the table can be divided into two different types. One type is a filling control that needs to be filled in by input, referred to as an input control. The other type is a filling control that provides a drop-down menu and needs the user to select appropriate content in the drop-down menu for filling, referred to as a drop-down control. In some embodiments, the table automatic filling process can implement automatic filling of empty input controls and empty drop-down controls. That is, the above-mentioned empty filling control specifically refers to an empty filling control, and can also be an empty drop-down control. The empty filling control can also be named as other names, such as a to-be-filled control, and the like.

[0270] In the embodiments of the present application, the table automatic filling process can be divided into the following parts: a visual detection part, a to-be-filled content acquisition part, a to-be-filled content maintenance part, and a filling execution part.

[0271] The visual detection part is used to perform visual detection on the first interface after triggering the table automatic filling process, to determine whether the first interface includes a table, and to determine, when the first interface includes a table, an empty filling control (including an input control and / or a drop-down control), a display position of the empty filling control, a table label associated with the empty filling control, a display position of the table label, and the like. The to-be-filled content acquisition part is used to acquire to-be-filled content corresponding to the empty filling control when the first interface includes a table. The to-be-filled content maintenance part is used to maintain and update the acquired to-be-filled content in the process of performing a filling operation. For example, the mobile phone can maintain and update the to-be-filled content according to the filling progress of the current table, such as deleting to-be-filled content that has been filled in, or updating coordinate information of an input control corresponding to the to-be-filled content when table displacement occurs, and the like. The filling execution part is used to perform a filling operation and automatically fill the acquired to-be-filled content into the corresponding empty filling control.

[0272] In other embodiments, the table automatic filling process in the embodiments of the present application further includes an exception handling part. The exception handling part is used to detect whether an exception occurs in the process of performing a filling operation, and to perform exception handling for the occurred exception. Specifically, the exception handling part is used to detect whether an exception occurs after the filling operation for one empty filling control is completed.

[0273] The information processing method proposed in the embodiments of the present application is described in detail below with reference to the accompanying drawings. Figure 13 The specific process of the information processing method in some embodiments is shown.

[0274] S1201. In response to a user operation, a table in a first interface is identified.

[0275] The user operation can trigger the automatic form filling process. The first interface includes the form and the first control, and does not include a second control for inputting or triggering display of the first information. The first information includes all information to be filled in the first interface. In some embodiments, the user operation can correspond to Figure 14 The user triggers the one-key filling control. The one-key filling control can be the first control.

[0276] In some embodiments, S1201 can specifically include: in response to the user operation, the mobile phone acquires a first interface screenshot. Then, the first interface screenshot is recognized to obtain the form therein. The acquisition of the first interface screenshot can be implemented in any manner.

[0277] Optionally, as an example, the mobile phone can implement the recognition of the form in the first interface through a model. For example, the mobile phone recognizes the first interface screenshot through a first model to obtain the form in the first interface. It should be noted that the specific implementation of obtaining the form through the first model will be described in detail in the following embodiments. It should be noted that the first model can also be named as other names, such as a visual detection model.

[0278] The mobile phone recognizes the first interface to determine whether the first interface includes the form, and in the case that the first interface includes the form, to recognize and obtain each form label in the form and the filling control associated with each form label. Further, in the case that the first interface includes the form, the mobile phone can recognize and determine the label region corresponding to each form label and the filling region corresponding to the filling control associated with each form label, to locate the display positions of the form label and the filling control in the first interface, so as to facilitate the automatic filling of the form according to the display positions of the form label and the filling control in the subsequent process. Common label regions and filling regions are rectangular regions, and the rectangular region can represent the display position of the rectangular region in the first interface by a bounding box (bbox); such as a label box and a filling box. For example, in the case that the first interface includes the form, the mobile phone can acquire the coordinate information of the bounding box corresponding to the label region and the coordinate information of the bounding box corresponding to the filling region.

[0279] The mobile phone identifies the table in the first interface, and can obtain all table labels in the table in the first interface and filling controls associated with each table label. The specific implementation of the mobile phone identifying the table in the first interface and determining the association between the table label and the filling control can refer to the description in other embodiments or the description in related technologies. In an actual scenario, the table in the first interface can include some filled filling controls (i.e., filled controls) and some unfilled filling controls, i.e., empty filling controls. It can be understood that in the embodiments of the present application, the mobile phone only needs to automatically fill the empty filling controls when performing the table automatic filling process. Therefore, after identifying the table in the first interface, the empty filling controls in the table and the first table label associated with the empty filling controls can be obtained, so as to automatically fill the unfilled filling controls; for example, S1202.

[0280] S1202. Obtain the empty filling control in the table and the first table label associated with the empty filling control.

[0281] The empty filling control is a filling control to be filled in the table, and the first table label is used to indicate the type of content to be filled in the empty filling control associated with the first table label. In the embodiments of the present application, the empty filling control is specifically an input control and / or an empty drop-down control to be filled in the table.

[0282] In some embodiments, S1202 can specifically include obtaining coordinate information of the empty filling control and coordinate information of the first table label associated with the empty filling control. The coordinate information of the empty filling control is specifically coordinate information of a filling area corresponding to the empty filling control; and the coordinate information of the first table label is specifically coordinate information of a label area corresponding to the first table label. For example, the information obtained in S1202 can be represented as: filling control 1: coordinate information of filling area 1 and coordinate information of label area 1 associated with filling control 1; filling control 2: coordinate information of filling area 2 and coordinate information of label area 2 associated with filling control 2; … filling control n: coordinate information of filling area n and coordinate information of label area n associated with filling control n. Subsequently, the filling operation can be performed on the empty filling control based on the coordinate information of the filling area and the coordinate information of the label area associated with the filling area.

[0283] S1203. Obtain the to-be-filled content matched with the first table label from the user information library.

[0284] To determine what the empty filling control should fill in, after obtaining the coordinate information where the first table label associated with the empty filling control is located in S1202, it is necessary to identify what the first table label is, so as to find the corresponding to-be-filled content. In some embodiments, S1203 can specifically include: identifying the label text corresponding to the first table label, and obtaining the to-be-filled content matching the first table label from the user information library based on the label text.

[0285] In some embodiments, the label text corresponding to the first table label can be identified specifically by using a character recognition technology. For example, the mobile phone can identify the label text corresponding to the first table label by using an optical character recognition (OCR) model.

[0286] The user information library stores some content to be filled in a table, which can be content to be filled in a table commonly used by the user, such as a receiving address, identity information, and a sending address. For example, the user information library stores the following content: a table label and label data associated with the table label. The label data is data to be filled in an empty filling control in a table automatic filling process. In some embodiments, in the user information library, the table label and the label data are stored in the form of a key-value pair; the table label is the key, and the label data is the value. In this embodiment, S1203 includes: obtaining the label data matching the first table label from the user information library, and taking the matched label data as the to-be-filled content.

[0287] The specific implementation of obtaining the to-be-filled content matching the first table label from the user information library will be described in detail in subsequent embodiments.

[0288] S1204. According to the correspondence among the first table label, the empty filling control, and the to-be-filled content, performing a filling operation on each empty filling control to fill the to-be-filled content into the empty filling control.

[0289] In S1202, the association between the empty filling control and the first table label is obtained, and the to-be-filled content is found from the user information library based on the first table label. Therefore, the mobile phone can determine the correspondence among the first table label, the empty filling control, and the to-be-filled content based on the above information. Subsequently, the mobile phone can fill each to-be-filled content into the corresponding empty filling control based on the correspondence among the three.

[0290] In some embodiments, in the S1204 above, the mobile phone performs the filling operation on each empty filling control, and specifically, can perform the filling operation on each empty filling control in sequence. Optionally, as an example, in the S1204 above, the specific implementation logic of performing the filling operation on each empty filling control in sequence is as follows: the mobile phone selects one empty filling control as a target filling control, and performs the filling operation on the target filling control; then selects another empty filling control as a new target filling control, and performs the filling operation on the new target filling control, and so on, until all the empty filling controls are filled with the corresponding to-be-filled content. For example, in the interface 201 shown in FIG. 8, taking the table corresponding to the editing of the recipient address as an example, the empty filling controls include four filling controls associated with the table labels “Name”, “Phone”, “Region” and “Detailed Address” (i.e., the first table label) respectively. When performing the filling operation on each empty filling control in sequence, the mobile phone can first perform the filling operation on the empty filling control associated with the table label “Name”, then perform the filling operation on the empty filling controls associated with the table labels “Phone” and “Region”, and finally perform the filling operation on the empty filling control associated with the table label “Detailed Address”. Figure 15

[0291] In some embodiments, performing the filling operation on the target filling control can specifically include: performing a simulated click on the target filling control based on the coordinate information of the filling area corresponding to the target filling control. The to-be-filled content matched with the first table label associated with the target filling control is filled into the filling area corresponding to the target filling control. The coordinate information of the filling area corresponding to the target filling control can be obtained in the S1202.

[0292] Generally, after the user performs a click operation on the filling control, the mobile phone displays a keyboard for the user to input, or in other words, the mobile phone enters a keyboard mode. Optionally, in some embodiments of the present application, after the simulated click operation on the target filling control, the mobile phone also displays a keyboard. After the to-be-filled content is filled into the corresponding target filling control, the mobile phone needs to be controlled to exit the keyboard mode, and then another empty filling control can be selected as a new target filling control to perform the filling operation. That is, in some embodiments, during the process of performing the filling operation on the target filling control, after the to-be-filled content is filled into the filling area corresponding to the target filling control, a keyboard exit operation is also performed. Alternatively, in some other embodiments of the present application, after the simulated click operation on the target filling control, the mobile phone enters the keyboard mode, and at the same time, the keyboard can also not be displayed on the display interface.

[0293] It should be noted that the specific implementation of performing the keyboard exit operation can refer to the description in the related art, which is not described herein.

[0294] ​In some embodiments, the implementation of S1204 can include generating content vectors according to the obtained content to be filled, the first table label, and the empty filling control. Then, the content vectors are stored in a preset data structure. The empty filling control in the content vector can be represented by the coordinate information of the filling area corresponding to the empty filling control. Then, the mobile phone performs filling operations on each empty filling control based on the preset data structure.

[0295] In some embodiments, the mobile phone performs filling operations on each empty filling control based on the preset data structure, which can include taking out a content vector, such as a first content vector, from the preset data structure. According to the coordinate information of the filling area in the first content vector, a target filling control is determined. The content to be filled in the first content vector is filled into the target filling control. The step of taking out a content vector from the preset data structure is returned. Then, based on the newly taken out content vector, the filling operation is performed on the next empty filling control until the preset data structure is empty; that is, the filling operation has been completed for all empty filling controls.

[0296] For example, the preset data structure can be a queue, such as a to-be-filled queue.

[0297] The to-be-filled queue can include a plurality of queue elements, each of which stores a content vector including a first indication filling control, an empty filling control, and a content to be filled. The first table label, the empty filling control, and the content to be filled correspond one-to-one. For example, a queue element is composed of (first table label, content to be filled, coordinate information of the filling area corresponding to the empty filling control).

[0298] In other embodiments, in order to better locate the display position of the empty filling control, the composition of the queue element also includes the coordinate information of the first table label, such as: a queue element is composed of (first table label, content to be filled, coordinate information of the label area, coordinate information of the filling area).

[0299] Further, the mobile phone selects an empty fill-in control as a target fill-in control, and performs the fill-in operation on the target fill-in control. The implementation of the fill-in operation is as follows: taking the first queue element in the to-be-filled queue, for example, as queue element A; performing a simulated click operation on the fill-in area A (i.e. the empty fill-in control) according to the coordinate information of the fill-in area A in the queue element A. The content to be filled in the queue element A is filled into the fill-in area A. It can be understood that the empty fill-in control corresponding to the fill-in area A in the queue element A is the target fill-in control for performing the fill-in operation. Since the first queue element in the to-be-filled queue has been taken, the second queue element in the to-be-filled queue is taken, and the first queue element in the to-be-filled queue after taking is taken. After the fill-in operation on the empty fill-in control corresponding to the fill-in area A based on the queue element A is performed, the first queue element in the to-be-filled queue is taken again, for example, as queue element B. Then, the empty fill-in control corresponding to the fill-in area B in the queue element B is taken as a new target fill-in control, and the above steps are repeated to perform the fill-in operation on the new fill-in control based on the queue element B, until the to-be-filled queue is empty. It can be understood that in this embodiment, when the to-be-filled queue is empty, it indicates that the fill-in operation on all empty fill-in controls is completed.

[0300] Through the above method, the fill-in operation is performed on each empty fill-in control based on the to-be-filled queue, and then the fill-in operation on all empty fill-in controls in the table is completed.

[0301] In the technical solution proposed in the embodiments of the present application, the mobile phone can automatically fill in the table in the first interface by one key in response to one user operation, which can greatly improve the efficiency of filling in the table.

[0302] In addition, during the process of moving from the completion of the fill-in operation on one empty fill-in control in the table to the fill-in operation on the next empty fill-in control, some abnormal situations may occur. In order to successfully fill in all empty fill-in controls by one key in response to one user operation, it is necessary to detect whether an abnormal situation occurs, and perform corresponding abnormal handling for the abnormal situation. The process of detecting the abnormal situation and performing the abnormal handling for the abnormal situation is described in detail below based on the implementation of sequentially performing the fill-in operation on each empty fill-in control based on the to-be-filled queue. For example, Figure 14 The corresponding flow is shown.

[0303] In this embodiment, the to-be-filled queue includes n queue elements, and the structure of one queue element is as follows: (k n ,v n ,bbox 表格标签n , bbox 填写控件n ). Wherein, k nlabel text representing a table label, v n representing label data matched with k n 表格标签n representing coordinate information of a label region corresponding to the table label n, bbox 填写控件n representing coordinate information of a fill-in region corresponding to the fill-in control n.

[0304] S1301. Take out the first queue element (k1, v1, bbox 表格标签1 , bbox 填写控件1 ), determine the display position of the fill-in control 1 according to bbox 填写控件1 , and fill v1 into the fill-in control 1.

[0305] The specific implementation of S1 can refer to the description of performing the fill-in operation on the empty fill-in control based on the to-be-filled queue in the above embodiment.

[0306] After taking out the first queue element (k1, v1, bbox 表格标签1 , bbox 填写控件1 ) from the to-be-filled queue, the element position in the to-be-filled queue will change, for example, the second queue element before taking out becomes the first queue element after taking out.

[0307] S1302. Detect whether an abnormal situation occurs.

[0308] In some embodiments, the abnormal situation can include that the keyboard mode is not exited. As can be known from the description of the above embodiment, after filling the to-be-filled content into the fill-in region corresponding to the target fill-in control, the mobile phone will also perform the keyboard exit operation. Therefore, under normal circumstances, after the fill-in operation on the empty fill-in control is completed, the keyboard should be exited.

[0309] Optionally, as an example, the mobile phone can detect whether the abnormal situation that the keyboard mode is not exited by the following manner: the mobile phone identifies whether the first interface includes the keyboard to determine whether the abnormal situation that the keyboard mode is not exited exists. Specifically, if the first interface includes the keyboard, it indicates that the abnormal situation that the keyboard mode is not exited exists. On the contrary, if the first interface does not include the keyboard, it indicates that the abnormal situation that the keyboard mode is not exited does not exist. Wherein, identifying whether the first interface includes the keyboard can be implemented by any manner.

[0310] As another example, in the embodiment in which the mobile phone is in the keyboard mode but does not display the keyboard, the mobile phone can determine whether the abnormal situation that the keyboard mode is not exited exists by acquiring the keyboard mode related information from the background.

[0311] ​In other embodiments, abnormal situations may include table displacement. When a fill-in operation is performed on a fill-in control in a table, the table may shift. After the table shifts, the display position of the fill-in control changes. This change in the display position of the fill-in control may affect the automatic form filling process, causing the automatic fill-in operation to fail. For example, after a simulated click operation is performed on a fill-in control in the table, the cursor is activated; that is, the phone displays the cursor in the fill-in area corresponding to the empty fill-in control. The cursor is used to indicate the current operation position; the cursor position moves with the filled content. Figure 14 After a simulated click is performed on the fill control associated with detailed information, the cursor position is 1401; after the fill operation on the fill control is completed, the cursor position is 1402. In other scenarios, such as Figure 15 In the example shown, the cursor position is 1501 when the cursor is activated; after the cursor is activated, the table will shift. In this example, after the fill operation on the fill control is completed, the cursor position is 1502.

[0312] Optionally, as an example, detecting whether a table has shifted can be achieved as follows: Obtain the initial cursor position when the cursor is active, and the second cursor position after the filling operation on the current control has been completed. Calculate the difference between the y-coordinates of the first and second cursor positions. If the difference is greater than or equal to an error threshold, the table is considered to have shifted. The error threshold can be set according to the actual situation; specifically, the error threshold is a value greater than 0. The current filling control refers to the currently selected filling control that has completed the filling operation, i.e., the filling control where the cursor is located.

[0313] like Figure 15 In the example, the first cursor position is 1401, and the second cursor position is 1402. These two positions have the same ordinate, and the difference is less than the error threshold; that is, in... Figure 14 In the example, the table was not shifted. However, in... Figure 16 In the example shown, the first cursor position is 1501, and the second cursor position is 1502. These two positions have different ordinates. The difference is greater than or equal to the error threshold; that is, in... Figure 17 The table in the example has been shifted.

[0314] If the result of S1302 is negative, it means that no abnormal situation has occurred, and S1304 can be executed directly. If the result of S1302 is positive, it means that an abnormal situation has occurred, so S1303 is executed first, and then S1304 is executed.

[0315] S1303. Execute exception handling.

[0316] The main purpose of the abnormal handling for the keyboard mode not being exited is to control the mobile phone to exit the keyboard mode.

[0317] In the case that the mobile phone is in the keyboard mode, a common operation to exit the keyboard is to click the area corresponding to the non-filling control outside the keyboard in the first interface, such as the area corresponding to the empty area in the first interface or the table label in the table. In some embodiments, the abnormal handling for the keyboard mode not being exited can include determining the first table label associated with the current filling control, obtaining the coordinate information of the label area corresponding to the first table label, and performing a simulated click operation in the label area based on the coordinate information of the label area.

[0318] In the case that the mobile phone is in the keyboard mode, another common operation to exit the keyboard is to click the control corresponding to the complete key or the enter key in the keyboard area in the first interface. For example, the control of the complete key can be displayed as the word "complete" / "next item" / …, or as the symbol "√" in the keyboard as shown in FIG. 8B, and so on. Similarly, the control of the enter key can be displayed as the word "enter" / "enter" / …, or as the enter symbol. In other embodiments, the abnormal handling for the keyboard mode not being exited can include obtaining the coordinate information of the preset control in the keyboard area in the first interface, and performing a simulated click operation on the preset control based on the coordinate information of the preset control. The preset control can be the control corresponding to the complete key or the enter key. Figure 18 In the case of abnormality that the table is shifted, the display position of the empty filling control which has not performed the filling operation yet is changed, and the to-be-filled queue generated according to the to-be-filled content before this time cannot correspond to the shifted table, so the empty filling control which has not performed the filling operation yet cannot complete the automatic filling based on the to-be-filled queue. Therefore, the main purpose of the abnormal handling for the table being shifted is to continue to perform the filling operation on the empty filling control behind.

[0319]

[0320] ​In some embodiments, the abnormality handling for the table displacement includes: after the table displacement, re-acquiring the to-be-filled content in the displaced table, and generating a new to-be-filled queue. That is, after determining that the table is displaced, the displaced table is re-identified, and the empty filling control in the displaced table and the table label (denoted as the second table label) associated with the empty filling control are obtained. Then, the new to-be-filled content can be re-acquired from the user database based on the second table label in the displaced table, and a new to-be-filled queue is generated. Finally, the mobile phone can fill the new to-be-filled content into the empty filling control in the displaced table based on the new to-be-filled queue. The specific implementation of identifying the displaced table, obtaining the to-be-filled content, and filling the to-be-filled content into the empty filling control is consistent with the corresponding operation specific implementation in the above-mentioned embodiments, and will not be repeated here.

[0321] In other embodiments, the abnormality handling for the table displacement includes: after the table displacement, updating the to-be-filled queue based on the displaced table. Generally, in the case of table displacement, the displaced table and the table before displacement itself only have the display positions of the table labels and the filling controls changed, and the table labels and the filling controls, and the association relationship between the table labels and the filling controls will not change. Among them, only the display positions of the empty filling controls that have not performed the filling operation when the table is displaced may affect the execution of the subsequent filling operation. As can be known from the description of the above-mentioned embodiments, the to-be-filled elements in the to-be-filled queue, the k n The label text of the table label. Therefore, the above-mentioned updating the to-be-filled queue based on the displaced table can specifically include: re-identifying the empty filling controls (denoted as the remaining empty filling controls) in the table, and the table labels (denoted as the remaining table labels) associated with the remaining empty filling controls, obtaining the coordinate information of the remaining empty filling controls, and the label text of the remaining table labels. Then, based on the label text of the remaining table labels, the coordinate information of the remaining empty filling controls is updated to the corresponding queue element of the to-be-filled queue.

[0322] It should be noted that the new to-be-filled queue generated based on the displaced table, and the updated to-be-filled queue correspond to the empty filling controls that have not performed the filling operation after the table displacement.

[0323] In some embodiments, the abnormal situation can include that the keyboard mode is not exited and the table is shifted. Alternatively, as an example, when detecting whether there is an abnormal situation, the keyboard mode can be first detected whether it is not exited, and then the table is detected whether it is shifted when the keyboard mode is exited. When there is an abnormal situation that the keyboard mode is not exited, the abnormal processing for the keyboard mode not exited is first performed, and then whether the keyboard mode is exited is detected again after the abnormal processing for the keyboard mode not exited is performed. If the keyboard mode is exited after the abnormal processing for the keyboard mode not exited is performed, whether the table is shifted is detected. If the keyboard mode is not exited after the abnormal processing for the keyboard mode not exited is performed, the automatic form filling process of the table can be ended, and whether the table is shifted is not detected.

[0324] Further, when the keyboard mode is first detected whether it is exited, if it is detected that the keyboard mode is exited, whether the table is shifted can be further detected. Then, if the table is shifted, the abnormal processing for the table shifted can be performed according to the manner in the above embodiments, the new to-be-filled queue is regenerated, or the to-be-filled queue is updated, and then the filling operation for the next empty filling control can be continued, i.e., S1304 is performed. If it is detected that the keyboard mode is exited and the table is not shifted, the filling operation for the next empty filling control can be directly performed, i.e., S1304 is performed.

[0325] In the embodiments in which the abnormal situation can include that the keyboard mode is not exited and the table is shifted, the result of S1302 is yes, corresponding to the following scenario: when the abnormal situation is first detected, the keyboard mode is exited, and the table is not shifted. S1302 above first detects whether the keyboard mode is not exited, if yes, S1303 is performed to perform the abnormal processing for the keyboard mode not exited. Then whether the keyboard mode is not exited is judged, if the keyboard mode is not exited, the automatic form filling process of the table is directly ended. If the keyboard mode is exited, S1302 is performed again to detect whether the table is shifted. If the table is shifted, S1303 is performed again to perform the abnormal processing for the table shifted. It should be noted that after the abnormal processing for the table shifted is performed, S1304 can be directly performed.

[0326] S1304. The first queue element (k2, v2, bbox 表格标签2 , bbox 填写控件2 ) is taken out, the display position of the filling control 2 is determined according to bbox 填写控件2 , and v2 is filled into the filling control 2.

[0327] In the technical solution provided in the embodiments of the present application, after the filling operation on one empty filling control is completed, it is detected whether an abnormal situation occurs. When an abnormal situation occurs, the mobile phone automatically performs corresponding abnormal processing without user operation. In this way, it can be ensured that the mobile phone can automatically complete one-key filling of the entire form.

[0328] In some embodiments, several parts of the above form automatic filling process can be implemented by state machines respectively. The form of state machine is convenient to manage and can realize efficient and accurate form automatic filling. The state machines corresponding to the parts of the above form automatic filling process are described in detail below.

[0329] Each state machine can include two parts, one part is state running action; the state running action part is used for the state machine to execute related logic. The other part is state transition condition; the state transition condition is used to determine the next operation to be performed according to the corresponding judgment condition, such as jumping to another state machine, or ending the form automatic filling process, etc.

[0330] 1. The state machine corresponding to the visual detection part, such as state machine 1 or visual detection state machine, the specific implementation of the state machine 1 is as follows Figure 19 .

[0331] State running action: take a screenshot of the first interface to obtain a first interface screenshot. Use the first model to recognize the first interface screenshot, detect several categories of table elements included in the first interface and give their respective corresponding bbox. In this embodiment, the table elements are divided into five categories, including: ① table label L t ; ② empty input control (i.e. empty filling control) I u ; ③ filled input control I f ; ④ empty drop-down control (i.e. empty drop-down control) D u ; ⑤ filled drop-down control D f .

[0332] And when the mobile phone uses the first model to recognize the first interface screenshot, the table label and the associated input control or drop-down control will be given in the form of a sequence of paired bbox . Each sequence element represents the bbox of a pair of filling controls and the bbox of a table label. Among them, the bbox of the input control is specifically the bbox of the filling area corresponding to the input control; the bbox of the table label is specifically the bbox of the label area corresponding to the table label. It should be noted that this part corresponds to S1201 and S1202 described above. The following takes the input control as an example for description.

[0333] For example, the bbox of the input control is represented as Its upper left corner coordinates are The coordinates of the bottom right corner are The bbox of the table label is represented as Its upper left corner coordinates are The coordinates of the bottom right corner are Paired bbox sequences The input control portion in the code can be represented as:

[0334] in:

[0335]

[0336] Understandable, paired bbox sequences The dropdown control portion is represented similarly to the input control.

[0337] State transition condition: First determine the paired bounding box sequence. Is it empty? If the paired bbox sequence If empty, it means there are currently no fill controls in the table, and the automatic form filling process ends. If the paired bbox sequence... If not empty, then check if the queue Q to be filled is empty. If the paired bounding box sequence... If the field is not empty and the queue Q is empty, it means that no content matching the empty fill control has been retrieved yet; in this case, we can jump to state machine 2. If the paired bbox sequence... If the field is not empty and the queue Q is not empty, it means that a field matching the empty field control has been found; at this point, the process can jump to state machine 3.

[0338] 2. The state machine corresponding to the part where the content to be filled is to be retrieved, such as state machine 2 or the content to be filled retrieval state machine. The specific implementation of state machine 2 is as follows: Figure 20 .

[0339] State execution action: For the empty input control I identified in state machine 1 using visual inspection. u Related table label (the first table label mentioned above) L t and empty dropdown control D u Associated table label L t Combine OCR model to obtain table label L t The corresponding label text. And, the table label L... t The corresponding tag text is used as the key (key, K), and the tag data that matches the tag text corresponding to the table tag Lt is retrieved from the user information database and used as the value (value, V).

[0340] The bbox of an empty input control I u and the bbox of the table label L u associated with the empty input control I t are combined into a key-value pair vector, and the key-value pair vector constitutes a to-be-filled queue Q. The to-be-filled queue Q including the to-be-filled content corresponding to the empty input control can be expressed as:

[0341]

[0342] Similarly, the bbox of an empty drop-down control D u and the bbox of the table label L u associated with the empty drop-down control D t are combined into a key-value pair vector.

[0343] The state running action of the state machine 2 specifically corresponds to S1203 in the above embodiment.

[0344] State transition condition: it is judged whether the to-be-filled queue Q is empty. If the to-be-filled queue Q is not empty, it indicates that the filling operation can be performed; at this time, it is jumped to the state machine 4. If the to-be-filled queue Q is empty, it indicates that no label data matched with the label text corresponding to the table label L t is found, and the table automatic filling cannot be realized, and the table automatic filling process is ended.

[0345] 3. The state machine corresponding to the to-be-filled content maintenance part, such as state machine 3 or to-be-filled content maintenance state machine, the specific implementation of the state machine 3 is as follows: Figure 21 .

[0346] State running action: the bbox of the empty filling control re-detected by the visual detection state machine is obtained. The bbox of the empty filling control in the to-be-filled queue Q is updated to be consistent with the bbox of the empty filling control re-detected by the visual detection state machine. This part corresponds to the process of executing the exception handling when there is a table exception in S1302 and S1303 in the above embodiment.

[0347] State transition condition: after the state running action is executed, it can be jumped to the state machine 4.

[0348] 4. The state machine corresponding to the filling execution part, such as state machine 4 or filling execution state machine, the specific implementation of the state machine 4 is as follows: Figure 22 .

[0349] State running action: it is first judged whether the to-be-filled queue Q is empty. If the to-be-filled queue is empty, it indicates that the empty filling control has been completed automatic filling, and the table automatic filling process is ended.

[0350] On the contrary, if the to-be-filled queue Q is not empty, it indicates that there is still a blank filling control in the first interface that needs to be automatically filled. The following takes the blank input control corresponding to the first queue element as an example for illustration. Take out the first queue element in the to-be-filled queue Q According to the first queue element , determine the top-left vertex coordinate u and the bottom-right vertex coordinate of the bbox of the blank input control I u . Calculate the center coordinate of the blank input control I i :

[0351]

[0352] Perform a simulated click operation on the center coordinate of the blank input control I u , activate the cursor, and obtain the position of the cursor (i.e., the first cursor position mentioned above) f , and inject the value v f in the first queue element into the blank input control I f . The position of the cursor can be specifically the top-left vertex coordinate of the cursor. Finally, control the mobile phone to exit the keyboard mode.

[0353] It should be noted that if the first queue element in the to-be-filled queue Q corresponds to a blank drop-down control, the implementation manner is similar to that of the filling execution process of the blank input control, which will not be described herein.

[0354] The state running action of the state machine 4 corresponds to the process of performing the filling operation on one blank filling control in S1204 in the above embodiment.

[0355] State transition condition: judge again whether the to-be-filled queue Q is empty. If the to-be-filled queue Q is empty, it indicates that all blank filling controls have been automatically filled, and the table automatic filling process ends. Otherwise, jump to the state machine 5.

[0356] 5, the state machine corresponding to the exception handling part. The following takes the abnormal situation including the keyboard mode not being exited and the table being shifted as an example for illustration. The state machine corresponding to the exception handling part includes the state machine 5, the state machine 6 and the state machine 7.

[0357] Among them, the state machine 5 is also called the keyboard exit abnormality detection state machine, which is used to detect whether there is an abnormal situation that the keyboard mode is not exited. And in this embodiment, the keyboard mode not being exited is taken as an example for illustration that the mobile phone displays the keyboard. The specific implementation of the state machine 5 is as follows Figure 23A :

[0358] State running action: detect whether the keyboard exists.

[0359] The state operation of state machine 6 corresponds to the process in the above embodiment for detecting whether there is an abnormal situation where the keyboard mode has not been exited.

[0360] State transition condition: Determine if the keyboard exists. If the keyboard exists, it indicates an exception where keyboard mode has not been exited, and the process jumps to state machine 7. Conversely, if the keyboard does not exist, it indicates no exception where keyboard mode has not been exited, and the process jumps to state machine 6.

[0361] State machine 6, also known as the table shift exception handling state machine, is used to detect whether an exception has occurred in table shifting and, if so, to determine the next action to be taken. The specific implementation of state machine 6 is as follows: Figure 23B :

[0362] Status execution action: Get the cursor position (i.e., the second cursor position mentioned above). according to The cursor position obtained in state machine 4 Calculate the cursor vertical offset value

[0363] The state operation of state machine 6 corresponds to the process of detecting whether the table has shifted in the above embodiment.

[0364] State transition condition: Determine if the cursor's vertical offset value is greater than or equal to the error threshold. If If the cursor's vertical offset is less than the error threshold ε, the automatic table filling process can continue, i.e., jump to state machine 4. Conversely, if the cursor's vertical offset is greater than or equal to the error ε, it indicates that an abnormal situation of table shifting has occurred after the current filling control has been filled. When an abnormal situation of table shifting occurs, the mobile phone can re-identify the shifted table, thus jumping to state machine 1.

[0365] State machine 7, also known as the keyboard exit exception handling state machine, is used to re-execute the keyboard exit operation if the exit fails. The specific implementation of state machine 7 is as follows... Figure 23A :

[0366] Status running action: Execute keyboard exit operation.

[0367] The keyboard exit operation can be performed in two ways: Method 1 and / or Method 2.

[0368] Method 1: Get the cursor position According to location Determine the target click location. Then, perform a simulated click at that location to exit the keyboard.

[0369] For example, the mobile phone can calculate the center coordinates of the bbox of the table label associated with the current fill-in control according to the bbox of the fill-in control and the bbox of the table label, and take the center coordinates of the table label as the target click position. The center coordinates of the bbox of the table label As follows:

[0370]

[0371] Method two: locate the display position of the corresponding control such as the complete key or the enter key on the keyboard through the first model, and then perform a simulated click operation on the display position of the corresponding control such as the complete key or the enter key. This method two exits the keyboard by confirming the input.

[0372] In some embodiments, when an abnormal situation of the keyboard mode not being exited occurs, the failed keyboard to exit can be a secure keyboard. The state running action of the state machine 7 corresponds to the process of the abnormal handling for the abnormal situation of the keyboard mode not being exited in the above embodiments.

[0373] State transition condition: detect again whether the keyboard exists. If it still exists, the automatic table filling process ends. If the keyboard does not exist after the abnormal handling for the abnormal situation of the keyboard mode not being exited, jump to the state machine 6.

[0374] In the above embodiments, the information processing method is implemented through the state machine, which can efficiently and accurately implement one-key automatic filling of the table. One-key filling of multiple fill-in controls in the table can improve the table filling efficiency.

[0375] Further, after completing the table filling, the content in the filled table can also be extracted and updated to the user information library. Therefore, in some embodiments, the above method can also include:

[0376] In response to a user operation, the first model is used to recognize the first interface screenshot, and several categories of table elements included in the table in the first interface are detected and their respective corresponding bboxes are given. It should be noted that the table in the first interface has completed the table filling. Since the content extraction is performed on the filled table in this method, the main extraction is on the filled input control I f and the filled drop-down control D f and their respective associated table labels. In the process of recognizing the table, the association between the table label and the filled input control I f and the association between the table label and the filled drop-down control D f also need to be determined. For example, in the output result of the first model, the association between the table label and the filled input control I fthe association relationship between the table label and the filled-in drop-down control D f The association relationship between the table label and the filled-in drop-down control D

[0377] Further, based on the OCR model, the text of the filled-in input control I f , and the label text in the table label L t associated therewith. Further, the corresponding key-value pair of the filled-in input control I f may be obtained, which can be represented as:

[0378]

[0379] Similarly, based on the OCR model, the text of the filled-in drop-down control D f , and the label text in the table label L t associated therewith can also be obtained. Further, the corresponding key-value pair of the filled-in drop-down control D f may be obtained, which can be represented as:

[0380]

[0381] In order to avoid extracting duplicate table content, after obtaining the corresponding key-value pair of the filled-in input control I f and the corresponding key-value pair of the filled-in drop-down control D f , the phone can also perform a set operation on the two key-value pairs, and the set is the final extracted table content. Specifically, the set operation on the two key-value pairs can be a set operation based on the bbox of the table label: only one associated filled-in control, input control or drop-down control, is retained for the bbox of a table label. Specifically, the set of formula (6) and formula (7) can be represented as:

[0382]

[0383] After obtaining the KV total , the phone can store the KV total to the user information library. Optionally, as an example, the phone stores the KV total to the user information library, and specifically, the KV total may be directly stored to the user information library. As another example, before storing the KV total to the user information library, the phone can also compare the KV total extracted this time with the data in the user information library. If there is duplicate data, only the data in the KV total extracted this time that has been updated can be stored to the user information library.

[0384] In the technical solutions provided in the embodiments of the present application, the first interface is used for screenshot recognition, the control text of the filled-in fill-in control and the label text of the table label associated with the filled-in fill-in control can be extracted, and then the key-value pair of the table label and the fill-in control can be generated. The key-value pair can help update the user information library and maintain the richness and accuracy of the data of the user information library.

[0385] In addition, the embodiments of the present application also provide an information processing method, which is specifically used for extracting the content of a filled-in form. For example, the method can be applied after the user manually fills in the form. For the filled-in form, the detailed description of the content extraction is referred to the above embodiments.

[0386] Based on the exemplary information processing method of the above embodiments, in the process of automatically filling in the form, the electronic device needs to obtain the user information from the user information library and fill the user information into the correct position of the form. Therefore, the electronic device needs to know the position of the fill-in box in the form and the type of the form associated with the fill-in box, so as to obtain the correct user information.

[0387] Exemplarily, the embodiments of the present application provide a form visual detection method, an electronic device, a computer readable storage medium and a computer program product. With the form visual detection module and the non-maximum suppression (NMS) algorithm, the electronic device can not only locate and classify the table label and the fill-in box in the form, but also pair the table label with the fill-in box in the form, so as to realize the automatic filling of the form.

[0388] The electronic device can include but is not limited to: a mobile phone, a tablet computer, a notebook computer, an ultra-mobile personal computer (UMPC), a netbook, a personal digital assistant (PDA), a mobile phone, a wearable device, a vehicle-mounted device, an augmented reality (AR) / virtual reality (VR) device, a smart television, etc. The embodiments of the present application do not limit this.

[0389] The style of the form is various, such as the various styles of forms shown in Figure 23B and Figure 23A The form can be displayed in an interface, and the form has a form structure, that is, the form. The form can include but is not limited to: a form in an application, a web form in a browser, etc., such as an online registration form, filling in a survey questionnaire, filling in a courier information form, etc.

[0390] The table can include one or more targets, and a target category of the target can include a table label and a fill-in box, as shown in Figure 23B and Figure 23A .

[0391] The table label generally refers to an identifying word or symbol in the table for describing the content, attribute or category of the table. For example, the table label can include a label name, or a label name and a label box. The label name can be "name" as shown in Figure 23B and Figure 23A . The label box can be a box surrounding "name" as shown in Figure 23B and Figure 23A .

[0392] The label box mentioned in the embodiments of the present application can also be referred to as a prompt box or a prompt control. The label name mentioned in the embodiments of the present application can also be referred to as label text.

[0393] The fill-in box generally refers to an area where the user can input or edit data. The fill-in box mentioned in the embodiments of the present application can also be referred to as a fill-in control.

[0394] The following describes various specific implementation modes of the fill-in box.

[0395] In some embodiments, as shown in Figure 23B and Figure 23A , the fill-in box can include an unfilled input box, a filled input box, an unfilled drop-down box and a filled drop-down box.

[0396] In some embodiments, as shown in Figure 23B and Figure 23A , the fill-in box can include an input box and a drop-down box. The input box includes an unfilled input box and a filled input box. The drop-down box includes an unfilled drop-down box and a filled drop-down box.

[0397] The input box generally refers to an interactive control that allows the user to input text, numbers and other characters. The drop-down box is also called a drop-down list or a combo box. The drop-down box generally refers to an interactive control that provides limited options for the user to choose from. The input box mentioned in the embodiments of the present application can also be referred to as an input control. The drop-down box mentioned in the embodiments of the present application can also be referred to as a drop-down control.

[0398] In some embodiments, as shown in Figure 23B and Figure 24 , the fill-in box can include an unfilled fill-in box and a filled fill-in box. The unfilled fill-in box includes an unfilled input box and an unfilled drop-down box. The filled fill-in box includes a filled input box and a filled drop-down box.

[0399] It can be seen that in the table, the target category of the target can include five categories, namely, table label, unfilled input box, filled input box, unfilled drop-down box, and filled drop-down box.

[0400] In addition, the fill-in box can further include an information element. When the fill-in control is not filled with content, the information element can be prompt content, such as "Please enter your name here" shown in Figure 24 When the fill-in control is filled with content, the information element can be filled content, such as "Zhao Yiyi" shown in Figure 24 Among them, the prompt content mentioned in the embodiments of the present application can also be referred to as prompt text, and the filled content mentioned in the embodiments of the present application can also be referred to as filled text.

[0401] Of course, the target category of the target can also include two categories, namely, table label and fill-in box.

[0402] Alternatively, the target category of the target can include three categories, namely, table label, input box, and drop-down box.

[0403] Alternatively, the target category of the target can include three categories, namely, table label, unfilled fill-in box, and filled fill-in box.

[0404] Please refer to Figure 24 , Figure 24 for a partial software structure schematic diagram of an electronic device provided by the embodiments of the present application. As shown in Figure 23A The first model in the electronic device can include an image preprocessing module 2411, a table visual detection module 2412, and an NMS algorithm module 2413.

[0405] The first model is used for table visual detection of the table in the interface, and can determine a paired target in the table that has a pairing relationship, so that based on one of the paired targets, the other target in the paired target can be accurately determined to be filled with user information.

[0406] The image preprocessing module 2411 can preprocess parameters such as size, dimension, pixel, or grayscale of the interface to obtain a preprocessed interface. The interface includes a table. For example, Figure 23B The interface can be the first interface in the following text. The format of the preprocessed interface is consistent with the input format of the table visual detection module 2412.

[0407] In some embodiments, the image preprocessing module 2411 can adjust the size or the space occupied by the table interface, which can reduce the storage space and speed up the transmission. In addition, the image preprocessing module 2411 can normalize the table interface, so that the pixel value of the interface is adjusted to a certain interval, which facilitates the subsequent use of the interface and ensures the performance and stability of the interface processing.

[0408] Therefore, the image preprocessing module 2411 can transmit the preprocessed interface to the table visual detection module 2412.

[0409] The image preprocessing module 2411 is a software code or a software module. The image preprocessing module 2411 can be stored in the memory of the electronic device and called by the processor such as the central processing unit (CPU) or the graphics processing unit (GPU) of the electronic device.

[0410] Based on the preprocessed interface, the table visual detection module 2412 can detect a plurality of targets in the table, such as positioning, classification, and pairing, to obtain a plurality of detection results.

[0411] Each detection result corresponds to a set of paired bounding boxes. Each set of paired bounding boxes can include a plurality of bounding boxes, such as two bounding boxes. The plurality of bounding boxes correspond to targets of different target categories in the table.

[0412] For example, in the two bounding boxes, one bounding box is of a table note category, and the other bounding box can be of a fill-in-the-blank category, or the other bounding box can be of a drop-down list or an input box, or the other bounding box can be of an unfilled fill-in-the-blank or a filled fill-in-the-blank, or the other bounding box can be of an unfilled input box, a filled input box, an unfilled drop-down list, or a filled drop-down list.

[0413] Each detection result can include relevant feature information of each bounding box in the set of paired bounding boxes, such as the position and the confidence of each target category. Here, each target category refers to each target category of the target.

[0414] In some examples, the table visual detection module 2412 can extract a feature map from the preprocessed interface. The feature map F can be represented as Generally, R represents a real set, C is the number of channels, H is the height of the feature map, and W is the width of the feature map.

[0415] The table visual detection module 2412 can apply multiple convolution layers on the feature map, i.e., the table visual detection module 2412 performs convolution operations on the feature map, which can reduce the channel number C to the output dimension required by the table visual detection module 2412.

[0416] In some embodiments, the table visual detection module 2412 can represent the process of the above-mentioned convolution operation by the tensor C in Formula Seven:

[0417] C = Conv2D (F, 2 x (4 + K)) Formula Seven

[0418] wherein Conv2D represents a two-dimensional convolution operation; 2 x (4 + K) represents that two bounding boxes are predicted on each spatial position of the feature map, each bounding box contains 4 coordinate values and K confidence values of categories, and the 4 coordinate values represent the position of the corresponding bounding box, such as coordinates and dimensions. For example, the 4 coordinate values include the coordinates of the top-left corner of the bounding box, the width of the bounding box, and the height of the bounding box. K is the number of target categories.

[0419] After the convolution operation, the size of the tensor C output by the table visual detection module 2412 is (2 x (4 + K)) x H x W, i.e., the table visual detection module 2412 can output the related feature information of multiple pairs of bounding boxes on each spatial position of the feature map. In this way, the table visual detection module 2412 can realize the positioning, classification, and pairing of each target in the table.

[0420] Therefore, the table visual detection module 2412 can transmit the related feature information of multiple pairs of bounding boxes to the NMS algorithm module 2413.

[0421] Wherein, the specific implementation of the table visual detection module 2412 is not limited in the embodiments of the present application. In some embodiments, in combination with the functions of the table visual detection module 2412, continue to combine Figure 24 , the table visual detection module 2412 can include a feature extraction module 24121, a feature post-processing module 24122, and a table label pairing detection head module 24123.

[0422] The feature extraction module 24121 can extract features from the preprocessed interface to generate a feature map. The feature extraction module 24121 can transmit the feature map to the feature post-processing module 24122. The feature post-processing module 24122 performs post-processing and feature selection on the feature map to obtain a processed feature map. Therefore, the feature post-processing module 24122 can transmit the processed feature map to the table label pairing detection head module 24123.

[0423] The table label pairing detection head module 24123 can perform positioning, classification and pairing on each target in the table according to the processed feature map, and obtain multiple detection results, i.e., multiple sets of related feature information of the paired bounding boxes. That is, the table label pairing detection head module 24123 can detect each target in the table, and determine the target category, position and pairing relationship of each target.

[0424] Therefore, the table label pairing detection head module 24123 transmits the related feature information of the multiple sets of paired bounding boxes to the NMS algorithm module 2413.

[0425] In the table visual detection module 2412, the feature extraction module 24121 and the feature post-processing module 24122 can be different software modules, or can be integrated in the same software module. The feature extraction module 24121 and the feature post-processing module 24122 can use a conventional feature extraction algorithm, such as a you only look once (YOLO) algorithm. The feature extraction module 24121 can use an algorithm such as a convolutional neural network. The feature post-processing module 24122 can use an algorithm such as a feature pyramid network (FPN).

[0426] The table label pairing detection head module 24123 is a software module, and the table label pairing detection head module 24123 can be separately arranged from the feature extraction module 24121 and the feature post-processing module 24122, or can be integrated with the feature extraction module 24121 and the feature post-processing module 24122.

[0427] The table label pairing detection head module 24123 needs to be trained to realize positioning, classification and pairing of each target in the table. In some embodiments, the table label pairing detection head module 24123 can be trained alone, or can be trained together with the feature extraction module 24121 and the feature post-processing module 24122, i.e., the whole table visual detection module 2412 is trained.

[0428] In some examples, the table visual detection module 2412 is usually trained by using training samples, so that the table label pairing detection head module 24123 in the table visual detection module 2412 can accurately position, classify and pair each target in the table.

[0429] In the training samples, multiple tables are included, and the targets in each table can be labeled, such as position, target category and pairing relationship.

[0430] In addition, when the target category of the target is a fill-in box, the style of the target and / or the number of prompt words of the target can also be labeled, which helps to distinguish whether the target category of the target is a table label or a fill-in box.

[0431] In addition, when the target category of the target is a drop-down box, a symbol indicating that the target is a drop-down box, such as the symbol ">" shown in Figure 25 and Figure 25 , can also be labeled, which helps to distinguish whether the target category of the target is an input box or a drop-down box.

[0432] In addition, when the target category of the target is a fill-in box, the state of whether the target is filled can also be labeled, which helps to distinguish whether the target category of the target is an unfilled fill-in box or a filled fill-in box.

[0433] It should be noted that the information of the labeled target in the training sample is not limited by the embodiments of the present application.

[0434] In other examples, the table visual detection module 2412 is trained by using the training sample in combination with the multi-task loss function, so that the table label pairing detection head module 24123 in the table visual detection module 2412 takes into account the losses brought by positioning, classification and pairing, and can more accurately position, classify and pair each target in the table.

[0435] The NMS algorithm module 2413 can filter out paired bounding boxes with high confidence and low overlap from the multiple groups of paired bounding boxes based on the related feature information of the multiple groups of paired bounding boxes, remove redundant and low-quality paired bounding boxes, and obtain filtered multiple groups of paired bounding boxes, so as to ensure that these groups of paired bounding boxes do not overlap excessively in the same category.

[0436] In this way, the NMS algorithm module 2413 can determine each target in the table and the target that has a pairing relationship with the corresponding target, i.e., multiple groups of paired targets, according to the filtered multiple groups of paired bounding boxes.

[0437] For example, for each table label in the table, the NMS algorithm module 2413 can determine the fill-in box that has a pairing relationship with each table label according to the filtered multiple groups of paired bounding boxes.

[0438] Therefore, the NMS algorithm module 2413 can obtain the pairing relationship and related feature information, such as the position and the confidence of the target category, of the multiple groups of paired targets in the table.

[0439] The NMS algorithm module 2413 is a software code or a software module. The NMS algorithm module 2413 can be set in a memory in an electronic device and called by a processor such as a CPU or a GPU in the electronic device.

[0440] In the following, the embodiments of the present application will be described in detail with the structure shown in Figure 25 The table visual detection method provided by the embodiments of the present application will be described in detail in combination with the drawings and application scenarios.

[0441] Please refer to Figure 26A , Figure 26B The flowchart of a table visual detection method provided by the embodiments of the present application. As shown in Figure 26A The table visual detection method provided by the embodiments of the present application can include:

[0442] S2501, acquire a first interface, the first interface including a plurality of targets of a table.

[0443] In the case that the first interface includes a plurality of targets of a table, the electronic device can acquire the first interface in multiple ways based on the user's filling intention. In some embodiments, the electronic device can intercept an image displayed on the display screen after receiving an operation on the first control or an operation on the hardware button.

[0444] The first control is a virtual control on the image, and the operation on the first control can include but is not limited to touch operations such as single-click, double-click, and swipe. The hardware button is a physical entity in the electronic device, and the operation on the hardware button can include but is not limited to physical pressing operations such as single-click and double-click.

[0445] When the format of the image conforms to the input format of the table visual detection module 2412, the electronic device can determine the image as the first interface.

[0446] When the format of the image does not conform to the input format of the table visual detection module 2412, the electronic device can preprocess the image through the image preprocessing module 2411 to process the format of the image to conform to the input format of the table visual detection module 2412, obtaining a preprocessed interface. Thus, the electronic device can determine the preprocessed interface as the first interface.

[0447] In summary, the electronic device can acquire the first interface based on the user's filling intention.

[0448] S2502, detect the plurality of targets of the table to obtain a plurality of detection results, the detection results including a position of a set of paired bounding boxes and a confidence of each target category, and the set of paired bounding boxes including a plurality of bounding boxes having a pairing relationship.

[0449] The electronic device can perform positioning, classification, pairing and other detection on the targets in the table according to the first interface, obtaining a plurality of detection results.

[0450] Each detection result includes relevant feature information of a set of paired bounding boxes. The set of paired bounding boxes can include a plurality of bounding boxes, and the plurality of bounding boxes have a pairing relationship. In some embodiments, the plurality of bounding boxes can be two bounding boxes, or three or more bounding boxes.

[0451] One detection result includes relevant feature information of all bounding boxes in the paired bounding boxes. The relevant feature information of one bounding box can include a position of the bounding box, and a confidence of each target category of the bounding box. The confidence of one target category refers to a probability that the target belongs to the target category.

[0452] For example, the position of one bounding box can be represented by a coordinate of one vertex of the bounding box, a width of the bounding box, and a height of the bounding box. Alternatively, the position of one bounding box can be represented by coordinates of four vertices of the bounding box. Alternatively, the position of one bounding box can be represented by a coordinate of an intersection point of two diagonal lines of the bounding box, a width of the bounding box, and a height of the bounding box.

[0453] In the set of paired bounding boxes, target categories of the plurality of bounding boxes are different at least. For example, when the plurality of bounding boxes include a first bounding box and a second bounding box, the target category of the first bounding box is different from the target category of the second bounding box.

[0454] When the target category of the first bounding box is a table label, the target category of the second bounding box can be any one of the aforementioned unfilled input box, filled input box, unfilled drop-down box, and filled drop-down box. Alternatively, the target category of the second bounding box can be the aforementioned unfilled fill-in box or filled fill-in box. Alternatively, the target category of the second bounding box can be the aforementioned input box or drop-down box. Alternatively, the target category of the second bounding box can be the aforementioned fill-in box.

[0455] For another example, when the plurality of bounding boxes include a first bounding box, a second bounding box, and a third bounding box, at least two of the target categories corresponding to the first bounding box, the second bounding box, and the third bounding box are different.

[0456] S2503, pairing a plurality of targets of the table according to the plurality of detection results to obtain one or more pairing results, the pairing result including a set of paired target positions and confidence of target categories, and the set of paired targets including a plurality of targets having a pairing relationship.

[0457] The electronic device can obtain all the group paired bounding boxes from the multiple detection results. Among the all the group paired bounding boxes, there can be a case that one target in the table corresponds to multiple group paired bounding boxes. Therefore, the electronic device can determine, by the NMS algorithm module 2413, from the multiple detection results, a group paired bounding box corresponding to each target in the table from among all the group paired bounding boxes.

[0458] Thus, the electronic device can determine a group paired bounding box corresponding to each target in the table. In this way, the electronic device can determine which targets in the table have a pairing relationship. Among the two targets having the pairing relationship, the two targets are referred to as paired targets.

[0459] The electronic device can further determine the relevant feature information of each target in the table. The relevant feature information of the target can include the position of the target and the target category of the target.

[0460] For example, in the group paired bounding box corresponding to the target 1, the first bounding box corresponds to the target 1, and the second bounding box corresponds to the target 2 having a pairing relationship with the target 1.

[0461] Thus, the target 1 and the target 2 can be referred to as paired targets. The position of the first bounding box can be regarded as the position of the target 1, and the target category of the first bounding box can be regarded as the category of the target 1. The position of the second bounding box can be regarded as the position of the target 2, and the target category of the second bounding box can be regarded as the category of the target 2.

[0462] In summary, the electronic device can obtain the relevant feature information of one or more group paired targets in the table.

[0463] Taking the target category corresponding to one of the bounding boxes in the group paired bounding box as the table label and the target category corresponding to the other bounding box as the input box / drop-down box as an example, one corresponding pairing result can be represented as:

[0464] Label box bbox, category confidence of the label box, input box / drop-down box bbox, category confidence of the input box / drop-down box.

[0465] Among them, the label box bbox refers to the position of the table label. The category confidence of the label box refers to the confidence of the table label. The input box / drop-down box bbox refers to the position of the input box / drop-down box. The category confidence of the input box / drop-down box refers to the confidence of the input box / drop-down box. Bbox is usually an abbreviation of bounding box, i.e., bounding box.

[0466] In some embodiments, the electronic device can store the related feature information of each group of paired targets as a queue element in a to-be-filled queue. During the process of filling in the form, the electronic device can automatically fill in the form according to the queue elements in the to-be-filled queue and the user information in the user information library.

[0467] The table visual detection method of the embodiments of the present application can obtain a first interface by an electronic device, and can obtain a plurality of targets of a table in the first interface. The electronic device can detect the plurality of targets of the table to locate, classify and pair each target, and then obtain a plurality of detection results, each detection result including the position of each group of paired bounding boxes and the confidence of each target category, so that each target corresponds to one or more groups of paired bounding boxes. The electronic device can pair the plurality of targets of the table according to the plurality of detection results to obtain one or more pairing results, which can filter out paired bounding boxes with high repetition and low confidence, so that each target corresponds to one group of paired bounding boxes. Then, the electronic device can determine one or more groups of paired targets in the plurality of targets, and the electronic device can also obtain the position and category of each group of paired targets according to the position of the group of paired bounding boxes and the confidence of each target category. That is, the electronic device can determine which target or targets each target is paired with from the plurality of targets, as well as the position and target category of the target.

[0468] Thus, the electronic device can obtain the position and target category of one or more groups of paired targets in the table, which facilitates the recommendation of user information to be filled in by another target in the paired targets based on one of the paired targets during the automatic filling process, which can complete the process of writing while filling, or directly fill in the user information to be filled in by another target in the paired targets in the interface, which can complete the filling process at one time.

[0469] In this way, the pairing of multiple targets for the table in the interface is achieved, so that the electronic device can automatically fill in various forms according to the pairing relationship of the targets, and is not limited to a specific form style, which can comprehensively cover all styles of forms, avoiding the problems of filling errors, frequent filling and high filling requirements, and improving the filling experience and efficiency of the user.

[0470] Based on the above description of the embodiments, taking the representation of the position of the bounding box by using the coordinates of one vertex of the bounding box, the width of the bounding box and the height of the bounding box, and taking the example of a group of paired bounding boxes including a first bounding box and a second bounding box, in some embodiments, the plurality of detection results obtained by S2501 are represented by the tensor O in Formula I;

[0471]

[0472] wherein B is the number of the first interfaces; N is the total number of the plurality of detection results; (x1, y1, w1, h1) is the coordinate and size of one of the bounding boxes in the pair of bounding boxes; (x2, y2, w2, h2) is the coordinate and size of the other of the bounding boxes in the pair of bounding boxes; is the confidence of the K categories of the one of the bounding boxes; is the confidence of the K target categories of the other of the bounding boxes; and K is the number of the target categories.

[0473] In summary, the electronic device can process one interface alone to determine the pairs of targets in the table included in the interface. Alternatively, the electronic device can also process a plurality of interfaces in batches to determine the pairs of targets in the table included in each of the interfaces.

[0474] Based on the description of the above embodiments, in S2502, the electronic device can locate and classify the plurality of targets in the table in combination with the table visual detection module 2412 mentioned in the foregoing.

[0475] In some embodiments, the electronic device can input the first interface into the table visual detection module 2412. The table visual detection module 2412 can locate, classify, and pair the targets in the first interface and output a plurality of detection results.

[0476] In combination with the multi-task loss function L, the table visual detection module 2412 can be trained using the training samples, so that the table visual detection module 2412 can take into account the losses caused by the location, classification, and pairing, and can more accurately locate, classify, and pair the targets in the table.

[0477] wherein the multi-task loss function L can be represented by Formula Two:

[0478] L = aL cls + bL bbox Formula Two;

[0479] wherein L cls is the classification loss function, which is used to represent the accuracy of the target categories of the bounding boxes; L bbox is the paired bounding box regression loss function, which is used to represent the accuracy of the paired bounding boxes; a and b are weight coefficients, which are used to balance the influence of the classification loss function and the paired bounding box regression loss function.

[0480] In summary, the multi-task loss function L can include the weighted sum of the classification loss function L cls and the paired bounding box regression loss function L bbox . Thus, the performance of classifying, locating, and pairing the targets in the table is optimized.

[0481] In some embodiments, the classification loss function can use a classification loss function L cls to measure the accuracy of classifying and locating each target in the table.

[0482] wherein the cross-entropy loss function can be represented by Formula Three;

[0483]

[0484] wherein N is the total number of samples; K is the number of target classes; y i,1,k and y i,2,k are the real labels of the paired bounding boxes of the i-th sample; p i,1,k and p i,2,k are the probabilities that the i-th sample belongs to the target class k; and γ is a penalty coefficient for weighting the loss of the paired bounding boxes.

[0485] In some embodiments, the bounding box regression loss function L bbox can use a smooth L1 loss function to measure the accuracy of pairing each target in the table.

[0486] wherein the paired bounding box regression loss function L bbox can be represented by Formula Four;

[0487]

[0488] wherein N is the total number of samples; is the real paired bounding box of the i-th sample; is the predicted paired bounding box; and the smooth L1 loss function is defined as follows:

[0489]

[0490] Based on the above description of the embodiments, in S2503, the NMS algorithm module 2413 in the electronic device can use a variety of ways to effectively process the paired bounding boxes output by the table visual detection module 2412, filter out duplicate and low-quality paired bounding boxes, and obtain one or more pairing results.

[0491] Next, combined with Figure 26B and Figure 26A , the detailed process of filtering out duplicate and low-quality paired bounding boxes according to the confidence and overlap of the bounding box by the electronic device will be specifically introduced. The confidence of the bounding box refers to the probability that the target corresponding to the bounding box is a target of a certain target class. The overlap of the bounding box refers to the degree of overlap or area in position with other bounding boxes.

[0492] Please refer toFigure 26B and Figure 26A , Figure 26B A flowchart of a table visual detection method provided by an embodiment of the present application is shown in FIG. 1. Figure 27A A schematic diagram of an intersection over union provided by an embodiment of the present application is shown in FIG. 2. As shown in FIG. 2, the table visual detection method of the present application can include the following steps. Figure 27A

[0493] S2601, determining the weighted confidence and the weighted intersection over union of all pairs of bounding boxes in the plurality of detection results.

[0494] The electronic device can obtain the relevant feature information of all pairs of bounding boxes in the plurality of detection results. For any pair of bounding boxes, the electronic device can determine the confidence of each target type of each bounding box in the pair of bounding boxes. Thus, the electronic device can perform weighted processing on the confidence of each target type of all bounding boxes in each pair of bounding boxes to obtain the weighted confidence of each pair of bounding boxes.

[0495] In some embodiments, taking a pair of bounding boxes including two bounding boxes as an example, the weighted confidence ω of each pair of bounding boxes can be represented by Formula Five.

[0496]

[0497] wherein λ is a weighted coefficient; λ is used to control the contribution of one bounding box in the pair of bounding boxes in the confidence; and 1-λ is used to control the contribution of the other bounding box in the pair of bounding boxes.

[0498] For any pair of bounding boxes, the electronic device can determine the position of each bounding box in the pair of bounding boxes. Thus, the electronic device can perform weighted processing on the position of all bounding boxes in each pair of bounding boxes to obtain the weighted intersection over union of each pair of bounding boxes.

[0499] In some embodiments, taking a pair of bounding boxes including two bounding boxes as an example, the weighted intersection over union IoUφ of each pair of bounding boxes can be represented by Formula Six.

[0500]

[0501] wherein, is the weighted intersection over union IoU value of one bounding box B1 in the pair of bounding boxes and the first reference bounding box B1 ref; is the weighted intersection over union IoU value of the other bounding box B2 in the pair of bounding boxes and the second reference bounding box B2 ref. ​​The weighted intersection-union ratio (IoU) is denoted by μ; μ is the weighting coefficient; μ is used to control the contribution of one bounding box B1 to the weighted intersection-union ratio (IoU); 1-μ controls the contribution of another bounding box B2 to the weighted intersection-union ratio (IoU).

[0502] The concept of intersection over union (IoU) is used in calculating the weighted overlap. IoU originates from sets in mathematics and is used to describe the relationship between two sets A and B. It is equal to the number of elements in the intersection of the two sets divided by the number of elements in their union. The specific formula is as follows:

[0503]

[0504] Electronic devices use the concept of Intersection over Union (IoU) to describe the degree of overlap or coincidence between two bounding boxes. Two bounding boxes can be viewed as sets of two pixels, and the IoU between these two bounding boxes is equal to the area of ​​the overlapping portion of the two bounding boxes divided by the combined area of ​​the two bounding boxes.

[0505] For example, such as Figure 27A As shown, the intersection region 1 in "Intersection" is the overlapping area of ​​the two bounding boxes, and the slash region 2 in "Union" is the combined area of ​​the two bounding boxes. Dividing the area of ​​intersection region 1 by the area of ​​slash region 2 gives the intersection-union ratio between the two bounding boxes.

[0506] S2602. Based on the weighted confidence and weighted overlap of all paired bounding boxes, determine the first detection result among multiple detection results. The first detection result includes one or more detection results, and the targets corresponding to each pair of paired bounding boxes in different detection results are different.

[0507] The electronic device filters out duplicate and low-quality paired bounding boxes from multiple detection results based on the weighted confidence and weighted overlap of all paired bounding boxes, selecting those with high confidence and low overlap to ensure that each target corresponds to a set of paired bounding boxes. Thus, the electronic device can determine the first detection result from multiple detection results.

[0508] S2603. Based on each detection result in the first detection result, determine the position of each pair of paired targets in the multiple targets as the position of a pair of paired bounding boxes in the corresponding detection result, and determine the confidence of the target category of each pair of paired targets as the target category with the highest confidence in a pair of paired bounding boxes in the corresponding detection result.

[0509] Based on the first detection result, the electronic device can determine each pair of targets among multiple targets, as well as the relevant feature information of each pair of targets.

[0510] For any given detection result, the position of each paired target is the position of a set of paired bounding boxes in that detection result; that is, the position of each target in each paired target is the position of the corresponding bounding box in that set of paired bounding boxes. The target category of each paired target is the target category with the highest confidence in that set of paired bounding boxes in that detection result; that is, the position of each target in each paired target is the target category with the highest confidence in the corresponding bounding box in that set of paired bounding boxes.

[0511] In summary, the electronic device can effectively process the paired bounding boxes output by the table visual inspection module 2412, filter out paired bounding boxes with high confidence and low overlap, and remove duplicate and low-quality paired bounding boxes, so that each target in the table can correspond to a set of paired bounding boxes.

[0512] Based on this, in S2602, the electronic device can first consider the confidence level and then the overlap. Alternatively, the electronic device can first consider the overlap and then the confidence level. Or, the electronic device can consider both the confidence level and the overlap level simultaneously. Thus, the electronic device can obtain the first detection result.

[0513] For example, regarding the screening principle that considers confidence level first and then overlap, please refer to [link / reference]. Figure 27B , Figure 27B This is a flowchart illustrating a table visual inspection method provided in an embodiment of this application. Figure 27B As shown, the table visual detection method in this application embodiment may include:

[0514] S2701. The electronic device calculates the weighted confidence ω for each pair of bounding boxes.

[0515] S2702. The electronic device calculates the weighted overlap IoUφ for each pair of bounding boxes.

[0516] S2703. The electronic device sorts all paired bounding boxes according to the weighted confidence ω of each pair of paired bounding boxes.

[0517] The sorting of all paired bounding boxes can be in descending order of weighted confidence or in ascending order of weighted confidence; this embodiment does not limit the sorting.

[0518] S2704. The electronic device performs NMS operation, which can filter out paired bounding boxes with high confidence and low overlap, and obtain the first detection result.

[0519] Below, in conjunction with Figure 27B This section details the specific implementation process of S2703 and S2704 by the electronic device.

[0520] Please see Figure 28 , Figure 28A flowchart of a table visual detection method provided by an embodiment of the present application is shown in FIG. 8. As shown in FIG. 8, the table visual detection method of the embodiment of the present application can include the following steps. Figure 29

[0521] S2801, according to the order from large to small of the weighted confidence, the electronic device sorts all the paired bounding boxes.

[0522] The electronic device can sort all the paired bounding boxes according to the order from large to small of the weighted confidence. The electronic device can sort all the paired bounding boxes according to the order from large to small of the weighted confidence. The weighted confidence of the sorted paired bounding boxes satisfies the following condition:

[0523]

[0524] S2802, the electronic device traverses all the paired bounding boxes except the first group of paired bounding boxes, and when the difference between the weighted overlap of a group of paired bounding boxes in the other group of paired bounding boxes and the first group of paired bounding boxes is greater than or equal to the first threshold, the electronic device deletes the group of paired bounding boxes; when the difference between the weighted overlap of a group of paired bounding boxes in the other group of paired bounding boxes and the first group of paired bounding boxes is less than the first threshold, the electronic device deletes the first group of paired bounding boxes and updates the group of paired bounding boxes as the first group of paired bounding boxes, to obtain the first detection result.

[0525] The electronic device compares the difference between the weighted overlap of the second group of paired bounding boxes and the first group of paired bounding boxes.

[0526] When the difference between the weighted overlap of the second group of paired bounding boxes and the first group of paired bounding boxes is greater than or equal to the first threshold, the electronic device deletes the second group of paired bounding boxes. When the difference between the weighted overlap of the second group of paired bounding boxes and the first group of paired bounding boxes is less than the first threshold, the electronic device deletes the first group of paired bounding boxes and updates the second group of paired bounding boxes as the first group of paired bounding boxes.

[0527] Then, the electronic device compares the difference between the weighted overlap of the third group of paired bounding boxes and the first group of paired bounding boxes. The comparison process of the weighted overlap of the third group of paired bounding boxes and the first group of paired bounding boxes can refer to the description of the weighted overlap of the second group of paired bounding boxes and the first group of paired bounding boxes, which will not be described here.

[0528] By analogy, the electronic device can traverse and compare the last group of paired bounding boxes in the sorted all paired bounding boxes. Thus, the electronic device can determine that the first detection result is the final screened paired bounding box.

[0529] In one specific embodiment, the NMS operation can include the following process:​

[0530] The electronic device can initialize a result list of one control Result list For storing the final screening of the paired bounding boxes.

[0531] Then, the electronic device iterates through all the paired bounding boxes after sorting:

[0532] If the result list is empty, the electronic device adds the set of paired bounding boxes to the result list .

[0533] If the result list is not empty, the electronic device calculates the weighted IoU of the bounding box and the bounding box already screened in the result list and determines whether the weighted IoU is less than the first threshold value, which can be expressed as follows:

[0534] If

[0535] , the electronic device adds the set of paired bounding boxes to the result list .

[0536] where IoU thresh is the first threshold value.

[0537] In summary, the electronic device can screen out paired bounding boxes with high confidence and low overlap, remove repeated and low-quality paired bounding boxes, ensure that the paired bounding boxes in the first detection result do not overlap excessively in the same target category, and have the highest weighted confidence, so that each target in the table can correspond to a set of paired bounding boxes.

[0538] ​Based on the above-mentioned exemplary information processing method, during the process in which the user uses any application program of the electronic device, the electronic device can display a to-be-entered interface of the application program. The to-be-entered interface is the first interface mentioned above. The to-be-entered interface is used to display information to be filled in by the user, that is, the to-be-entered interface is an interface in which user information needs to be entered in the application program. The to-be-entered interface can include a plurality of filling controls and name controls corresponding to the filling controls, that is, the name controls correspond one-to-one to the filling controls. The filling control is used to display user information filled or selected by the user. The name control is the table label mentioned above. The name control is used to display the information name to which the filling control belongs. Then, in response to a filling operation of the user on any filling control in the to-be-entered interface, the electronic device displays the filling content of the user. That is, the user needs to fill in the user information in the to-be-entered interface one by one to complete the information entry.

[0539] Exemplarily, the above-mentioned application program can be an application program (such as a shopping APP) installed on the electronic device, can be a web version application program, or can be a small program in the application program, and the specific implementation is not limited. The web version application program can be an application searched in a browser. The small program in the application program can be an application that does not need to be downloaded and installed for use. The information name to which the above-mentioned filling control belongs can include at least one of a user name, a name, a login password, a gender, a certificate type, a contact method, and a delivery address.

[0540] However, if the user needs to fill in the same information on different interfaces (such as account registration interfaces of different application programs), that is, needs to input the repeated user information multiple times, the information filling process can be relatively cumbersome, a long filling time is wasted, and the user experience is poor.

[0541] Therefore, in order to simplify the information filling process and improve the filling efficiency of the information, an embodiment of the present application provides an information entry method. In the method, the electronic device displays a to-be-entered interface, wherein the to-be-entered interface includes a plurality of filling controls and name controls corresponding to the filling controls. Then, in response to a triggering operation of the user on any filling control in the to-be-entered interface, the electronic device obtains a label name of the name control corresponding to the triggered filling control. Then, the electronic device selects candidate user information corresponding to the label name of the name control from a user information library. The user information library includes a plurality of user information, and the user information is obtained by the electronic device from a plurality of application programs. Then, the electronic device displays the candidate user information in the to-be-entered interface.

[0542] In this embodiment, after detecting a user's trigger operation on any fill control in the input interface, the electronic device can select the corresponding candidate user information from the user information database based on the label name of the name control corresponding to the triggered fill control, and display it on the input interface. This enables intelligent information filling; users only need to select the corresponding user information according to their actual needs, eliminating the need for manual entry, simplifying the information filling process, improving efficiency, and enhancing the user experience. Furthermore, since the candidate user information is collected by the electronic device from multiple applications, meaning it can select from all previously collected user information and is not limited to information collected by the application to which the input interface belongs, comprehensive display of user information is achieved. This reduces the likelihood of users repeatedly entering the same user information in different applications, providing convenience for users to quickly enter information.

[0543] For example, the aforementioned electronic devices may be mobile phones, tablets, desktops, laptops, handheld computers, notebook computers, ultra-mobile personal computers (UMPCs), netbooks, as well as cellular phones, personal digital assistants (PDAs), augmented reality (AR) / virtual reality (VR) devices, and other devices capable of installing and using applications that require user information. This application does not impose any special limitations on the specific form of the electronic device.

[0544] Figure 29 This is a schematic diagram of the hardware structure of the electronic device 2800 provided in the embodiments of this application, as shown below. Figure 29 As shown, the electronic device 2800 may include a processor 2810, an external memory interface 2820, an internal memory 2821, a universal serial bus (USB) interface 2830, a charging management module 2840, a power management module 2841, a battery 2842, antenna 1, antenna 2, a mobile communication module 2850, a wireless communication module 2860, an audio module 2870, a speaker 2870A, a receiver 2870B, a microphone 2870C, a headphone jack 2870D, a sensor 2880, a button 2890, a motor 2891, an indicator 2892, a camera 2893, a display screen 2894, and a subscriber identification module (SIM) card interface 2895, etc.

[0545] It can be understood that the structural schematic of the embodiments of the present application does not constitute a specific limitation on the electronic device 2800. In other embodiments of the present application, the electronic device 2800 can include more or fewer components than shown, or combine certain components, or split certain components, or different arrangement of components. The components shown can be implemented in hardware, software, or a combination of software and hardware.

[0546] The processor 2810 can include one or more processing units, for example: the processor 2810 can include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Among them, different processing units can be independent devices, or can be integrated in one or more processors.

[0547] In some embodiments, the electronic device 2800 can complete the information input method provided by the present application through the processor 2810.

[0548] The wireless communication function of the electronic device 2800 can be realized by the antenna 1, the antenna 2, the mobile communication module 2850, the wireless communication module 2860, the modem processor, and the baseband processor, etc.

[0549] The electronic device 2800 realizes the display function through the GPU, the display screen 2894, and the application processor, etc. The GPU is a microprocessor for image processing, connected to the display screen 2894 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 2810 can include one or more GPUs that execute program instructions to generate or change display information.

[0550] The display screen 2894 is used to display images, videos, etc. In some embodiments, the electronic device 2800 can display the to-be-input interface and the candidate user information corresponding to any fill-in control in the to-be-input interface through the display screen 2894.

[0551] The electronic device 2800 can realize the shooting function through the ISP, the camera 2893, the video codec, the GPU, the display screen 2894, and the application processor, etc.

[0552] The electronic device 2800 can implement audio functions through an audio module 2870, a speaker 2870A, a receiver 2870B, a microphone 2870C, a headset interface 2870D, and an application processor, etc. For example, music playing, recording, etc. In some embodiments, the electronic device 2800 can collect a voice instruction input by a user through the microphone 2870C. Then, the electronic device 2800 can determine an information input mode of a to-be-input interface according to the voice instruction. The information input mode includes a one-key input mode and a separate input mode. The one-key input mode is used to represent that the electronic device 2800 displays all candidate user information corresponding to the filling control at one time. The separate input mode is used to represent that the electronic device 2800 displays the candidate user information corresponding to the filling control according to the filling control triggered by the user.

[0553] The keys 2890 can include a power key, a volume key, etc. The keys 2890 can be mechanical keys. They can also be touch keys. The electronic device 2800 can receive key input, generate key signal input related to user settings and function control of the electronic device. The motor 2891 can generate a vibration prompt. The motor 2891 can be used for incoming call vibration prompt, and can also be used for touch vibration feedback. The indicator 2892 can be an indicator light, which can be used to indicate the charging state, the power change, and can also be used to indicate messages, missed calls, notifications, etc. The SIM card interface 2895 is used to connect the SIM card. The SIM card can be inserted into or pulled out of the SIM card interface 2895 to realize contact and separation with the electronic device.

[0554] For example, the software system of the electronic device 2800 described above can adopt a layered architecture, an event-driven architecture, a microkernel architecture, a microservice architecture, or a cloud architecture. The embodiment of the application takes the Android system with a layered architecture as an example to exemplarily illustrate the software structure of the electronic device 2800.

[0555] Figure 29 The software structure block diagram of the electronic device 2800 of the embodiment of the application is shown in FIG. 28.

[0556] The layered architecture divides the software into several layers, each layer has a clear role and division of labor. The layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into four layers, from top to bottom, the application layer (referred to as the application layer), the application framework layer (referred to as the framework layer), the Android runtime and the system library, and the kernel layer (or referred to as the driver layer). The application layer can include a series of application packages.

[0557] As Figure 30As shown, the application package can include short messages, galleries, notes, form filling assistants, information management modules, function setting modules, etc.

[0558] The form filling assistant is configured to determine a name control corresponding to a triggered filling control in a to-be-entered interface. In some embodiments, in response to a triggering operation of a user on any filling control in the to-be-entered interface, the form filling assistant determines the name control corresponding to the triggered filling control. The to-be-entered interface includes a plurality of filling controls and the name controls corresponding to the filling controls. That is, the filling controls and the name controls correspond to each other one by one.

[0559] The information management module is configured to receive a target operation of a user on user information in a user information library. The target operation includes an information query operation and / or an information adjustment operation. The information query operation is configured to cause the electronic device to display user information that the user wants to query. The information adjustment operation is configured to cause the electronic device to adjust the user information in the user information library. For example, the information adjustment operation can be an information deletion operation on any user in the user information library, an information modification operation on any user information in the user information library, an information addition operation on the user information library, etc.

[0560] The function setting module is configured to set a collection source (i.e., a preset collection application) of user information in the user information library. In some embodiments, if a switch control with an automatic collection function in a function setting interface (or an information collection setting interface) is turned on by the user, it indicates that the electronic device is allowed to collect user information saved by all applications, and the function setting module can collect user information saved by any application in the electronic device. In other embodiments, if the switch control with the automatic collection function in the function setting interface is not turned on by the user, and any application belongs to a switch control turned on by the user, the function setting module can collect user information saved by the application corresponding to the turned-on control.

[0561] The application framework layer provides an application programming interface (API) and a programming framework for the applications in the application layer. The application framework layer includes some pre-defined functions.

[0562] As shown, the application framework layer can include a window manager (WMS), a content provider, a view system, an image memory module, etc. Figure 31

[0563] The window manager is configured to manage window programs. The window manager can obtain the size of a display screen, determine whether there is a status bar, lock a screen, and capture a screen, etc. ​

[0564] The content provider is used to store and retrieve data and make the data accessible to applications. The data can include videos, images, audio, dialed and received calls, browsing history and bookmarks, phonebook, etc.

[0565] The view system includes visual controls, such as a control for displaying text, a control for displaying an image, etc. The view system can be used to build an application. A display interface can be composed of one or more views. For example, a display interface including a short message notification icon can include a view for displaying text and a view for displaying an image.

[0566] The image memory module is configured to determine at least one candidate user information matched with a target label name. The target label name is obtained by standardizing a label name of a name control corresponding to the triggered fill-in control. In some embodiments, upon receiving the information query instruction sent by the form filling assistant, the image memory module can standardize the label name of the name control corresponding to the triggered fill-in control to obtain the target label name. Then, the image memory module can select at least one user information matched with the target label name from the user information library, and display the at least one user information as at least one candidate user information in the to-be-entered interface for the user to select.

[0567] Optionally, after obtaining the at least one candidate user information, the image memory module can sort the at least one candidate user information according to a preset display rule to obtain an information candidate list. The preset display rule can include at least one of an order of information update time, a descending order of information entry times, and an association with historical user information. The information update time refers to the time when the candidate user information is added to the user information library. The information entry times refer to the number of times the candidate user information is matched, i.e., the number of times the candidate user information is selected by the user. The historical user information refers to the user information that has been entered in the to-be-entered interface. Then, the image memory module can send the information candidate list to the form filling assistant, so that the electronic device displays the candidate user information in the information candidate list in the order.

[0568] The Android runtime includes a core library and a virtual machine. The Android runtime is responsible for scheduling and managing the Android system. The core library includes two parts: one part is the function function that the java language needs to call, and the other part is the core library of Android.

[0569] The application program layer and the application framework layer run in a virtual machine. The virtual machine executes the java files of the application program layer and the application framework layer as binary files. The virtual machine is used to perform functions such as management of object life cycle, stack management, thread management, management of security and exceptions, and garbage collection.

[0570] The system library can include a plurality of functional modules. For example, a surface manager, a three-dimensional graphics processing library (e.g., openGL ES), a two-dimensional graphics engine (e.g., SGL), media libraries, and the like.

[0571] The surface manager is used to manage the display subsystem and provides fusion of 2D and 3D layers for a plurality of applications. The three-dimensional graphics processing library is used to implement three-dimensional graphics drawing, image rendering, composition, and layer processing. The two-dimensional graphics engine is a drawing engine for 2D drawing. The media libraries support playback and recording of a plurality of commonly used audio, video formats, and still image files. The media libraries can support a plurality of audio and video encoding formats, such as MPEG4, H.264, MP3, AAC, AMR, JPG, PNG, and the like.

[0572] The kernel layer is a layer between hardware and software. The kernel layer includes at least display drivers, camera drivers, audio drivers, sensor drivers, and the like.

[0573] For example, the software structure of the electronic device 2800 described above can further include a hardware abstraction layer. The hardware abstraction layer is a package of Linux kernel drivers that provides an interface upward, which hides the hardware interface details of a specific platform and provides a virtual hardware platform for the operating system. In embodiments of the present application, the hardware abstraction layer can include camera HAL, audio HAL, GPS HAL, Wi-Fi HAL, and the like.

[0574] It can be understood that, Figure 31 The layers in the structure shown and the components included in each layer do not constitute a specific limitation on the electronic device 2800, i.e., a mobile phone. In other embodiments of the present application, the structure can include more or fewer layers than shown, and each layer can include more or fewer components, which are not limited in the present application.

[0575] In the following, based on the above-described example of the information processing method, in the automatic form filling process or the form automatic filling process, the content to be filled can be obtained from the following user information library to be displayed in the to-be-entered interface of the electronic device. That is, the user information in the user information library can be used as the source of the content to be filled in the automatic filling process.

[0576] Next, the specific implementation of the foregoing process is introduced in detail. Illustratively, the embodiment of the present application provides an information entry method. The method can enter the user information in the user information library into a to-be-entered interface. Next, the method of the embodiment of the present application is described by taking the electronic device as a mobile phone. Specifically, as shown in Figure 31 , the information entry method can include S3001-S3014.

[0577] S3001, the form-filling assistant in the mobile phone receives a trigger operation of a user on any fill-in control in the to-be-entered interface.

[0578] The to-be-entered interface refers to an interface in an application program that needs to enter user information. The to-be-entered interface can include a plurality of fill-in controls and a name control corresponding to the fill-in control, that is, the name control and the fill-in control one-to-one correspond. The fill-in control is used to display the user information filled or selected by the user. The name control is used to display the information name to which the fill-in control belongs. Illustratively, as shown in Figure 32 , the to-be-entered interface displayed by the mobile phone is an add shipping address interface (that is, the (a) interface in Figure 32 ), which can include a "recipient" control 3111, a fill-in control 3121 corresponding to the "recipient" control, a "mobile phone number" control 3112, and a fill-in control 3122 corresponding to the "mobile phone number" control 3112. The "recipient" control 3111 and the "mobile phone number" control 3112 are both name controls.

[0579] Illustratively, the trigger operation refers to an operation performed by the user by touching the mobile phone display screen with a finger. The trigger operation can be a single-click operation on the fill-in control, or a double-click operation on the fill-in control, and the like, which is not limited.

[0580] It should be noted that the application program to which the to-be-entered interface belongs can be an application program installed on the mobile phone, or a web-based application program, or a small program in any application program in the mobile phone, and the like, which is not limited.

[0581] S3002, in response to the trigger operation of the user on any fill-in control in the to-be-entered interface, the form-filling assistant sends an information query instruction to the portrait memory module in the mobile phone.

[0582] Specifically, after receiving the trigger operation of the user on any fill-in control in the to-be-entered interface, the form-filling assistant can send an information query instruction to the portrait memory module according to the triggered fill-in control. The information query instruction carries attribute information of the name control corresponding to the triggered fill-in control.

[0583] In some embodiments, the form-filling assistant can determine the triggered fill-in control according to the trigger position of the user and the display position of the fill-in control in the interface to be input. The trigger position of the user is determined based on the touch point generated when the user's finger touches the display screen of the mobile phone. Then, the form-filling assistant can determine the name control corresponding to the triggered fill-in control according to the position mapping relationship between the fill-in control and the name control in the interface to be input, and obtain the attribute information of the name control.

[0584] In the present embodiment, the attribute information of the name control can include the label name of the name control, the application of the interface to which the name control belongs (or the interface belonging application), and the title name of the interface to which the name control belongs (or the interface title name). The label name of the name control refers to the information name displayed by the name control in the interface to be input. For example, if the name control is a "recipient" control, the label name is "recipient". The interface belonging application is the application program to which the interface to be input belongs. The interface title name is used to represent the purpose of filling in the user information in the interface to be input. For example, the interface title name can be the interface information displayed at the top position of the interface to be input. For example, in the interface (a) shown in the above embodiment, the interface title name can be "add delivery address". Figure 33

[0585] S3003, the image memory module determines the label name in the attribute information according to the attribute information of the name control corresponding to the triggered fill-in control carried by the information query instruction sent by the form-filling assistant, and obtains a detection result.

[0586] Specifically, after receiving the information query instruction sent by the form-filling assistant, the image memory module can detect the label name of the name control according to the attribute information of the name control corresponding to the triggered fill-in control carried by the information query instruction, to determine whether the label name can be standardized. That is, the detection result is used to indicate whether the label name of the name control can be standardized. It can be understood that the standardization refers to converting the label name into the label name stored in the user information library, so that the mobile phone can determine the candidate user information according to the label name, thereby improving the accuracy of information recommendation.

[0587] ​In some embodiments, the mobile phone can detect the label name by determining whether the attribute information of the name control meets a preset condition. That is, the preset condition is used to evaluate whether the label name can be standardized. Specifically, if the attribute information of the name control meets the preset condition, it means that the label name can be standardized, and the image memory module can perform S3004 to standardize the label name. If the attribute information of the name control does not meet the preset condition, it means that the label name cannot be standardized, and therefore the image memory module can perform S3011 to send the label name to the cloud service, thereby providing a basis for subsequent standardization of the label name.

[0588] The preset condition can include that the attribute information of the name control meets any mapping relationship in the preset mapping rule, and / or that the similarity between the label name and the label name of the name control in the user information library is greater than a preset similarity. It should be noted that the specific judgment process of the preset condition will be described in detail below.

[0589] S3004, in the case where the detection result indicates that the label name can be standardized, the image memory module standardizes the label name to obtain a target label name.

[0590] Specifically, after the detection result indicates that the label name of the name control can be standardized, the image memory module standardizes the label name of the name control to obtain a target label name of the name control. The target label name is any label name existing in the user information library. For example, if the label name is the recipient, the image memory module can standardize the recipient to the name, that is, the target label name is the name. For another example, if the label name is the mobile phone number, the image memory module can standardize the mobile phone number to the mobile phone number, that is, the target label name is the mobile phone number.

[0591] The standardization process of the label name will be described in detail below.

[0592] S3005, the image memory module determines whether there is user information matching the target label name in the user information library.

[0593] In some embodiments, after obtaining the target label name, the image memory module can determine whether there is user information matching the target label name in the user information library. If there is user information matching the target label name in the user information library, it means that the user has filled in the user information of the target label name before, and therefore the image memory module can execute S3007. If there is no user information matching the target label name in the user information library, it means that the user has not filled in the user information of the target label name before, and therefore the image memory module can execute S3006.

[0594] In an example, the user information library can include a plurality of user information of the machine owner and a label name of each user information. For example, taking the user information library including the gender of the machine owner as "female" as an example, the label name is gender, and the user information is female. In another example, the user information library can also include a plurality of user information of at least one associated user and a label name of each user information. The associated user refers to a user who has a contact with the machine owner. For example, the associated user can be a relative of the machine owner (such as a child, a parent, etc.), can be a friend of the machine owner, and can be a customer of the machine owner, etc., which is not limited in particular.

[0595] In an implementation manner, the user information in the user information library can be the user information filled in by the user in advance in the information editing interface of the setting application. Specifically, in the case that the information filling event of the user is detected, the mobile phone can store the user information filled in by the user as new user information in the user information library. The information filling event can be the filling operation of the user information in the information editing interface. In this way, a basis for subsequent calling of user information is provided.

[0596] In some embodiments, in response to the information filling operation of the user on the information editing interface, the mobile phone can obtain the newly filled user information. Then, the mobile phone can store the newly filled user information in the user information library, so as to facilitate subsequent calling of the newly filled user information. For example, as shown in Figure 33 As shown in (a), the mobile phone is displaying the personal information interface, if the adding control 3201 in the personal information interface is clicked by the user, the mobile phone can display the information editing interface as shown in (b) of Figure 33 In response to the filling operation of the user on the information control 3202 of the user name in the information editing interface, the mobile phone can obtain and display the filling content "Zhao Yiyi" of the information control 3202.

[0597] In another implementation, the user information in the user information library can also be extracted by the mobile phone from a preset collection application. The preset collection application is an application with information collection function. That is, before extracting the user information saved by any application in the mobile phone, the mobile phone can determine whether the application has information collection function. Only when the application has information collection function, the mobile phone can perform information extraction on the application to obtain the user information saved by the application.

[0598] In some embodiments, the preset collection application can be preset by the mobile phone, i.e. the default setting of the mobile phone. For example, referring to Figure 33 , the mobile phone is displaying interface (a) in Figure 34 , i.e. the setting interface of information collection, which includes the switch control 3301A corresponding to the automatic collection function, the switch control 3302 corresponding to the SMS application, the switch control 3303 corresponding to the gallery application, and the switch control 3304 corresponding to the note application. It can be seen that the switch control 3301 corresponding to the automatic collection function, the switch control 3302 corresponding to the SMS application, and the switch control 3303 corresponding to the gallery application are all in the off state, and the switch control 3304 corresponding to the note application is in the on state. That is, the mobile phone only defaults the note application as the preset collection application.

[0599] In some other embodiments, the preset collection application can be preset by the user. That is, the mobile phone can determine the application with which the user enables the information collection function as the preset collection application. For example, as shown in Figure 34 , if the switch control 3301A corresponding to the automatic collection function in the setting interface of information collection is enabled by the user, it means that the user allows the mobile phone to extract user information from any application in the mobile phone, and therefore, the mobile phone can regard all the installed applications in the mobile phone as the preset collection application, i.e. the mobile phone can extract the user information saved by each application in the mobile phone. The switch control 3301A is used to represent that the automatic collection function is in the off state. Then, the mobile phone can display the target setting interface shown in (b) in Figure 34 , which includes the switch control 3301B corresponding to the automatic collection function. The switch control 3301B is used to represent that the automatic collection function is in the on state.

[0600] In the case that the switch control 3301A corresponding to the automatic collection function in the setting interface of information collection is not enabled by the user, if the switch control 3302 corresponding to the SMS application in the setting interface 3300 of information collection is clicked by the user, it means that the user allows the mobile phone to extract user information from the SMS application, and therefore, the mobile phone can regard the SMS application as the preset collection application, i.e. the mobile phone can extract the user information saved by the SMS application.

[0601] It is understandable that if the default collection app is an SMS app, the phone can extract user information from SMS messages saved by the SMS app. These SMS messages can be received by the phone or edited by the user. If the default collection app is a gallery app, the phone can extract user information from images saved by the gallery app. These images can be taken by the user or downloaded and cached from any app on the phone (such as a social media app). If the default collection app is a notes app, the phone can extract user information from notes saved by the notes app. These notes are edited by the user. If the default collection app is a third-party app (such as a social media app), the phone can extract user information from the information entry interface of that third-party app. This information entry interface can be any interface within the app that allows user information to be entered. This interface can be an account registration interface, a page for adding traveler information, a page for adding a shipping address, etc., and is not specifically limited.

[0602] In some embodiments, the process of the mobile phone extracting user information from different applications can be automated. For example, taking a gallery application as a preset collection application, when a new image is detected in the gallery application, the mobile phone can extract information from the new image to obtain new user information and add it to the user information database. This achieves automatic information extraction without requiring user intervention, simplifying the process and facilitating the subsequent display of candidate user information.

[0603] In some embodiments, the process of the mobile phone extracting user information from different applications can also be achieved by the mobile phone extracting information based on the user's information extraction operation. Specifically, in response to the user's information extraction operation on the display interface of a preset collection application, the mobile phone can extract information from the display interface to obtain new user information. The display interface may include at least one piece of user information. The information extraction operation can be a user's click operation on an information extraction control, or a voice command from the user regarding information extraction, etc., and is not specifically limited. For example, the voice command could be to extract user information. Afterwards, the mobile phone can add the new user information to the user information database. This allows for accurate extraction of user information, confirms with the user before extraction, reduces the risk of information security issues due to excessive extraction, ensures the security of the extraction environment, and provides convenient conditions for subsequently displaying candidate user information.

[0604] For example, such as Figure 34 As shown, taking the aforementioned preset collection application as an example, the phone is displaying...Figure 34 The image display interface shown in (a) includes an ID card image 3410. Subsequently, in response to the user's wake-up operation for the voice assistant, the phone can display as shown... Figure 34 The interface shown in (b) displays a "Save to Personal Information" control, a "Copy ID Number" control, a "Copy Address" control, and a voice assistant prompt card 3420 overlaid on the image display interface. The wake-up operation can be initiated by the user uttering a wake-up phrase for the voice assistant, such as "Hello Youyou." The "Save to Personal Information" control is used to extract user information from the ID card image 3410 and save it to the user information database. It can be understood that the types of controls overlaid on the phone are determined by the content displayed on the phone when the voice assistant is activated.

[0605] Afterwards, with the voice assistant activated, the user can input dialogue. This dialogue can be in voice format, such as inputting a command like "Save user information." Alternatively, it can be in text format, such as inputting a command like "Save user information" into the voice assistant's prompt card 3420. Upon detecting the user's input, the phone can perform the corresponding operation, namely saving the user information from the ID card image 3410 to the user information database. Alternatively, in response to the user's click on the "Save to Personal Information" control, the phone can save the user information from the ID card image 3410 to the user information database.

[0606] In one implementation, before extracting information from the aforementioned display interface, the mobile phone needs to authenticate the user. That is, the phone can only extract information from the display interface if the authentication is successful (i.e., the user is the owner of the device). Specifically, the phone can authenticate the user and obtain an authentication result. This result indicates whether the user is the owner of the device. If the authentication result indicates the user is the owner, the phone can extract information from the display interface to obtain new user information. If the authentication result indicates the user is not the owner, i.e., the user is not the owner, the phone does not perform the information extraction operation, i.e., it does not extract information from the display interface. This reduces the possibility of non-owner users extracting user information, improves the security of information extraction, and ultimately enhances the user experience.

[0607] The mobile phone can authenticate users through facial recognition, fingerprint recognition, or password input, among other methods; the specific method is not limited. For example, please see... Figure 34If the "Save to Personal Information" control in interface (b) is clicked by the user, the phone can display the following: Figure 34 The security verification interface shown in (c) is used to authenticate the user's identity. This security verification interface is a fingerprint recognition interface. After successful authentication, the mobile phone can save the user information from the ID card image 3410 to the user information database.

[0608] In another implementation, upon obtaining new user information, the mobile phone can retrieve multiple user information entries corresponding to the user to whom the new user information belongs from the user information database. Then, the mobile phone can determine whether user information with that tag name exists among the multiple user information entries based on the tag name corresponding to the new user information. If the tag name does not exist among the multiple user information entries, it means that the user information database does not store user information with that tag name. Therefore, the mobile phone can save the new user information to the user information database and display the personal information interface. This personal information interface includes the new user information. This allows for accurate saving of user information, reducing the occurrence of missed or incorrectly saved user information, and providing convenient conditions for timely retrieval of user information later.

[0609] For example, such as Figure 34 As shown, the phone is displaying... Figure 34 In interface (c), if the user's identity authentication is successful but the user information in ID card image 3410 is not stored in the user information database, the mobile phone can display the personal information interface of the user to whom ID card image 3410 belongs, that is, display... Figure 34 The (d) interface is shown in the image. This personal information interface includes the newly added ID card (i.e., ID card 3411). Optionally, if the user clicks on ID card 3411, it indicates that the user wants to view an image of the ID card. The phone can then display an image of ID card 3410 for the user to view and verify the information.

[0610] If user information with the tag name exists among the multiple user information entries, it means that user information with the tag name has already been stored in the user information database. Therefore, in order to improve the accuracy of information storage in the user information database, the mobile phone can use the user information with the tag name among the multiple user information entries as the original user information and determine whether the original user information is the same as the new user information. If the original user information is the same as the new user information, it means that the new user information has been saved to the user information database. Therefore, the mobile phone does not need to save the new user information again, so as to avoid wasting too much storage space due to duplicate saving of user information and improve the utilization rate of storage resources.

[0611] If the original user information differs from the new user information, it indicates an error in either. Therefore, the phone can display an information update interface for the user to determine whether an update is necessary. This update interface may include at least one of the following: original user information, new user information, a prompt about "updating user information," a back control, and an update control. If the user clicks the back control, it means they do not wish to update the user information, and the phone can continue to save the original information. If the user clicks the update control, it means they want to update the user information, and the phone can replace the original information with the new information in the user database. This ensures accurate user information updates, reduces errors, and improves the user experience.

[0612] For example, such as Figure 31 As shown, the phone is displaying... Figure 31 In interface (c), if the user's identity authentication is successful and the user's ID number is stored in the user information database, but the ID number stored in the user information database is different from the ID number in ID card image 3410, the mobile phone can display... Figure 31 The information update interface shown in (e) includes a prompt (i.e., "Update card information?"), the current number (i.e., the original user information), the new number (i.e., the new user information), a "Save Current Information" control (i.e., a return control), and an "Update" control. If the user clicks the "Save Current Information" control, the phone can continue to save the original ID number in the user information database. If the user clicks the "Update" control, the phone can replace the ID number in ID image 3410 with the original ID number and save it to the user information database.

[0613] S3006, if there is no user information in the user information database that matches the target tag name, the profile memory module sends a matching result instruction to the form filling assistant.

[0614] The matching result indicates that there is no user information in the user information database that matches the target tag name, meaning that the profile memory module did not find any user information that matches the tag name.

[0615] In some embodiments, when the form-filling assistant receives a matching result instruction sent by the profile memory module, the mobile phone can continue to display the input interface, that is, it does not display the recommended user information (i.e., candidate user information) on the input interface, so that the user can fill in the information for the aforementioned triggered input control.

[0616] S3007, in a case where the user information database contains user information matching the target tag name, the portrait memory module determines the user information matching the target tag name as candidate user information.

[0617] Specifically, in a case where the user information database contains user information matching the target tag name, the portrait memory module can display the user information matching the target tag name as candidate user information in the to-be-entered interface.

[0618] In some embodiments, the portrait memory module can determine the number of information of the candidate user information. In a case where the number of information of the candidate user information is 1, the portrait memory module can directly execute S3008. In a case where the number of information of the candidate user information is not 1, that is, the user information database contains multiple user information matching the target tag name, the portrait memory module can sort the multiple candidate user information according to a preset display rule to obtain an information candidate list. The information candidate list is used to represent the arrangement order of the multiple candidate user information.

[0619] The preset display rule can include at least one of the following: the order of information update time, the order of information entry times, and association with historical user information. The information update time refers to the time when the candidate user information is added to the user information database. The information entry times refer to the number of times the candidate user information is selected by the user. The historical user information refers to the user information that has been entered in the to-be-entered interface.

[0620] In an implementation manner, the portrait memory module can preferentially place the candidate user information close to the information filling time in the front of the information candidate list, and place the candidate user information far from the information filling time in the back of the information candidate list. Specifically, the portrait memory module can obtain the information update time of each candidate user information. Then, the portrait memory module can sort the multiple candidate user information according to the order of information update time to obtain the information candidate list. For example, the candidate user information is A user, B user and C user, and the information update time of A user is T1, the information update time of B user is T2, and the information update time of C user is T3. T1, T2 and T3 include but are not limited to year, month, date, clock and minute, etc. Therefore, the portrait memory module can place C user in the first position of the information candidate list, place B user in the second position of the information candidate list, and place A user in the third position of the information candidate list, that is, the information candidate list is C user, B user and A user.

[0621] In another implementation manner, the image memory module can preferentially place the candidate user information with a large number of information query times in front of the information candidate list, and place the candidate user information with a small number of information query times at the back of the information candidate list. Specifically, the image memory module can obtain the information query times of each candidate user information. Then, the image memory module can sort the plurality of candidate user information in descending order of the information query times to obtain the information candidate list. For example, the candidate user information are A user, B user and C user respectively, and the information query times of A user is 10, the information query times of B user is 5, and the information query times of C user is 1. Therefore, the image memory module can place A user at the first position of the information candidate list, place B user at the second position of the information candidate list, and place C user at the third position of the information candidate list, that is, the information candidate list is A user, B user and C user.

[0622] In another implementation manner, the image memory module can preferentially place the candidate user information associated with the historical user information in front of the information candidate list, and place the candidate user information not associated with the historical user information at the back of the information candidate list. Specifically, the image memory module can obtain the associated user information of each candidate user information in the user information library. Then, the image memory module can sequentially determine whether the historical user information is same as the associated user information of the candidate user information. If the historical user information is same as the associated user information of any candidate user information, the image memory module can place the candidate user information at the first position of the information candidate list. For example, the candidate user information are the mobile phone number “189****4045” of A user and the mobile phone number “189****4123” of C user respectively, and the historical user information is A user. It can be seen that the image memory module can determine that the mobile phone number “189****4045” is the candidate user information associated with the historical user information, and therefore, the image memory module can place the mobile phone number “189****4045” at the first position of the information candidate list, and place the mobile phone number “189****4123” at the second position of the information candidate list, that is, the information candidate list is A user and C user.

[0623] It should be noted that the historical user information described above can be carried by the information query indication described above, that is, after receiving the trigger operation of the user on any fill-in control in the to-be-entered interface, the form filling assistant can send the information query indication carrying the historical user information and the attribute information of the name control corresponding to the triggered fill-in control to the image memory module, thereby providing a basis for subsequent accurate display of recommended information. Then, the image memory module can store the historical user information in the user information library, thereby providing a basis for subsequent calling of user information.

[0624] In some cases, before storing the historical user information into the user information library, the portrait memory module can perform mapping processing on the label name to which the historical user information belongs, to obtain a mapped label name. Then, the portrait memory module can store the mapped label name as the label name of the historical user information and the historical user information into the user information library. In this way, the label names in the user information library can all be standardized label names, that is, the label names corresponding to the same type of user information are all the same, so as to facilitate subsequent accurate calling of user information and improve the calling efficiency of user information.

[0625] It can be understood that the historical user information can have certain rules (such as the format of the mobile phone number is 11 digits, the email number contains an "@" symbol), therefore, the portrait memory module can perform mapping processing on the label name to which the historical user information belongs according to the historical user information and a preset information rule, to obtain a mapped label name. The preset information rule includes but is not limited to at least one of the following: the historical user information includes an "@" character, the total number of characters of the historical user information is 11 digits, and the number of characters of the historical user information is within a preset number interval. For example, the historical user information is "189****4045", and the label name to which the historical user information belongs is a contact method. Since the preset format of the mobile phone number is 11 digits, the portrait memory module can determine that the mapped label name to which the historical user information belongs is a mobile phone number, that is, the portrait memory module can standardize the contact method to a mobile phone number. For another example, the historical user information is "Zhao Yi Yi", and the label name to which the historical user information belongs is a sender. Since the number of characters of the name is generally 2-4, that is, the preset number interval is 2-4, the portrait memory module can determine that the mapped label name to which the historical user information belongs is a name, that is, the portrait memory module can standardize the sender to a name.

[0626] S3008, the portrait memory module sends the candidate user information to the form filling assistant.

[0627] In some embodiments, after obtaining the above-mentioned candidate user information, the portrait memory module can send the candidate user information to the form filling assistant. For example, if the number of information of the candidate user information is 1, the portrait memory module can directly send the candidate user information to the form filling assistant. For another example, if the number of information of the candidate user information is not 1, the portrait memory module can send the above-mentioned information candidate list to the form filling assistant, that is, send the arranged candidate user information to the form filling assistant.

[0628] S3009, the form filling assistant displays the candidate user information in the to-be-entered interface in the case of receiving the candidate user information sent by the portrait memory module.

[0629] In some embodiments, after receiving candidate user information from the profile memory module, the form-filling assistant can display that candidate user information on the input interface. For example, if the number of candidate user information entries is one, the form-filling assistant can directly display that entry. Alternatively, if the number of candidate user information entries is not one, the form-filling assistant can display the candidate user information sequentially on the input interface according to the order in the candidate information list. This allows users to easily select the appropriate user information, providing convenient conditions for quickly filling in information and thus improving the user experience.

[0630] In one implementation, the form-filling assistant can display candidate user information floating below the triggered form-filling control. For example, such as... Figure 31 As shown, the phone is displaying... Figure 31 In interface (a), if the input control 3121 corresponding to the "Recipient" control is triggered by the user, the mobile phone can display the candidate user information for the "Recipient" in a floating manner below the input control 3121. In other words, the mobile phone can display... Figure 31 The interface is shown in (b). The candidate user information for "Recipient" may include Zhao Yiyi and Zhao Sansan.

[0631] In another implementation, the form-filling assistant can display candidate user information as a dropdown list based on the triggered form-filling control. The dropdown list allows the user to select user information from the candidate list to fill in.

[0632] It's understandable that, to simplify the display of user information, if the number of candidate user information entries is not one, the form-filling assistant can only display a preset number of candidate user information entries that appear at the beginning of the candidate list. This preset number of entries can be set according to actual needs; for example, it can be one, three, etc., without specific limitations. For instance, if the candidate list includes 10 candidate user information entries, the form-filling assistant can retrieve the preset number (e.g., three) of candidate user information entries that appear at the beginning of the list and display that preset number. In other words, the phone can only display the first three candidate user information entries. This allows for accurate display of user information, reducing interface clutter caused by displaying too much information, improving interface cleanliness, and facilitating subsequent selection of user information by the user.

[0633] S3010, in response to the user's selection operation for any candidate user information, the form filling assistant displays the selected candidate user information in the triggered fill-in control.

[0634] In some embodiments, after the candidate user information is displayed in the interface to be filled in, if any candidate user information is selected by the user, the form-filling assistant can display the selected candidate user information in the triggered fill-in control. For example, referring to FIG. 27, the mobile phone is displaying the interface shown in (b) of FIG. 27, and if the candidate user information "Zhao Yiyi" is clicked by the user, the mobile phone can directly display "Zhao Yiyi" in the fill-in control 3121, that is, display the interface shown in (c) of FIG. 27. In this way, intelligent filling of user information can be realized, and the user only needs to select the corresponding user information according to the actual needs to fill in, without manually filling in the user information, simplifying the information filling process, improving the filling efficiency of the user information, and further improving the user experience. Label name Target label name Contact

[0635] It can be understood that in the case where the triggered fill-in control displays user information, that is, displays selected candidate user information or filled-in user information, the form-filling assistant can send the user information to the portrait memory module, so that the portrait memory module stores the user information in the user information library. In this way, not only can the user information be saved in time, improving the storage efficiency of the user information, but also the mobile phone can determine the input content of the blank fill-in control in the current interface (that is, the interface to be filled in), so that the input content is associated with the user information, that is, the user information of the same user, improving the selection efficiency of the candidate user information, and further improving the user experience.

[0636]

[0637] S3011, in the case where the detection result indicates that the label name cannot be standardized, the portrait memory module sends the label name to a name management module in the cloud service.

[0638] In some embodiments, after the detection result indicates that the label name cannot be standardized, the portrait memory module can send the label name to a name management module in the cloud service (or server) for subsequent standardized learning of the label name, thereby improving the standardization efficiency of the attribute name.

[0639] The cloud service can include a name management module for standardized learning of the label name of the name control.

[0640] ​​​​In some embodiments, after the detection result indicates that the label name cannot be standardized, the portrait memory module can send the detection result to the form filling assistant. Then, the form filling assistant continues to display the above-mentioned to-be-entered interface, that is, does not display any candidate user information in the to-be-entered interface, in a case where the form filling assistant receives the detection result sent by the portrait memory module. Then, in response to the information entering operation of the user on the triggered fill-in control, the form filling assistant can display the entered user information in the triggered fill-in control. It can be understood that the entered user information is obtained based on the information entering operation of the user. That is, if the label name cannot be standardized, it means that the mobile phone cannot query the corresponding user information from the user information library, that is, the mobile phone cannot display the candidate user information on the to-be-entered interface, so the mobile phone can continue to display the to-be-entered interface to remind the user to manually fill in the user information.

[0641] Then, the form filling assistant can send the above-mentioned entered user information to the portrait memory module as historical user information. Then, the portrait memory module, in a case where the portrait memory module receives the historical user information sent by the form filling assistant, maps the label name of the historical user information according to the historical user information and the above-mentioned preset information rule, to obtain a mapped label name. Then, the portrait memory module can store the mapped label name as the label name of the historical user information and the historical user information in the user information library. In this way, the label names in the user information library are all standardized label names, that is, the label names corresponding to the same type of user information are all the same, so as to facilitate subsequent accurate calling of user information and improve the calling efficiency of user information.

[0642] S3012, the name management module, in a case where the name management module receives the label name sent by the portrait memory module, standardizes the label name to obtain a target attribute name.

[0643] Specifically, after the name management module receives the label name sent by the portrait memory module, the name management module can save the label name. Then, the name management module can standardize the label name every interval of a preset time to obtain a target label name. In this way, the accuracy of determining the target label name can be improved, thereby providing a basis for subsequent determination of the target label name.

[0644] In some embodiments, the operation and maintenance personnel can standardize the above-mentioned label name to obtain a target label name. In this way, the accuracy of determining the target label name can be improved by artificial standardization, thereby providing a basis for subsequent determination of the target label name.

[0645] In some embodiments, the name management module can normalize the label name according to a name adjustment model to obtain a target label name. Specifically, the name management module can input the label name into the name adjustment model to obtain the target label name. The name adjustment model is a supervised learning algorithm. The name adjustment model is trained based on a label name set, which includes a plurality of label names and the real label name carried by each label name. That is, the name management module can input the label name into the name adjustment model to be trained to obtain a predicted label name. Then, the name management module can adjust the parameters of the name adjustment model according to the preset label name and the real label name to obtain a trained name adjustment model. In this way, the accuracy of determining the target label name can be improved by machine learning, without the need for manual standardization learning, thereby improving the standardization efficiency of the label name and providing a basis for subsequent determination of the target label name.

[0646] S3013, the name management module sends the mapping relationship between the target label name and the label name to the portrait memory module.

[0647] S3014, the portrait memory module adds the mapping relationship between the target label name and the label name to the preset mapping rule after receiving the mapping relationship between the target label name and the label name sent by the name management module.

[0648] Specifically, after receiving the mapping relationship between the target label name and the label name sent by the name management module, the portrait memory module can add the mapping relationship to the preset mapping rule to facilitate subsequent standardization of the label name. The preset mapping rule includes a plurality of mapping relationships between label names and target label names.

[0649] In an implementation manner, the portrait memory module can display the preset mapping rule through a mapping table. The mapping table is used to represent the mapping relationship between a plurality of label names and target label names.

[0650] In some embodiments, the mapping table can include a plurality of target label names and the label name corresponding to each target label name. That is, the target label name in the preset mapping rule can be determined based on the label name.

[0651] Exemplarily, as shown in Table 1, the mapping table can include a label name "contact", a target label name "name" corresponding to the label name "contact", a label name "recipient", a target label name "name" corresponding to the label name "recipient", a label name "contact number", a target label name "mobile number" corresponding to the label name "contact number", a label name "mobile number", and a target label name "mobile number" corresponding to the label name "mobile number". That is, the label name "contact" and the label name "recipient" both correspond to the target label name "name", and the label name "contact number" and the label name "mobile number" both correspond to the target label name "mobile number".

[0652] Table 1

[0653] Name Recipient Name Contact number Mobile phone number Mobile phone Mobile phone number Interface belonging application Interface title name Label name

[0654] The mapping table can further include an interface title name corresponding to each target label name and an application to which the interface belongs. That is, the target label name in the preset mapping rule can be determined based on the interface title name, the application to which the interface belongs, and the label name. The interface title name is interface information displayed at the top of the mobile display interface. It can be understood that the interface title name is used to represent the purpose of filling in user information in the mobile display interface. In this way, a basis for subsequent accurate determination of the target label name is provided.

[0655] Exemplarily, as shown in Table 2, the mapping table includes a target label name "mobile number" corresponding to an application "application 1" to which an interface belongs, an interface title name "add sender information", and a label name "contact information", a target label name "email number" corresponding to the application "application 1" to which the interface belongs, an interface title name "apply for invoice", and a label name "contact information", a target label name "file number" corresponding to an application "application 2" to which an interface belongs, an interface title name "add student file", and a label name "ID number", and a target label name "ID number" corresponding to an application "application 3" to which an interface belongs, an interface title name "add travel information", and a label name "ID number".

[0656] Table 2

[0657] Target label name Application 1 Add sender information Contact information Mobile phone number Application 1 Apply for invoice Contact information Email number Application 2 Add student file Certificate number File number Application 3 Add travel information Certificate number ID number Figure 29 Figure 35 Figure 29

[0658] The above describes a specific implementation process of how the mobile phone displays corresponding candidate user information according to the triggered filling control for the user to select. Next, the process of how to standardize the label name will be described in detail in combination with the structure shown in Figure 36 and the standardization process of the label name shown in Figure 33 .

[0659] S3501, the image memory module invokes a preset mapping rule.

[0660] Specifically, after obtaining the attribute information of the name control, the image memory module can invoke the preset mapping rule. It can be understood that the plurality of mapping relationships included in the preset mapping rule can be obtained from the name management module in the cloud service. For example, the image memory module can obtain new mapping relationships within a preset period from the name management module every interval of the preset period. For another example, the name management module can directly send new mapping relationships to the image memory module in the case of generating new mapping relationships.

[0661] S3502, the image memory module determines whether the attribute information of the name control satisfies any mapping relationship in the preset mapping rule.

[0662] In some embodiments, after invoking the preset mapping rule, the image memory module can determine whether the attribute information of the name control satisfies any mapping relationship in the preset mapping rule. If the attribute information of the name control satisfies any mapping relationship in the preset mapping rule, it means that the attribute information of the name control satisfies the preset condition, that is, the label name can be standardized, and therefore the image memory module can perform S3503. If the attribute information of the name control does not satisfy any mapping relationship in the preset mapping rule, it means that the label name can not be standardized, and therefore, in order to further determine whether the label name can be standardized, the image memory module can perform S3504.

[0663] In an implementation manner, if the mapping table representing the preset mapping rule only includes the label name and the target label name, that is, the target label name can be directly determined based on the label name, therefore, the image memory module can only determine whether there is a label name same as the label name of the name control in the preset mapping rule. If there is a label name same as the label name of the name control in the preset mapping rule, the image memory module can determine that the attribute information of the name control satisfies the preset condition. If there is no label name same as the label name of the name control in the preset mapping rule, that is, the label name is different from the label name in the preset mapping rule, the image memory module can determine that the attribute information of the name control does not satisfy the preset condition.

[0664] For example, if the label name of the name control is "recipient" and the mapping table shown in Table 1 also includes the label name "recipient", it indicates that the mapping table includes the same label name as the label name "recipient" of the name control, and thus the image memory module can determine that the attribute information of the name control satisfies the preset condition, i.e., the label name "recipient" of the name control can be standardized. If the label name of the name control is "consignee" and the mapping table shown in Table 1 does not include "consignee", it indicates that the mapping table does not include the same label name as the label name "consignee" of the name control, and thus the image memory module can determine that the attribute information of the name control does not satisfy the preset condition, i.e., the label name "consignee" of the name control cannot be standardized.

[0665] In another implementation, if the mapping table representing the preset mapping rule includes the application to which the interface belongs, the interface title name, the label name, and the target label name, i.e., the target label name is determined based on the interface title name, the application to which the interface belongs, and the label name, the image memory module needs to determine whether the preset mapping rule includes a mapping relationship that is the same as the application to which the interface belongs, the interface title name, and the label name included in the attribute information. If the preset mapping rule includes a mapping relationship that is the same as the application to which the interface belongs, the interface title name, and the label name included in the attribute information, the image memory module can determine that the attribute information of the name control satisfies the preset condition. If the preset mapping rule does not include a mapping relationship that is the same as the application to which the interface belongs, the interface title name, and the label name included in the attribute information, i.e., any content included in the attribute information is different from the mapping relationship in the preset mapping rule, the image memory module can determine that the attribute information of the name control does not satisfy the preset condition.

[0666] For example, please refer to Table 2. If the application to which the interface belongs is Application 1, the interface title is "Add Item Sending Information," and the label name of the name control is "Contact Information," and the mapping table shown in Table 2 includes mapping relationships that simultaneously satisfy the application "Application 1," the interface title "Add Item Sending Information," and the label name of the name control "Contact Information," the profile memory module can determine that the attribute information of the name control meets the preset conditions, that is, the label name "Contact Information" of the name control can be standardized. If the application to which the interface belongs is Application 2, the interface title is "Add Item Sending Information," and the label name of the name control is "Contact Information," and the mapping table shown in Table 2 does not include mapping relationships that simultaneously satisfy the application "Application 2," the interface title "Add Item Sending Information," and the label name of the name control "Contact Information," the profile memory module can determine that the attribute information of the name control does not meet the preset conditions, that is, the label name "Contact Information" of the name control cannot be standardized.

[0667] S3503, the portrait memory module uses the target label name included in the mapping relationship satisfied by the attribute information as the target label name of the name control.

[0668] It is understandable that if the attribute information of the aforementioned name control satisfies any of the mapping relationships in the preset mapping rules, it means that the label name of the name control has been standardized. Therefore, the profile memory module can directly use the target label name in the mapping relationship satisfied by the attribute information of the name control as the target label name of the name control. In this way, not only can the power consumption loss caused by repeated standardization be reduced and the utilization rate of mobile phone resources be improved, but the standardization efficiency of label names can also be improved, thereby speeding up the efficiency of information display and providing convenient conditions for users to select the corresponding information later.

[0669] S3504, the profile memory module determines whether there exists a tag name in the user information database whose similarity to the tag name of the name control is greater than the preset similarity.

[0670] It is understandable that if the attribute information of the aforementioned name control does not satisfy any of the mapping relationships in the preset mapping rules, it indicates that the label name of the name control has not been standardized. Therefore, in order to further determine whether the label name of the name control can be standardized, the profile memory module can continue to determine whether there is a label name in the user information database whose similarity to the label name of the name control is greater than a preset similarity. This preset similarity can be pre-set based on actual conditions. In this way, multi-angle standardization of label names can be achieved, meaning that label names can be standardized in different ways, providing a foundation for the accurate display of candidate user information subsequently.

[0671] In some embodiments, if there is a label name in the user information library that has a similarity greater than a preset similarity with the label name of the name control, it indicates that the label name of the name control is more likely to be the same as the label name in the user information library, that is, the label name of the name control can be standardized, and therefore the image memory module can perform S3505. If there is no label name in the user information library that has a similarity greater than a preset similarity with the label name of the name control, that is, the similarity between each label name in the user information library and the label name of the name control is less than or equal to the preset similarity, that is, the label name of the name control cannot be standardized, and therefore the image memory module can perform S3506.

[0672] Specifically, for each label name in the user information library, the image memory module can calculate the similarity between the label name and the label name of the name control. The similarity is used to represent the similarity of the label name of the name control and the label name. Then, in the case that there is any label name in the user information library that has a similarity greater than a preset similarity with the label name of the name control, the image memory module can determine that the attribute information of the name control satisfies the preset condition, that is, the label name of the name control can be standardized. In the case that the similarity between each label name in the user information library and the label name of the name control is less than or equal to the preset similarity, the image memory module can determine that the attribute information of the name control does not satisfy the preset condition, that is, the label name of the name control cannot be standardized.

[0673] The similarity can be determined based on the semantic vector (or first semantic vector) of the label name of the name control and the semantic vector (or second semantic vector) of the label name in the user information library. Specifically, the image memory module can extract features from the label name of the name control based on a semantic vector model to obtain the first semantic vector. The semantic vector model is used to map words to vectors, so that each word corresponds to a semantic vector. Then, the image memory module can obtain the second semantic vector of each label name in the user information library. The second semantic vector is obtained by feature extraction based on the semantic vector model. Then, for the second semantic vector of each piece of user information in the user information library, the image memory module can calculate the similarity between the second semantic vector and the first semantic vector, that is, the similarity between the label name of the user information and the label name.

[0674] For example, if the semantic vector of the username is [0.99, 0.12, 0.03, 0.8], the semantic vector of the name in the user information library is [0.79, 0.42, 0.08, 0.7], and the semantic vector of the mobile phone number in the user information library is [0.19, 0.22, 0.48, 0.7], the image memory module calculates the similarity between the semantic vector of the username and the semantic vector of the name as 0.88, and the similarity between the semantic vector of the username and the semantic vector of the mobile phone number as 0.60. It can be seen that the similarity between the semantic vector of the username and the semantic vector of the name is 0.88, which is greater than the preset similarity 0.85, and therefore the image memory module can determine that the label name "username" of the name control meets the preset condition.

[0675] In S3505, the image memory module determines the label name of the name control as the target label name if the similarity between the label name and the label name of the name control is greater than the preset similarity.

[0676] It can be understood that if the similarity between the label name in the user information library and the label name of the name control is greater than the preset similarity, it means that the label name in the user information library is similar to the label name of the name control, and therefore the image memory module can directly determine the label name with the similarity greater than the preset similarity as the target label name of the name control. In this way, the target label name can be quickly and accurately determined, which provides a basis for subsequent determination of the candidate user information.

[0677] In some embodiments, if there are multiple label names in the user information library with the similarity greater than the preset similarity with the label name of the name control, the image memory module can determine all the multiple label names as the target label name of the name control. For example, if the label name of the name control is the certificate number, and the label names in the user information library can include the ID number, the student number, and the filing number, the similarity between the ID number and the certificate number is 1, the similarity between the student number and the certificate number is 2, and the similarity between the filing number and the certificate number is 3, all of which are greater than the preset similarity, the image memory module can determine the ID number, the student number, and the filing number as the target label name of the name control. In this way, the situation of missing display of user information can be reduced, and the accuracy of information display can be improved.

[0678] In S3506, the image memory module uploads the label name of the name control to the cloud service.

[0679] Specifically, after determining that there is no tag name in the aforementioned user information database whose similarity to the tag name of the name control is greater than a preset similarity, it indicates that the tag name of the name control cannot be directly standardized. Therefore, the profile memory module can upload the tag name of the name control to the cloud service. Subsequently, the name management module in the cloud service can perform standardization learning on the tag name of the name control. This provides a foundation for subsequent comprehensive standardization of tag names.

[0680] It should be noted that the above-described standardization process for tag names is based on the premise that the attribute information of the name control satisfies any mapping relationship in the preset mapping rules, and that the similarity between a tag name in the user information database and the tag name of the name control is greater than a preset similarity. However, if the preset condition only includes that the attribute information of the name control satisfies any mapping relationship in the preset mapping rules, the mobile phone can execute only processes S3501 to S3503, and upload the tag name of the name control to the cloud service if the attribute information of the name control does not satisfy any mapping relationship in the preset mapping rules. If the preset condition only includes that the similarity between a tag name in the user information database and the tag name of the name control is greater than a preset similarity, the mobile phone can execute only processes S3504 to S3506.

[0681] The above describes the specific implementation process of how the portrait memory module in a mobile phone standardizes the label names of name controls. The following will combine the above... Figure 33 The structure shown and Figure 33 The process of obtaining user information from the user information database is shown in detail. It describes how user information is obtained from a preset collection application and stored in the user information database. Specifically, it may include steps S3601 to S3606:

[0682] S3601, the function settings module in the mobile phone receives the user's trigger operation on any switch control in the settings interface.

[0683] The aforementioned settings interface (or information collection settings interface) is used to configure whether applications on the phone can collect user information. For example, this settings interface can be... Figure 33 The interface shown in (a) is shown in the middle.

[0684] S3602, in response to the user's trigger operation on any switch control in the settings interface, the function setting module sends a function setting instruction to the image memory module.

[0685] In some embodiments, if any of the switch controls in the setting interface is triggered by the user, it indicates that the user wants to adjust the information collection function of the application to which the switch control belongs, that is, the user wants to turn on or turn off the information collection function. Therefore, the function setting module can send a function setting instruction to the portrait memory module. The function setting instruction carries a setting result of the information collection function. The setting result is used to indicate whether the information collection function of the application to which the triggered switch control belongs is turned on.

[0686] It can be understood that the application program to which each switch control belongs is pre-set. For example, in the (a) interface in FIG. 33, the application program to which the switch control 3301A belongs is all the applications in the mobile phone, that is, as long as the switch control 3301A is turned on by the user, the mobile phone can collect the user information saved by all the applications. The application program to which the switch control 3302 belongs is the message application in the mobile phone, that is, as long as the switch control 3302 is turned on by the user, the mobile phone can collect the user information saved by the message application. Figure 29

[0687] S3603, in a case where the function setting instruction sent by the function setting module is received, the portrait memory module updates the collection application list according to the setting result carried by the function setting instruction. The collection application list includes at least one preset collection application.

[0688] Specifically, after receiving the function setting instruction, the portrait memory module can update the collection application list according to the setting result carried by the function setting instruction. In this way, not only the privacy of the user can be protected, but also a basis for subsequent acquisition of user information is provided.

[0689] In an implementation manner, if the setting result indicates that the information collection function of the application to which the triggered switch control belongs is turned on, the portrait memory module can add the application to which the triggered switch control belongs to the collection application list to obtain a new collection application list.

[0690] In another implementation manner, if the setting result indicates that the information collection function of the application to which the triggered switch control belongs is turned off, the portrait memory module can delete the application to which the triggered switch control belongs from the collection application list to obtain a new collection application list.

[0691] S3604, the portrait memory module judges whether the application to which the triggered switch control belongs is an application program in the collection application list.

[0692] ​In some embodiments, after receiving the function setting instruction, the portrait memory module can determine whether the application to which the triggered switch control belongs is an application in the collection application list. If the application to which the triggered switch control belongs is an application in the collection application list, it indicates that the user wants to turn off the information collection function of the application. Therefore, the portrait memory module can perform S3605. If the application to which the triggered switch control belongs is not an application in the collection application list, it indicates that the user wants to turn on the information collection function of the application. Therefore, the portrait memory module can perform S3606.

[0693] S3605, in the case where the application to which the triggered switch control belongs is an application in the collection application list, the portrait memory module deletes all user information associated with the application from the user information library.

[0694] Specifically, if the application to which the triggered switch control belongs is an application in the collection application list, the portrait memory module can delete all user information associated with the application from the user information library. In this way, all user information associated with the application can be deleted while the information collection function of the application is turned off, thereby protecting user privacy and improving the security of mobile phone use.

[0695] In some embodiments, in the case where the application to which the triggered switch control belongs is an application in the collection application list, it indicates that the user does not want to disclose the user information entered by the application. Therefore, the portrait memory module can directly delete all user information associated with the application from the user information library. In this way, the information privacy of the user can be protected, and security assurance can be provided for the user.

[0696] In some embodiments, in the case where the application to which the triggered switch control belongs is an application in the collection application list, the mobile phone can display a pop-up interface. The pop-up interface can include prompt information, a delete control, and a retention control. The prompt information is used to represent whether the user information about the application to which the triggered switch control belongs saved in the user information library is retained. Then, in response to the triggering operation of the user on the delete control in the pop-up interface, the portrait memory module in the mobile phone can delete all user information associated with the application from the user information library. Alternatively, in response to the triggering operat...

Claims

1. An information entry method, characterized in that, Applied to an electronic device having multiple applications installed, the method includes: The electronic device displays an input interface; wherein the input interface includes multiple input controls and a name control corresponding to each input control; In response to a user's triggering operation on any fill control in the input interface, the electronic device obtains the label name of the name control corresponding to the triggered fill control; The electronic device selects candidate user information from a user information database that corresponds to the label name of the name control; wherein, the user information database includes multiple pieces of user information, which are collected by the electronic device from the multiple applications; The electronic device displays the candidate user information on the input interface.

2. The method according to claim 1, characterized in that, The electronic device selects candidate user information from the user information database that corresponds to the label name of the name control, including: The electronic device performs standardization processing on the label name of the name control to obtain the target label name of the name control; wherein, the target label name is any label name existing in the user information database; The electronic device selects candidate user information corresponding to the target tag name from the user information database.

3. The method according to claim 2, characterized in that, The electronic device performs standardization processing on the label name of the name control to obtain the target label name of the name control, including: If the attribute information of the name control satisfies any mapping relationship in the preset mapping rules, and / or if there is a tag name in the user information database with a similarity greater than the preset similarity between the tag name and the tag name of the name control, the electronic device performs standardization processing on the tag name of the name control to obtain the target tag name; wherein, the attribute information of the name control includes at least one of the tag name of the name control, the application of the interface to which the name control belongs, and the title name of the interface to which the name control belongs, and the preset mapping rules include multiple mapping relationships between tag names and target tag names.

4. The method according to claim 2, characterized in that, The electronic device performs standardization processing on the label name of the name control to obtain the target label name of the name control, including: If the attribute information of the name control does not satisfy any of the mapping relationships in the preset mapping rules, and there exists a tag name in the user information database whose similarity to the tag name of the name control is greater than a preset similarity, the electronic device will use the tag name with the greater than preset similarity as the target tag name of the name control; or, If the attribute information of the name control satisfies any of the mapping relationships in the preset mapping rules, the electronic device shall use the target tag name included in the mapping relationship satisfied by the attribute information as the target tag name of the name control.

5. The method according to claim 3 or 4, characterized in that, The method further includes: If the attribute information of the name control does not satisfy any of the mapping relationships in the preset mapping rules, and there is no tag name in the user information database whose similarity to the tag name of the name control is greater than the preset similarity, the electronic device will send the tag name of the name control to the server; wherein, the server is used to perform standardized learning on the tag name of the name control. In response to the user's information entry operation on the triggered fill-in control, the electronic device displays the entered user information in the triggered fill-in control; wherein the entered user information is obtained based on the user's information entry operation.

6. The method according to any one of claims 1-5, characterized in that, After the electronic device selects candidate user information corresponding to the label name of the name control from the user information database, the method further includes: When there are multiple candidate user information items, the electronic device sorts the multiple candidate user information items to obtain an information candidate list; wherein, the information candidate list is used to represent the arrangement order of the multiple candidate user information items; The electronic device displays the candidate user information on the input interface, including: The electronic device displays the candidate user information in the input interface according to the order in which the candidate user information is arranged.

7. The method according to claim 6, characterized in that, The electronic device sorts the multiple candidate user information to obtain a candidate information list, including: The electronic device sorts multiple candidate user information according to a preset display rule to obtain the information candidate list; wherein, the preset display rule includes at least one of the following: the order of information update time, the order of decreasing information entry count, and association with historical user information; the information update time is the time when the candidate user information is added to the user information database; the information entry count is the number of times the candidate user information is selected by the user; and the historical user information is the user information that has been entered in the interface to be entered.

8. The method according to any one of claims 1-7, characterized in that, Before the electronic device displays the input interface, the method further includes: In response to the user's information entry operation on the information editing interface, the electronic device updates the entered user information to the user information database; wherein, the information editing interface is the display interface in the settings application of the electronic device; and / or, The electronic device obtains new user information from a preset collection application and saves the new user information to the user information database; wherein, the preset collection application is an application with information collection function among the multiple applications.

9. The method according to claim 8, characterized in that, The electronic device obtains new user information from a preset collection application, including: In response to the user's information extraction operation on the display interface of the preset collection application, the electronic device performs identity authentication on the user and obtains an identity authentication result; wherein, the display interface includes at least one piece of user information, and the identity authentication result is used to indicate whether the user is the owner of the device; If the authentication result indicates that the user is the owner of the device, the electronic device extracts information from the display interface to obtain the new user information.

10. The method according to claim 8 or 9, characterized in that, Saving the new user information to the user information database includes: The electronic device retrieves multiple user information entries corresponding to the user to whom the new user information belongs from the user information database; If no user information with the same tag name as the new user information exists among the multiple user information entries, the electronic device adds the new user information to the user information database.

11. The method according to claim 10, characterized in that, The method further includes: If there is a user information with the same tag name as the new user information among the multiple user information, the electronic device will use the user information with the same tag name as the new user information as the original user information; If the original user information and the new user information are different, the electronic device displays an information update interface; In response to the user's trigger operation on the update control in the information update interface, the electronic device updates the user information database by replacing the original user information with the new user information.

12. The method according to any one of claims 8-11, characterized in that, The method further includes: In response to the shutdown operation of the information collection function of the preset collection application, the electronic device deletes all user information associated with the preset collection application whose information collection function has been shut down from the user information database.

13. The method according to any one of claims 8-12, characterized in that, The preset collection applications include at least one of SMS applications, photo gallery applications, and note-taking applications.

14. The method according to any one of claims 1-13, characterized in that, The method further includes: In response to the user's information adjustment operation, the electronic device adjusts the user information database.

15. The method according to claim 14, characterized in that, In response to the user's information adjustment operation, the electronic device adjusts the user information database, including: In response to the user's deletion operation on any user information in the user information database, the electronic device deletes the user information from the user information database; and / or, In response to the user's information modification operation on any user information in the user information database, the electronic device modifies the user information to obtain modified user information, and updates the modified user information to the user information database; and / or, In response to the user's operation of adding information to the user information database, the electronic device adds the newly added user information to the user information database.

16. The method according to any one of claims 1-15, characterized in that, The method further includes: In response to the user's selection of the candidate user information, the electronic device displays the selected candidate user information in the triggered fill-in control.

17. The method according to any one of claims 1-16, characterized in that, After the electronic device displays the input interface, the method further includes: In response to a user's one-click input operation on the input interface, the electronic device obtains the label name of each name control in the input interface; The electronic device performs standardization processing on the label name of each name control to obtain the target label name of the multiple name controls; The electronic device divides the target tag names of the multiple name controls to obtain at least one set of sub-attribute information lists; wherein, the sub-attribute information list includes the target tag name of at least one name control, and the target tag name of the at least one name control is the tag name of the same user; For each sub-attribute information list, the electronic device selects at least one candidate user information group that matches the sub-attribute information list from the user information database; The electronic device displays at least one candidate user information group corresponding to each group of sub-attribute information lists on the input interface.

18. The method according to claim 17, characterized in that, The electronic device divides the target tag names of the multiple name controls to obtain at least one set of sub-attribute information lists, including: When the target label names of the multiple name controls include multiple identical target label names, the electronic device divides the target label names of the multiple name controls according to the multiple identical target label names to obtain at least one set of sub-attribute information lists; wherein, each set of sub-attribute information lists includes one of the multiple identical target label names.

19. The method according to claim 17 or 18, characterized in that, The method further includes: In response to the user's selection operation for any candidate user information group corresponding to the sub-attribute information list, the electronic device determines at least one fill-in control corresponding to the selected candidate user information group from the plurality of fill-in controls to which the sub-attribute information list belongs; The electronic device displays the candidate user information in the at least one fill-in control.

20. An electronic device, characterized in that, include: A display screen, one or more processors, and one or more memories; the one or more processors are coupled to the display screen and the one or more memories; the display screen is used to display an interface to be entered, and the one or more memories are used to store computer program code, the computer program code including computer instructions, which, when executed by the one or more processors, cause the electronic device to perform the method as described in any one of claims 1-19.

21. A computer-readable storage medium, characterized in that, Includes computer instructions that, when executed on an electronic device, cause the electronic device to perform the method as described in any one of claims 1 to 19.

22. A computer program product, characterized in that, Includes computer instructions that, when executed on an electronic device, cause the electronic device to perform the method as described in any one of claims 1 to 19.

Images