Information processing method, electronic equipment, computer program product and readable storage medium

Abstract

The invention provides an information processing method, electronic equipment, a computer program product and a readable storage medium, and relates to the technical field of terminals. The electronic equipment displays a first interface, the first interface comprises a first filling box and a plurality of label boxes, each label box has a corresponding label name, and different label boxes have different label names. And the electronic equipment responds to a first operation acting on a first filling box in the first interface, and determines a corresponding first content text for the first filling box. Therefore, under the condition that the electronic equipment displays that the first interface is the form filling interface, the first filling box indicated to be filled by the user is detected, the first content text can be automatically matched for the first filling box without the need for the user to apply tedious input operation, and user operation is saved. In the embodiment of the invention, the electronic equipment searches the first tag box matched with the first filling box from the plurality of tag boxes, and matches the first content text for the first filling box according to the first tag name, so that the accuracy of the matched content text can be effectively improved.

Description

TECHNICAL FIELD

[0001] Embodiments of the present application relate to the technical field of terminal, and in particular, to an information processing method, an electronic device, a computer program product and a readable storage medium. BACKGROUND

[0002] With the development of computer technology, more and more application programs (APPs) are developed and installed on electronic devices to perfect the functions and individualization of the electronic devices and provide users with more abundant use experiences.

[0003] However, for some information filling scenarios (such as first registration of an application program), the user usually needs to fill in relevant information (such as name, mobile phone number, etc.), which makes the user need to input the same information on different application programs, possibly leading to a cumbersome information input process of the user, wasting a long filling time of the user, and making the user have a poor use experience. SUMMARY

[0004] Embodiments of the present application provide an information processing method, an electronic device, a computer program product and a readable storage medium, to solve the technical problem that the existing form filling scheme is cumbersome.

[0005] To achieve the above object, embodiments of the present application adopt the following technical solutions:

[0006] In a first aspect, an information processing method is provided, applied to an electronic device. The electronic device displays a first interface, and the first interface is a form filling interface. Specifically, the first interface is an information filling interface in an application interface provided by the electronic device, such as a mail address filling interface, a car-hailing address filling interface, a registration information filling interface, etc.

[0007] The first interface displayed by the electronic device includes a first filling box and a plurality of label boxes, each label box has a corresponding label name, and different label boxes have different label names. The first filling box is a table box indicated for the user to fill in, and the content text needs to be filled in the filling box. The first interface further includes a plurality of label boxes, and the label boxes have label names such as name, gender, region, detailed address, etc. The label name of the label box is used to prompt to fill in the content text matching the label name in the matching filling box.

[0008] The electronic device determines a corresponding first content text for the first filling box in response to a first operation on the first filling box in the first interface. Specifically, the electronic device first searches for a first label box matching the first filling box from the plurality of label boxes, and the first label box has a first label name. Usually, the first label box matching the first filling box is located in the left adjacent column or the upper adjacent row of the first filling box.

[0009] The electronic device finds a first label box that matches the first input box. Based on the first label name of the first label box, it finds content text that matches the first label name and uses it as the first content text. The electronic device displays the first content text in the associated area of ​​the first input box. For example, the electronic device can either directly enter the first content text in the first input box or display the first content text in the adjacent associated area of ​​the first input box, allowing the user to choose whether to enter the first content text in the first input box.

[0010] In this way, when the electronic device displays a form-filling interface, it detects the first input box indicated by the user and automatically matches the first content text to the first input box without requiring tedious user input, saving user effort. Furthermore, the electronic device first searches for a matching label box from multiple label boxes, and then searches for matching first content text for the first input box based on the first label box's first label name. This effectively improves the accuracy of the matched content text.

[0011] In one possible implementation of the first aspect, a new electronic device is added to automatically match the content text of the second fill box.

[0012] The first interface displayed by the electronic device also includes a second input box, which is a different form from the first input box. The electronic device receives a second operation from the user on the second input box, which instructs the second input box to be filled automatically.

[0013] In response to the second operation, the electronic device determines a second label box from a plurality of label boxes that matches the second fill box. The second label box that matches the first fill box is a different table frame from the first label box, and the second label box has a second label name. The second label box that matches the second fill box is typically located in the column to the left or the row above the second fill box.

[0014] The electronic device finds a second label box that matches the second input box. Based on the second label name of the second label box, it finds content text that matches the second label name and uses it as the second content text. The electronic device displays the second content text in the associated area of ​​the second input box. For example, the electronic device can directly enter the second content text in the second input box, or it can display the second content text in the adjacent associated area of ​​the second input box, allowing the user to choose whether to enter the second content text into the first input box.

[0015] In this way, the electronic device can match the corresponding form content in real time and accurately based on the fill box selected by the user, so that the fill box can be filled automatically when the user points to it. This greatly facilitates the user's operation and increases the fun of automatic form filling.

[0016] In a possible implementation manner of the first aspect, the manner in which the electronic device displays the first content text is further limited. The electronic device displays, in the associated region of the first fill-in box, at least one selectable content text matched with the first label name, and the at least one selectable content text includes the first content text. The electronic device can display the first content text in the first fill-in box in response to a touch operation of the user on the first content text. Alternatively, in other cases, the electronic device can also fill the content text with a higher matching probability into the first fill-in box as the first content text.

[0017] In a possible implementation manner of the first aspect, the manner in which the electronic device identifies the first fill-in box is further limited. The electronic device acquires a cursor coordinate of a cursor in the first interface corresponding to the first operation, and determines the first fill-in box according to the cursor coordinate; the table box coordinate range of the first fill-in box in the first interface includes the cursor coordinate. The electronic device accurately identifies the first fill-in box indicated by the user for filling in according to the cursor position, and further improves the accuracy of automatic form filling.

[0018] In a possible implementation manner of the first aspect, the manner in which the electronic device identifies the table box and the text information of the first interface is further limited. The electronic device identifies a plurality of table boxes of the first interface, and identifies a plurality of text information of the first interface; the plurality of table boxes include fill-in boxes and label boxes. The electronic device determines the table box in which each text information is located, and determines the label name possessed by each label box, according to the text box coordinate of each text information and the table box coordinate range of the plurality of table boxes.

[0019] For example, the electronic device acquires the center point coordinate of the text bounding box of each text information, and the table box coordinate range of each table box, and judges whether the center point coordinate of the text bounding box of the text information is in the table box coordinate range of the table box. If the center point coordinate of the text bounding box of the text information is in the table box coordinate range of the table box, it is determined that the text information matches the table box, otherwise the text information does not match the table box.

[0020] In a possible implementation manner of the first aspect, the electronic device determines the first label box matched with the first fill-in box from the plurality of label boxes by using a predetermined Gaussian distribution model. The Gaussian distribution model is constructed based on the relative positions of a plurality of table box group samples, each table box group sample includes a sample label box and a sample fill-in box, and the relative positions of the sample label boxes and the sample fill-in boxes in different table box group samples are different.

[0021] In a specific example, the electronic device determines the first label box matched with the first fill-in box from the plurality of label boxes by using the predetermined Gaussian distribution model, which can specifically include: calculating a first distance between each label box and the first fill-in box, screening candidate label boxes with a first distance greater than a first threshold, calculating a matching probability of each candidate label box and the first fill-in box by using the Gaussian distribution model, and determining the candidate label box with the maximum matching probability as the first label box.

[0022] The electronic device constructs a Gaussian model by analyzing the relative position distribution of the matched fill-in boxes and label boxes in a large number of table samples, and finds the first label box matched with the first fill-in box according to the Gaussian model. The electronic device finds the matched first label box for the first fill-in box by constructing the Gaussian model, which has less calculation and higher accuracy.

[0023] In a possible implementation of the first aspect, the electronic device finds the first content text matched with the first label name from the table database before displaying the first content text in the associated region of the first fill-in box; the table database includes a plurality of sets of associated data, and each set of associated data includes a label name and a content text matched with each other.

[0024] Further, before finding the first content text matched with the first label name from the table database, the electronic device identifies the filled-in table box of the first interface; the filled-in table box is a fill-in box with filled-in content text; each associated data of the table database further includes a plurality of label names and content texts matched with each label name. The electronic device finds the first associated data including the filled-in content text from the table database, and finds the first content text matched with the first label name from the first associated data.

[0025] In this example, when matching the first content text for the first fill-in box, the electronic device can also refer to the filled-in content text, so that the accuracy of the matched first content text and the first fill-in box is higher.

[0026] In a possible implementation of the first aspect, the electronic device identifies the first interface currently displayed as a form filling interface in response to a click operation of the user on the first control; the first control is used to trigger automatic form filling in the form filling interface. The electronic device triggers automatic form filling in response to the touch operation of the user on the first interface, which is more convenient to operate.

[0027] In a possible implementation of the first aspect, the electronic device can also identify the cursor coordinate change amount of the first interface, and determine that the first interface currently displayed is a form filling interface when the cursor coordinate change amount is greater than a second threshold. The electronic device identifies whether it is a form filling interface according to the cursor coordinate change amount, and triggers automatic form filling when it is identified as a form filling interface, which further saves user operation.

[0028] In a second aspect, an electronic device is provided, which includes a touch screen, a memory and a processor, the touch screen and the memory are coupled to the processor; the memory stores computer-executed instructions; and the processor executes the computer-executed instructions stored in the memory, so that the electronic device executes the information processing method according to any one of the first aspect.

[0029] In a third aspect, an electronic device is provided, which has a function of implementing the information processing method according to the first aspect. The function can be implemented by hardware, or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above functions.

[0030] In a fourth aspect, a computer-readable storage medium is provided, which stores instructions, when the instructions are run on a computer, the computer can execute the information processing method according to any one of the first aspect.

[0031] In a fifth aspect, a computer program product is provided, which includes instructions, when the instructions are run on a computer, the computer can execute the information processing method according to any one of the first aspect.

[0032] The technical effects brought by any one of the second aspect to the fifth aspect can refer to the technical effects brought by different design manners of the first aspect, which will not be repeated here. BRIEF DESCRIPTION OF DRAWINGS

[0033] Figure 1 A schematic diagram of a table related to an embodiment of the present application;

[0034] Figure 2 An interface and operation schematic diagram of a form filling interface related to an information processing method provided by an embodiment of the present application;

[0035] Figure 3 A flowchart of edge selection and form filling related to an information processing method provided by an embodiment of the present application;

[0036] Figure 4 An interface and operation schematic diagram of edge selection and form filling related to an information processing method provided by an embodiment of the present application;

[0037] Figure 5 A schematic diagram of relative positions of table frames related to an information processing method provided by an embodiment of the present application;

[0038] Figure 6 A schematic diagram of element matching related to an information processing method provided by an embodiment of the present application;

[0039] Figure 7The automatic form filling process involved in the information processing method provided by the embodiment of the present application;

[0040] Figure 8 The screenshot of the first interface involved in the information processing method provided by the embodiment of the present application;

[0041] Figure 9 The table frame matching diagram involved in the information processing method provided by the embodiment of the present application;

[0042] Figure 10 The relative position diagram of the filling box and the matched label box involved in the information processing method provided by the embodiment of the present application;

[0043] Figure 11 The table frame coordinate diagram involved in the information processing method provided by the embodiment of the present application;

[0044] Figure 12 The flowchart of the information processing method provided by the embodiment of the present application;

[0045] Figure 13 The flowchart of the information processing method provided by the embodiment of the present application;

[0046] Figure 14 The display diagram of the mobile phone provided by the embodiment of the present application;

[0047] Figure 15 The display diagram of the mobile phone provided by the embodiment of the present application;

[0048] Figure 16 The diagram of the state machine provided by the embodiment of the present application;

[0049] Figure 17 The diagram of the state machine provided by the embodiment of the present application;

[0050] Figure 18 The diagram of the state machine provided by the embodiment of the present application;

[0051] Figure 19 The diagram of the state machine provided by the embodiment of the present application;

[0052] Figure 20 The diagram of the state machine provided by the embodiment of the present application;

[0053] Figure 21 The diagram of the state machine provided by the embodiment of the present application;

[0054] Figure 22 The diagram of the state machine provided by the embodiment of the present application;

[0055] Figure 23A A schematic diagram of part of a pattern of a table provided for an embodiment of the application;

[0056] Figure 23B A schematic diagram of five target categories of a table provided for an embodiment of the application;

[0057] Figure 24 A schematic diagram of part of a software structure of an electronic device provided for an embodiment of the application;

[0058] Figure 25 A schematic diagram of a flow of a table visual inspection method provided for an embodiment of the application;

[0059] Figure 26A A schematic diagram of a flow of a table visual inspection method provided for an embodiment of the application;

[0060] Figure 26B A schematic diagram of an intersection over union provided for an embodiment of the application;

[0061] Figure 27A A schematic diagram of a flow of a table visual inspection method provided for an embodiment of the application;

[0062] Figure 27B A schematic diagram of a flow of a table visual inspection method provided for an embodiment of the application;

[0063] Figure 28 A schematic diagram of a hardware structure of an electronic device provided for an embodiment of the application;

[0064] Figure 29 A schematic diagram of a software structure of an electronic device provided for an embodiment of the application;

[0065] Figure 30 A flowchart of an information entry method provided for an embodiment of the application;

[0066] Figure 31 A schematic diagram of an interface for filling in user information provided for an embodiment of the application;

[0067] Figure 32 A schematic diagram of an interface for adding user information provided for an embodiment of the application;

[0068] Figure 33 A schematic diagram of an interface for setting a preset collection application provided for an embodiment of the application;

[0069] Figure 34 A schematic diagram of an interface for extracting user information provided for an embodiment of the application;

[0070] Figure 35A method flowchart for label name standardization provided by an embodiment of the present application is shown in FIG. 1.

[0071] Figure 36 A method flowchart for updating a user information database provided by an embodiment of the present application is shown in FIG. 2.

[0072] Figure 37 A method flowchart for querying and adjusting user information provided by an embodiment of the present application is shown in FIG. 3.

[0073] Figure 38 An interface schematic diagram for querying user information provided by an embodiment of the present application is shown in FIG. 4.

[0074] Figure 39 An interface schematic diagram for deleting user information provided by an embodiment of the present application is shown in FIG. 5.

[0075] Figure 40 A method flowchart for information entry provided by an embodiment of the present application is shown in FIG. 6.

[0076] Figure 41 An interface schematic diagram for selecting user information provided by an embodiment of the present application is shown in FIG. 7. DETAILED DESCRIPTION

[0077] Exemplary embodiments of the present application are described herein with reference to the accompanying drawings, which are meant to be exemplary and not limiting. Therefore, it should be understood that various changes and modifications can be made to the embodiments described herein without departing from the scope of the present application. Also, for the sake of brevity and clarity, descriptions of well-known functions and constructions are omitted herein.

[0078] For the sake of understanding, some technical common knowledge related to embodiments of the present application is introduced first.

[0079] A table is a kind of data organization information, used for structured presentation of information. A table includes multiple table boxes, and the table boxes can contain content text. A table displayed by an electronic device can include multiple table box groups, and each table box group includes a label and a fill-in box. The label includes a label box and a label name, and the label name is in the label box.

[0080] An electronic device provides multiple application programs (APP), and an application interface of an application program can include a form filling interface. For example, a purchase application provided by the electronic device involves a delivery address filling interface, an account registration interface of an application program, a starting address filling interface of a car-hailing application, and the like. The electronic device displays a table on such interfaces, and the table includes a label box and a fill-in control, and the fill-in control includes a fill-in box, which can or can not contain filled-in text.

[0081] It should be noted that the form displayed by the electronic device in the form filling interface can be various forms of forms. For example, the form displayed by the electronic device only includes label text and a fill-in box, or the form includes a label box and a fill-in box. The information processing method related to the embodiments of the present application is applicable to the information processing process of the electronic device in the information filling interface of the application program, and is not applicable to the information processing process of the conventional Excel format form document.

[0082] As shown in Figure 1 , it is a schematic diagram of a form related to the embodiments of the present application. As shown in Figure 2 , it is an interface and operation schematic diagram of a form filling interface related to the information processing method provided by the embodiments of the present application.

[0083] The fill-in control includes a fill-in box and fill-in content, and the fill-in content is in the fill-in box. In the case that there is fill-in content in the fill-in box of the fill-in control, the fill-in content can be divided into prompt text and content text. In the case that there is no fill-in content in the fill-in box of the fill-in control, the fill-in content can be a null value, and the electronic device can fill the fill-in box with content in response to a user operation.

[0084] Each form can include a plurality of table frame groups, and the label box and the fill-in box of each table frame group are relatively close in the form. For example, the label box and the fill-in box of each table frame group are usually located in the same row and adjacent columns in the form, or the label box and the fill-in box of each table frame group are usually located in the same column and adjacent rows in the form.

[0085] As shown in Figure 1 , it is a schematic diagram of a form. Figure 1 In the form, the form includes a plurality of table frames, and each table frame can fill in text. As shown in Figure 1 , the form includes a label 101 and a fill-in control 102, the label 101 includes a label box 1011 and a label name 1012, and the fill-in control includes a fill-in box 1021 and content text 1022.

[0086] As shown in Figure 1 , the form can include a plurality of label boxes 1011, and different label boxes 1011 include different label names 1012, such as name, mobile phone number, ID number, email, contact address, etc. Each label box 1011 has a matching fill-in box 1021, and the fill-in box 1021 is used to fill in the content text 1022 matched with the label name. The content text 1022 in different fill-in boxes 1021 is different.

[0087] According to the matching relationship between the label 101 and the fill-in control 102, Figure 1The table in the form can also be divided into multiple table frame groups 103, and the table frame group 103 includes a label 101 and a fill-in control 102, and the label 101 and the fill-in control 102 are located in adjacent columns of the same row of the table. In other cases, the label 101 and the fill-in control 102 of a table frame group 103 can also be located in adjacent rows in the table.

[0088] As shown in FIG. 1A, the table frame group 103 includes a label 101 and a fill-in control 102, and the label 101 and the fill-in control 102 are located in adjacent columns of the same row of the table. Figure 1 As shown in FIG. 1B, the fill-in box 1021A contains the content text "177****6666", the fill-in box 1021B contains the prompt text "Please enter the ID number here", the fill-in box 1021C does not contain the content text, and the fill-in box 1021D contains the prompt content "Please enter the contact address here".

[0089] Figure 2 As shown in FIG. 1C, the table frame group 103 includes a label 101 and a fill-in control 102, and the label 101 and the fill-in control 102 are located in adjacent rows of the table. Figure 2 As shown in FIG. 1D, the table frame group 103 includes a label 101 and a fill-in control 102, and the label 101 and the fill-in control 102 are located in adjacent rows of the table. Figure 2 As shown in FIG. 1E, the table frame group 103 includes a label 101 and a fill-in control 102, and the label 101 and the fill-in control 102 are located in adjacent rows of the table.

[0090] The fill-in interface provided by the electronic device includes a table including multiple labels and fill-in boxes, and the label name in the label and the label box is usually already existing and fixed, and the content text in the multiple fill-in boxes needs to be manually input by the user. The fill-in interface of the electronic device in multiple scenarios all needs the user to manually fill in the text in the table, and the operation is relatively cumbersome. In addition, the multiple fill-in interfaces provided by the electronic device may involve the same content text, and the electronic device needs the user to repeatedly input multiple times, which further increases the degree of user operation complexity, and the user fill-in experience is poor.

[0091] Based on this, the embodiments of the present application provide an information processing method. The electronic device triggers automatic form filling in response to the touch operation of the user. The electronic device automatically identifies the fill-in box to be filled in the table, finds the label matched with the fill-in box, and automatically matches the content text for the fill-in box according to the label, so as to realize automatic form filling. In this way, the interactive operation of the user in the fill-in interface can be greatly reduced, and the user experience is improved.

[0092] Exemplarily, the electronic device can be a mobile phone, a tablet computer, a personal computer (PC), a smart screen, a desktop computer, a laptop computer, a handheld computer, a notebook computer, an ultra-mobile personal computer (UMPC), a netbook, a smart watch and the like wearable device, an artificial intelligence (AI) sound box, and a vehicle-mounted device, can also be various teaching aids (such as a learning machine, an early education machine), a smart toy, a portable robot, a personal digital assistant (PDA), an augmented reality (AR) \ virtual reality (VR) device, a media player and the like device, and can also be a device with mobile office function, a device with smart home function, a device with audio-visual entertainment function, a device supporting smart travel, and the like. The specific form of the electronic device is not specially limited in the embodiments of the present application. The structure of the electronic device will be described in detail in the following embodiments.

[0093] The implementation scheme of triggering automatic form filling and the implementation scheme of various automatic form filling scenarios involved in the information processing method provided in the embodiment will be described in turn with reference to several embodiments respectively.

[0094] In embodiment 1, the electronic device triggers automatic form filling in response to a first operation.

[0095] The electronic device triggers the execution of the information processing method provided in the embodiment to automatically fill in the form filling interface in response to the touch operation of the user.

[0096] In an example, the electronic device triggers automatic form filling in response to the touch operation of the user in the form filling interface.

[0097] As shown in Figure 2 , it is an interface and operation schematic diagram of triggering automatic form filling involved in the information processing method provided in the embodiments of the present application. As shown in Figure 2 (1), the electronic device displays a form filling interface 2101, which can be a received information writing interface. The form filling interface 2101 displays a form including a label box and a writing control. The label name included in the label box includes name, phone, area, and detailed address, etc. The writing control includes or is used to fill in the content text corresponding to the label name in the label box, such as the content text corresponding to the name, the content text corresponding to the phone, the content text corresponding to the area, and the content text corresponding to the detailed address, etc.

[0098] As shown in Figure 2In the form-filling interface shown in (1), the form includes multiple blank form frames for filling controls, meaning that the corresponding text content has not yet been filled in the filling controls. The electronic device can respond to the user's touch operation and trigger the automatic form-filling process, automatically filling in the corresponding text content in the multiple filling controls within the form-filling interface.

[0099] Depending on the different input methods, fill-in controls can be divided into input forms and drop-down forms. For example, when the fill-in control is an input form, the electronic device can respond to the user's input and add corresponding text content to the input form. Input forms are suitable for situations where there are many types of text content or where it is impossible to recommend options. Figure 2 As shown in (1), the input controls for name, telephone number and detailed address are all input form boxes.

[0100] For example, user input can be text or voice input on the form interface. Another example is when the fill-in control is a dropdown list; the electronic device can respond to the user's selection by adding corresponding text to the dropdown list. Dropdown lists are suitable for situations where the types of text content are relatively few or where suggested selections are possible. Figure 2 As shown in (1), the input control for the region is a drop-down list.

[0101] Electronic devices can respond to user touch operations and trigger automatic form filling, automatically matching or filling in the corresponding text content in the input boxes and / or drop-down boxes in the form filling interface displayed on the electronic device.

[0102] In one example, the electronic device displays a first control on the form-filling interface, which the user triggers to autofill the form. The electronic device can respond to a user's touch operation on the first control to trigger autofill on the form-filling interface.

[0103] like Figure 2 As shown in (1), the electronic device displays a first control 202 on the form filling interface 201. The first control 202 may include a gesture prompt bar, a table-shaped control, an artificial intelligence (AI)-shaped control, an input pen-shaped control, etc., without limitation.

[0104] like Figure 2 As shown in (1), the electronic device can receive touch operations such as single click, double click or long press by the user on the first control 202, triggering the automatic form filling process.

[0105] In one scenario, the electronic device responds to a user's touch operation on the first control 202, directly triggering the autofill process and initiating the autofill operation on the form filling interface.

[0106] In another scenario, the electronic device, responding to a user's touch operation on the first control 202, can also display autofill options corresponding to different form-filling modes. For example... Figure 2 As shown in (2), the electronic device displays an auto-fill option, including a first mode control 203 corresponding to the first form filling mode and a second mode control 204 corresponding to the second form filling mode.

[0107] The first form-filling mode corresponds to a select-and-fill mode. The electronic device automatically fills in the text for the currently selected fill-in control. The electronic device then automatically fills in the text for the next selected fill-in control.

[0108] It should be noted that the "select-and-fill" method mentioned here refers only to a scheme where the electronic device automatically fills in the corresponding text for the currently selected fill control. In other cases, "select-and-fill" may also be called other names, such as form filling, sequential form filling, single-box form filling, etc., without limitation.

[0109] The second form-filling mode corresponds to the one-click form-filling mode. After the electronic device triggers auto-fill, it automatically fills in the text corresponding to multiple fill-in controls based on all the fill-in controls on the form-filling interface.

[0110] It should be noted that the term "one-click form filling" here refers only to a solution where an electronic device fills out multiple fields on a form interface at once. In other cases, "one-click form filling" may also be called by other names, such as quick form filling or one-click full form filling, without limitation.

[0111] like Figure 2 As shown in (2), the electronic device displays the first mode control 203 and the second mode control 204, which can be displayed as text on the first interface 201. For example, the electronic device displays "Fill in form" to mark the first mode control 203 and "One-click fill" to mark the second mode control 204. In other cases, the electronic device can also mark the first mode control and the second mode control as graphical controls, without limitation.

[0112] like Figure 3 As shown in (2), the electronic device receives a click operation from the user on the first mode control 203, triggering simultaneous selection and filling, such as... Figure 4 As shown in (3) above, the specific implementation scheme for selecting and filling in simultaneously can be found in the relevant description of Embodiment 2 below. Or as shown in... Figure 4 As shown in (4) and (5) in the figure, the electronic device can receive the user's click operation on the second mode control 204 to trigger one-click form filling. The specific implementation scheme of one-click form filling can be found in the relevant description of the following embodiment 3.

[0113] The information processing method provided in this example is that the electronic device displays a first control in a form filling interface. The electronic device can display option controls corresponding to different form filling modes in response to a touch operation of the user on the first control, for example, a first mode control corresponding to side selection and side filling and a second mode control corresponding to one-key form filling. The electronic device can trigger automatic form filling operation of the corresponding mode in response to a click operation of the user on the first mode control or the second mode control. The electronic device provides a more concise way to trigger form filling operation, saves user operation, and improves user experience.

[0114] In another example, the electronic device triggers automatic form filling after displaying the form filling interface in response to a touch operation of the user on the smart application.

[0115] For example, the electronic device can automatically start the automatic form filling process in response to a touch operation of the user on the smart application. The electronic device can display an automatic form filling option in the form filling interface when it is detected that the currently displayed application interface is a form filling interface.

[0116] For another example, the electronic device can also add an automatic form filling on-off control in the setting interface of the smart application. The electronic device can automatically start the automatic form filling process in response to a click operation of the user on the automatic form filling on-off control. The electronic device can display an automatic form filling option in the form filling interface when it is detected that the currently displayed application interface is a form filling interface.

[0117] In other examples, the electronic device can also trigger automatic form filling in response to a voice operation of the user.

[0118] For example, the electronic device can start the automatic form filling process when it is detected that the currently displayed application interface is a form filling interface, provided that the electronic device receives voice information of the user including a keyword indicating automatic form filling. The keyword indicating automatic form filling can include at least one of form filling, side selection and side filling, one-key form filling, automatic form filling, and AI form filling, without limitation.

[0119] The electronic device can also trigger automatic form filling through other ways. Other ways of triggering automatic form filling can be applied to the information processing method provided in the embodiments of the present application, and are not described herein.

[0120] In embodiment 2, the automatic form filling mode of the electronic device is side selection and side filling.

[0121] The electronic device triggers side selection and side filling in the form filling interface in response to a user operation. The electronic device identifies a currently selected fill-in box of the user, determines a label matched with the fill-in box, and fills the fill-in box with a matched content text according to a label name of the label.

[0122] It should be noted that the information processing method provided in the embodiments of the present application, the electronic device fills in the matched content text in the fill-in box, which can be directly displaying the obtained content text in the fill-in box, or displaying the obtained content text in the fill-in box or the adjacent area for the user to select. In the case that the content text already exists in the displayed fill-in box, the electronic device can also add, modify or delete the filled content text in the fill-in box in response to the user operation.

[0123] As shown in Figure 4 , it is a flow diagram of the edge selection and filling process involved in the information processing method provided by the embodiments of the present application. The electronic device performs edge selection and filling, mainly including the following steps:

[0124] S301: The electronic device displays a first interface, the first interface is a form filling interface, and the first interface includes a first fill-in box and a plurality of label boxes, each label box has a corresponding label name, and different label boxes have different label names.

[0125] The information processing method provided in the embodiments of the present application, the electronic device displays a first interface, the first interface can be the form filling interface in the foregoing example, and the first interface will be used to represent the form filling interface hereinafter. The electronic device triggers automatic form filling in response to the touch operation of the user on the first interface. The touch operation to which the electronic device responds can include click operation, single-click operation, double-click operation, sliding operation, etc., without limitation.

[0126] As shown in Figure 4 , the electronic device displays a first interface 401, and the first interface 401 includes a table. The table includes a plurality of labels 402, such as 402A, 402B, 402C, 402D, etc. Among them, the label names of different labels are different. For example, the label name of 402A is name, the label name of 402B is mobile phone number, the label name of 402C is region, and the label name of 402D is detailed address.

[0127] In addition, the table also includes a plurality of fill-in boxes 403, such as 403A, 403B, 403C, 403D, etc. Among the plurality of fill-in boxes displayed by the first interface, there can be some fill-in boxes that have filled content text, for example, the content text filled in 403A is Zhang San. Among the plurality of fill-in boxes displayed by the first interface, there can also be some fill-in boxes that have not filled content text, for example, 403B, 403C, 403D have not filled corresponding content text.

[0128] For ease of description, the fill-in box with filled content text in the table can be referred to as a filled table box, and the fill-in box without filled content text can be referred to as an unfilled table box.

[0129] The electronic device determines the current to-be-filled-in fill-in box in response to a touch operation of a user acting on an unfilled-in table box. For ease of description, the fill-in box to be filled in indicated by the touch operation of the current user is referred to as a first fill-in box. As shown in (1) of Figure 3 , the electronic device receives a click operation of a user acting on the fill-in box 403C and determines 403C as the first fill-in box to be filled in.

[0130] The electronic device receives a touch operation of a user acting on the first fill-in box and can also move the cursor into the first fill-in box. The electronic device can determine the first fill-in box to be filled in according to the position of the current cursor by detecting the position of the current cursor.

[0131] As shown in (1) of Figure 4 , the electronic device receives a touch operation of a user acting on the fill-in box 403C and displays the cursor 404 in the fill-in box 403C, as shown in (2) of Figure 4 .

[0132] In an example, the electronic device obtains a first screenshot of the first interface, obtains the current cursor position by analyzing the first screenshot, and further determines the first fill-in box in the first interface.

[0133] In implementation, as shown in Figure 5 , the electronic device performs S302 and determines the first label box matched with the first fill-in box before S301, which can further include the following steps:

[0134] S3011: The electronic device obtains a first screenshot of the first interface.

[0135] S3012: The electronic device performs feature extraction on the first screenshot and identifies a first cursor.

[0136] As shown in (2) of Figure 5 , the electronic device takes a screenshot of the first interface and obtains a corresponding first screenshot, as shown in (3) of Figure 5 .

[0137] The electronic device performs feature extraction on the first screenshot and extracts possible features of the corresponding cursor, including color features, shape features, texture features, etc. The electronic device extracts possible features of the corresponding cursor can identify the cursor in the first screenshot as a first cursor by using a pattern recognition or machine learning algorithm.

[0138] S3013: The electronic device determines the first fill-in box according to the pixel coordinates of the first cursor in the first screenshot.

[0139] The electronic device can identify the first cursor in the first screenshot and obtain the pixel coordinates of the first cursor in the first screenshot, thereby determining the first input box. For example, the electronic device can determine the input box whose coordinate range includes the coordinate range of the first cursor as the first input box, or the electronic device can determine the input box whose starting coordinates of the table frame and the coordinates of the first cursor are within a preset range as the first input box.

[0140] In another example, the electronic device can also obtain the cursor position by extracting the parameters associated with the cursor in the first interface, and then determine the first fill box to be filled in in the first interface based on the cursor position.

[0141] For example, for electronic devices with input and textarea elements, the selectionStart and selectionEnd properties can be used to get the cursor position. selectionStart returns the starting position of the cursor within the element, while selectionEnd returns the ending position of the cursor within the element.

[0142] In practice, electronic devices can also obtain the first cursor in the first interface through other means, and then determine the first fill box to be filled based on the first cursor.

[0143] S302: The electronic device responds to a first operation applied to a first input box in a first interface, and determines a first label box that matches the first input box from a plurality of label boxes; the first label box has a first label name.

[0144] The information processing method provided in this embodiment allows an electronic device to identify multiple table frames within a first interface, including multiple label boxes and multiple fill boxes, with the fill boxes including a first fill box to be filled. The table on the first interface comprises multiple table frame groups, each group including a matching label box and a fill box. The positions of the fill box and label box in each table frame group are typically relatively fixed. For example, the label box and fill box in each table frame group are located in adjacent columns of the same row, or they are typically located in adjacent rows of the same column in the table.

[0145] like Figure 5 The diagram shown illustrates the relative positions of table frames in the information processing method provided in this embodiment. Figure 4 As shown in (1), the table includes two table frame groups, wherein the first table frame group 501 includes a label box 5011 and a fill box 5012, and the second table frame group 502 includes a label box 5021 and a fill box 5022.

[0146] The label box and fill box in each table frame group are located in adjacent columns of the same row. For example, label box 5011 and fill box 5012 are in the same row, with label box 5011 located in the column to the left of fill box 5012. Label box 5021 and fill box 5022 are in the same row, with label box 5021 located in the column to the left of fill box 5022.

[0147] like Figure 4 As shown in (2), the table includes two table frame groups, namely table frame group A and table frame group B. Among them, the label box 5031 of table frame group A 503 is located in the column above the fill box 5032, and the label box 5041 of table frame group B 504 is located in the column above the fill box 5042.

[0148] Figure 4 The two relative positions of the form frames shown can include the relative positions of most matching label boxes and fill boxes within a form. Based on this, after determining the first fill box to be filled, the electronic device can determine the label box that matches the first fill box from among multiple label boxes, according to the position of the first fill box and the relative positions of multiple label boxes. For ease of description, the electronic device can refer to the label box that matches the first fill box among multiple label boxes as the first label box.

[0149] S303: The electronic device displays first content text in the associated area of ​​the first input box; the tag attribute of the first content text matches the first tag name.

[0150] The electronic device obtains the first tag that matches the first input field, extracts the tag name from the first tag's first tag box, and records it as the first tag name. The electronic device matches the first content text for the first input field based on the first tag name, and the tag attributes of the first content text match the first tag name.

[0151] like Figure 3 As shown in (3), the first input box determined by the electronic device is 403C, the first tag box matched by 403C is 402C, and the first tag name in 402C is "region". Based on the first tag name "region", the electronic device matches the first content text with the tag attribute "region" for the first input box, such as "Province A, City B, District C".

[0152] In other cases, if the electronic device determines that the first label name entered in the first label box is "gender", the electronic device will use the first content text with the label attribute of the first input box being "gender", such as "female" or "male".

[0153] In one specific implementation, the electronic device matches content text to the first fill box, and the number of content texts can be one or more.

[0154] In one scenario, if the electronic device obtains only one matched text, it can directly fill in the first text field as the first text field.

[0155] In another scenario, the electronic device matches at least one piece of content text to the first input field, and the electronic device can display at least one piece of content text in the associated area of ​​the first input field. The electronic device can fill in the first information table with the content text that received the user's touch operation from the at least one piece of content text as the first piece of content text.

[0156] like Figure 4 As shown in (4), the electronic device matches the content text "Province A, City B, District C" to the first fill box 403C, and displays the content text 405 in the adjacent column of 403C. If the electronic device receives a touch operation applied to the content text 405, the electronic device will fill "Province A, City B, District C" as the first content text into the first fill box 403C, as shown in (4). Figure 4 As shown in (5) of the table.

[0157] Based on the aforementioned implementation, after matching the corresponding first content text to the first input box, the electronic device can further detect when the user triggers other input boxes to be filled, such as a second input box, and match the corresponding content text, such as second content text, to the second input box. Figure 4 As shown, the electronic device can also perform the following steps:

[0158] S304: The electronic device responds to a second operation applied to the second fill box in the first interface, and determines a second label box that matches the second fill box from multiple label boxes; the first label box is a different table frame, and the second label box has a second label name.

[0159] like Figure 6 As shown in (5), after the electronic device matches the first content text to the first fill box 403C, the electronic device can also receive the user's click operation on 403D to determine that 403D is the second fill box to be filled.

[0160] As shown in (5) of Figure (4), the electronic device determines the matching second label box 402D for the second fill box 403D.

[0161] S305: The electronic device displays second content text in the associated area of ​​the second fill box; the label attribute of the second content text matches the second label name.

[0162] like Figure 6As shown in (5) of FIG. 4B, the electronic device extracts the second label name filled in the second label box 402D as "detailed address", and the electronic device determines the matched second content text for the second fill-in box 403D according to the "detailed address", and the label attribute of the second content text is "detailed address". As shown in (6) of FIG. 4B, the electronic device matches the content text 406 of the second fill-in box as "D Street E Community F Unit". If the electronic device receives a touch operation acting on the content text 406, the electronic device fills the content text 406 as the second content text into the second fill-in box 403D. Figure 7

[0163] It should be noted that the electronic device displays the fill-in box can be in the form of a control, and the control can respond to the touch operation of the user. After the electronic device fills the second fill-in box, the electronic device can also receive the touch operation of the user acting on the second fill-in box, and add, modify or delete the second content text filled in the second fill-in box.

[0164] The electronic device performs S301 and S302 to determine the first fill-in box to be filled and the first label box matched with the first fill-in box, and the electronic device performs S304 and S305 to determine the second fill-in box to be filled and the second label box matched with the second fill-in box, which involves the identification and matching operation of the electronic device on the table box and the text in the table. The specific implementation scheme will be explained in detail through several embodiments.

[0165] As shown in (1) of FIG. 6A, the electronic device displays the first fill-in box 601 to be filled in the first interface. The electronic device includes an information extraction module and a content matching module, wherein the information extraction module extracts the information of the first interface, such as scene information, filled information and to-be-filled information, and the content matching module is used to match the corresponding content text for the to-be-filled information. Figure 8

[0166] For example, as shown in (2) of FIG. 6A, the information extraction module of the electronic device extracts the title, title and other information of the first interface, and obtains the scene information of the table such as "XX address". The electronic device can also extract the filled information such as "Zhang San", and the label attribute corresponding to the filled information "Zhang San" is "name". The electronic device extracts the to-be-filled information, and the label attribute of the to-be-filled information is "phone". Figure 8 Based on this, the content matching module of the electronic device matches the content text with the label attribute "phone" for the to-be-filled information, for example, "177****6666". The electronic device displays the determined content text 602 in the associated area of the first fill-in box 601 to be filled.

[0167]

[0168] ​​​It should be noted that the content text determined by the electronic device for the first fill-in box to be filled in can be determined directly based on the label attribute of the content text, for example, according to the label attribute "phone", it is determined that the matching content text is "177****6666". The electronic device can also determine the matching content text based on the label attribute of the element and the filled information. For example, the electronic device determines that the content text "177****6666" matches both "Zhang San" and "phone" according to the label attribute "phone" and the filled information "Zhang San".

[0169] In a specific embodiment, the electronic device can identify the multiple table frames and filled elements in the first interface through a pre-trained neural network model, determine the first fill-in box to be filled in through the cursor position, and find the first label box matching the first fill-in box from multiple label boxes through a pre-configured first algorithm.

[0170] As shown in Figure 8 , the automatic form filling process involved in the information processing method provided by the embodiments of the present application is shown. The automatic form filling process performed by the electronic device mainly includes the following steps:

[0171] S701: The electronic device obtains a first screenshot of a first interface.

[0172] As shown in (1) of Figure 9 , the electronic device obtains a screenshot of the first interface, denoted as a first screenshot, and the first screenshot can be a screenshot of the entire interface of the first interface.

[0173] In one case, the electronic device displays the first interface, displays a table in a part of the interface region on the first interface, or newly adds a pop-up window to display the table on the first interface. The electronic device can also only take a screenshot of the part of the interface region or the pop-up window displaying the table as the first screenshot.

[0174] S702: The electronic device obtains a cursor position.

[0175] The electronic device can recognize the cursor feature in the first screenshot by performing image feature extraction on the first screenshot, and obtain the cursor position.

[0176] S703: The electronic device crops the first screenshot to obtain a second screenshot according to the cursor position.

[0177] In the automatic form filling process of the edge-selecting-edge-filling scenario, the electronic device needs to analyze and match the fill-in box to be filled in currently, and the filled fill-in box will be referred to in the analysis and matching process. In the first interface, the filled fill-in box is usually located above the unfilled fill-in box. The electronic device can only retain the table above the cursor position, which can obtain the relevant information required for automatic form filling and reduce unnecessary calculation amount.

[0178] like Figure 9 As shown in (1), the electronic device acquires the first screenshot. After acquiring the cursor position, the electronic device retains the image features located in the row and above the cursor in the first screenshot, and crops the first screenshot to obtain the second screenshot, as shown in (1). Figure 8 As shown in (2) of the text.

[0179] S704: The electronic device performs Type I detection on the second screenshot to identify the text bounding box and text information in the second screenshot.

[0180] S705: The electronic device performs a second type of detection on the second screenshot to identify the table frame and table frame type in the second screenshot.

[0181] The electronic device performs a first type of detection and a second type of detection on the second screenshot. The first type of detection is used to detect text information in the table and determine the text bounding box (or text box) corresponding to each segment of text information. The second type of detection is used to detect table frames in the table. The order in which the electronic device performs the first and second types of detection is not limited; for example, the electronic device can perform the first and second types of detection simultaneously.

[0182] The electronic device executes S704 to perform a first type of detection on the second screenshot. This can be done by using Optical Character Recognition (OCR) technology to perform optical character recognition on the second image and recognize the text information in the second screenshot.

[0183] OCR technology scans, processes, and analyzes text information in images, converting the analyzed text into a computer-readable format. The basic steps of OCR technology include image preprocessing, feature extraction, character recognition, and post-processing. Preprocessing typically involves operations such as noise reduction, binarization, and image enhancement to improve image quality and clarity. Feature extraction extracts text features, such as shape, size, and tilt, through image analysis. Character recognition uses these features to identify and classify characters, while post-processing includes operations such as correcting text position and fixing typos.

[0184] The electronic device performs Type I detection on the second screenshot and can also identify the text box corresponding to each piece of text information. For example... Figure 9 As shown, the table includes table frames 901 and 902, and text information 903 and 904. The electronic device, through a first type of detection, can identify text information 903 and 904, as well as the text bounding box 905 corresponding to text information 903 and the text bounding box 906 corresponding to text information 904.

[0185] The electronic device performs S705, and performs second type detection on the second screenshot. The second type detection can be implemented by using a first model. The first model can be a visual detection model or a model with table frame recognition capability, such as a YOLO model. The first model has the capability of identifying table frames from an input image and obtaining the coordinates of each table frame in the input image and the type of the table frame.

[0186] The electronic device inputs the second screenshot into the first model to obtain the coordinates of each table frame in the second screenshot. For example Figure 9 As shown in the second screenshot, the electronic device inputs the second screenshot into the first model, identifies table frame 901 and table frame 902, and obtains the coordinates of the four vertices (A, B, C, and D) of table frame 901 in the second screenshot and the coordinates of the four vertices (a, b, c, and d) of table frame 902 in the second screenshot.

[0187] In addition, the electronic device inputs the second screenshot into the first model, identifies multiple table frames existing in the second screenshot, and determines the type of each table frame according to the presence state or input mode of the elements in each table frame. The table frame types obtained by the electronic device using the first model can include a label box and a fill-in box, and the fill-in box can be further divided into an unfilled input box, an unfilled drop-down option box, a filled content text input box, and a filled drop-down option box.

[0188] The label box refers to a table frame filled with a label name, and the label box can also be defined as a prompt box. The unfilled input box is a fill-in box that has not been filled with content text and has an input mode. The unfilled drop-down option box is a fill-in box that has not been filled with content text and has a drop-down option mode. The filled input box is a fill-in box that has been filled with content text and has an input mode. The filled drop-down option box is a fill-in box that has been filled with content text and has an input mode. In some cases, a fill-in box can support both input and drop-down option modes. In this case, the fill-in box can be identified as an input box.

[0189] As shown in the second screenshot in (2) of Figure 5 The electronic device identifies multiple table frames in the second screenshot, where table frames 402A, 402B, 402C, and 402D are label boxes, 403A is a filled input box, 403B is a filled input box, 403C is an unfilled drop-down option box, and 403D is an unfilled input box.

[0190] S706: The electronic device matches the text bounding box with the table frame and matches the text information with the table frame.

[0191] As shown in the second screenshot in (2) of Figure 10As shown, the electronic device identifies the text bounding box 905 and the text bounding box 906 in the second screenshot through the first type of detection, and identifies the table box 901 and the table box 902 in the second screenshot through the second type of detection. The electronic device needs to further determine that the table box 901 matches the text bounding box 905 and the table box 902 matches the text bounding box 906.

[0192] In an example, the electronic device can employ a center point matching algorithm to determine the text bounding box matched by each table box.

[0193] Specifically, the electronic device obtains the text box center point of each text bounding box, and finds a table box in the plurality of table boxes, the table box coordinate range of which includes the text box center point coordinate, as the table box matched with the text bounding box. In a specific implementation, the electronic device can determine the table box coordinate range by the coordinates of two opposite vertices of each table box, and determine whether the table box coordinate range includes the text box center point coordinate.

[0194] For example, the electronic device determines the table box coordinate range according to the coordinates of the top-left vertex and the bottom-left vertex of the table box. As shown in the table box 901, the table box coordinate range of the table box 901 is determined by the coordinates of the top-left vertex A and the bottom-right vertex D. Figure 11 As shown, the electronic device obtains the text box center point O1 (X O1 , Y O1 ) of the text bounding box 905, the coordinates of the four vertices ABCD of the table box 901, and the coordinates of the four vertices abcd of the table box 902.

[0195] In a case, the electronic device obtains the coordinates (X A , Y A ) of the top-left vertex A and the coordinates (X D , Y D ) of the bottom-right vertex D of the table box 901, and determines whether the coordinates of the text box center point O1 and the table box coordinate range of the table box 901 satisfy one of the following conditions:

[0196] (X A ≤X O1 )&(X D ≥X O1 )&(Y A ≥Y O1 )&(Y D ≤Y O1 ), or (X A ≤X O1 ≤X D )&(Y A ≥Y O1 ≥Y D ).

[0197] When the coordinates of the center point O1 of the text box and the coordinate range of the table frame 901 meet the above conditions, the electronic device determines that the text outer box 905 matches the table frame 901, that is, the text information in the table frame 901 is "name".

[0198] In another scenario, the electronic device obtains the coordinates (X) of the top-left vertex a of frame 902. a Y a ) and the coordinates (X) of the lower right vertex D d Y d Determine whether the coordinates of the center point O1 of the text box and the coordinate range of the table frame 902 satisfy one of the following conditions:

[0199] (X a ≤X O1 )&(X d ≥X O1 )&(Y a ≥Y O1 )&(Y d ≤Y O1 ), or (X a ≤X O1 ≤X d )&(Y a ≥Y O1 ≥Y d ).

[0200] If the coordinates of the center point O1 of the text box and the coordinate range of the table frame 902 do not meet the above conditions, the electronic device determines that the text outer box 905 and the table frame 902 do not match, that is, the text information in the table frame 901 is not "name".

[0201] In another scenario, the electronic device can obtain the coordinates (X) of the top-left vertex a of frame 902. a The coordinates (X,Ya) and the coordinates of the lower right vertex D. d (Yd), determine whether the coordinates of the center point O2 of the text box of the text outer box 906 and the coordinate range of the table frame of the table frame 902 satisfy one of the following conditions:

[0202] (X a ≤X O2 )&(X d ≥X O2 )&(Y a ≥Y O2 )&(Y d ≤Y O2 ), or (X a ≤X O2 ≤X d )&(Y a ≥Y O2 ≥Y d).

[0203] The electronic device determines that the text bounding box 906 and the table box 902 match, i.e., the text information in the table box 901 is "Please enter the name", in a case where the coordinates of the text box center point O2 and the table box coordinate range of the table box 902 satisfy the above conditions.

[0204] It should be noted that the above examples take the top-left vertex and the bottom-right vertex of the text bounding box as the calculation factors of the table box coordinate range. In other cases, the bottom-left vertex and the top-right vertex of the text bounding box can also be taken as the calculation values of the table box, or all four vertices of each text bounding box can be taken as the calculation factors of the table box coordinate range, without limitation.

[0205] In another specific embodiment, the electronic device can use a table label and fill-in box pairing algorithm based on two-dimensional Gaussian distribution modeling.

[0206] As shown in the relative position relationship of the two table box groups Figure 11 , the electronic device can obtain a large number of table samples, and according to the relative position relationship between the label box and the fill-in box in the plurality of table box groups in the table samples, use two-dimensional Gaussian distribution modeling to statistically distribute the relative distribution between the center point coordinates of the fill-in box and the center point coordinates of the label box, and calculate the probability of matching each label box in the plurality of label boxes with the fill-in box based on the relative distribution. As shown in Figure 12 , the relative position diagram of the fill-in box and the matching label box, the X-axis coordinate difference of the label box and the fill-in box is negative, the Y-axis coordinate difference is 0 or positive, and the greater the probability of matching the label box and the fill-in box; otherwise, the X-axis coordinate of the label box and the fill-in box is positive, or the Y-axis coordinate is negative, and the smaller the probability of matching the label box and the fill-in box.

[0207] In this way, the electronic device can achieve high precision and low latency for matching the fill-in box with the label box.

[0208] The electronic device based on Gaussian distribution modeling, the scheme for matching the fill-in box with the table box mainly includes the following steps:

[0209] Step 1, the electronic device pre-processes the second screenshot.

[0210] The electronic device identifies the label box and the fill-in box in the second screenshot using the first model, and records the center point coordinates of each label box and each fill-in box.

[0211] Step 2, the electronic device constructs a two-dimensional Gaussian distribution statistical model

[0212] The electronic device statistically distributes the relative position of the center point coordinates of the label box and the center point coordinates of the fill-in box from a large number of table samples, and defines the relative position vector dij = C Ij - C Ti . Wherein C Ti is the center point coordinate of the label box, C Ti = (x Ti , y Ti ), C Ij represents the center point coordinate of the fill-in box, C Ij = (x Ij , y Ij ). Based on this, the relative position vector d ij = (x Ij - x Ti , y Ij - y Ti ), the value corresponding to the relative position vector is the distance between the center of the label box and the center of the fill-in box, that is, the distance between the center point of the matched label box and the center point of the matched fill-in box.

[0213] The electronic device constructs a two-dimensional Gaussian distribution model p(d ij ) based on the distribution results of the relative position vectors of a large number of table samples,

[0214]

[0215] Wherein μ represents the mean, ε represents the covariance matrix, exp represents the natural exponential function, and T represents the transpose matrix.

[0216] Step 3, the electronic device constructs a first candidate set, and the first candidate set includes candidate label boxes whose centers are within a first threshold value from the center of the first fill-in box.

[0217] The electronic device calculates the center distance from other candidate boxes for each fill-in box I j , finds the candidate label boxes T i whose center distance is within a predefined first threshold value r, forms a candidate set T j , T j = T i ||d ij |≤r. For example, the first threshold value r can be less than or equal to 100 pixels.

[0218] Step 4, the electronic device determines the probability of matching each candidate label box to the fill-in box.

[0219] The electronic device calculates the probability I i of matching each candidate label box T j in the candidate set to the fill-in box, and the calculation formula is:

[0220] The electronic device obtains a probability of each candidate label box in the candidate set relative to the fill-in box, and selects a label box T with the maximum probability i * as the fill-in box I j , the paired label box is

[0221] Based on this, the electronic device can determine the matched first label box for the first fill-in box indicated by the user. The electronic device analyzes the relative position distribution of the matched fill-in box and label box in a large number of table samples, constructs a Gaussian model, and finds the first label box matched with the first fill-in box according to the Gaussian model. The electronic device finds the matched first label box for the first fill-in box by constructing the Gaussian model, which has less calculation and higher accuracy.

[0222] S77: The electronic device obtains a first label name matched with the first fill-in box corresponding to the cursor position.

[0223] S78: The electronic device outputs first content text matched with the first label name.

[0224] The electronic device matches content text for the first fill-in box according to the first label name of the first label box.

[0225] In specific implementation, the electronic device can maintain a table information library, which stores content text and corresponding label names for filling in the table. The electronic device determines the first label name corresponding to the first fill-in box, finds the content text matched with the first label name from the table information library, and takes the content text as the first content text matched with the first fill-in box. The electronic device fills the first content text into the first fill-in box of the first interface. In this way, the user's manual input operation for filling in the table can be saved, and the user's filling experience can be optimized.

[0226] On the basis of the above-mentioned embodiments, the electronic device can also detect whether the current scene is a table filling scene by recognizing the cursor trajectory before executing the automatic table filling process, and execute the subsequent automatic table filling process if it is determined that the current scene is a table filling scene.

[0227] As Figure 2 shown, a table box coordinate schematic diagram involved in the information processing method provided by the embodiments of the present application. The table includes fill-in boxes 1101, 1102 and 1103, the vertical coordinate of the fill-in box 1101 is y1, the vertical coordinate of the fill-in box 1102 is y2, and the vertical coordinate of the fill-in box 1103 is y3. The electronic device can detect the region of the current input operation of the user based on the cursor coordinates.

[0228] In one case, the user touch operation received by the electronic device is a horizontal input operation or a line feed operation in a fill-in box, and the y value of the cursor changes little.

[0229] In another case, the user touch operation received by the electronic device is switching from one fill-in box to another fill-in box, and the y value of the cursor changes greatly. As shown in Figure 2 , the electronic device switches from fill-in box 1101 to fill-in box 1102, and the y value changes by Δy1, Δy1=y1-y2. The electronic device switches from fill-in box 1102 to fill-in box 1103, and the y value changes by Δy2, Δy2=y2-y3. Based on this, the electronic device can set a second threshold, and the second threshold ∈ y When the y value change exceeds the second threshold, the electronic device confirms that the form is filled in by line switching.

[0230] In an example, the electronic device provides a low-power monitoring strategy to monitor whether it is in a form filling scenario.

[0231] For example, the electronic device monitors the change of the cursor coordinates and calculates the Y value change of the cursor coordinates within an initial monitoring time T init When |Δy i |=|y i+1 -y i |>∈ y , it is confirmed that the form is filled in by line switching, and the electronic device enters a continuous monitoring state.

[0232] The electronic device monitors the cursor coordinates and counts the cursor trajectory features. The electronic device cursor trajectory is segmented, and the cursor state is determined according to the number of segments.

[0233] The electronic device segments the sequence of cursor movement according to the Y value change, and considers each segment as a line of input process. The average value of the X value change in each segment is counted, and the average value of the Y value of each segment is calculated, and then the average value of the Y value change between each segment is calculated.

[0234] The electronic device first segments the sequence of cursor movement.

[0235] The electronic device converts the trajectory of the cursor in the time sequence T into where i represents the time step, and n represents the total number of time steps. After the electronic device determines to start continuous monitoring according to the y value change, it continuously monitors and judges whether the Y value change is greater than the error threshold ∈ y When |Δy i |>∈ y , it is considered that the cursor ends the current segment and starts a new segment. The start and end time of each segment is where k is the segment number.

[0236] When the number of segments K is greater than a specified value N segments , the electronic device calculates the average value of the X value change in each segment and the average value of the Y value change between each segment

[0237] The electronic device calculates the average of the X value changes in each section The formula includes:

[0238] The electronic device calculates the average of the Y value in each section The formula includes:

[0239] The electronic device further calculates the average of the Y value changes between each section Δμ based on the average of the X value changes in each section and the average of the Y value in each section y The formula includes:

[0240] The electronic device obtains the average of the X value changes in each section If the average of the X value changes in all sections It indicates that the cursor moves from left to right in each section, that is, the electronic device responds to the user operation to input the content text in the fill-in box from left to right in turn.

[0241] The electronic device obtains the average of the Y value changes between each section Δμ y Δμ y > ΔY threshold , where ΔY threshold is a preset Y-axis interval threshold, indicating that the Y-axis interval of the input box of each section is large, which is different from inputting long text.

[0242] If the electronic device determines that the average of the X value changes in each section and Δμ y > ΔY threshold , it is determined as a form filling scene. The electronic device can trigger automatic form filling when it is determined that the current scene is a form filling scene by monitoring the cursor trajectory, saving the user's operation of triggering automatic form filling, and further improving the convenience of the user's form filling operation.

[0243] The information processing method provided by the above embodiments of the present application can identify the form filling scene based on the cursor, and the electronic device automatically triggers the automatic form filling. The electronic device can also trigger the automatic form filling based on the user's touch operation. The electronic device starts the automatic form filling process, determines the fill-in box to be filled, matches the label box for the fill-in box, matches the content text for the fill-in box according to the label name of the label box, saves the user's operation of manually inputting the content text, and improves the convenience of the user's form filling operation.

[0244] In embodiment 3, the automatic form filling mode executed by the electronic device is one-key filling.

[0245] ​The present application provides an information processing method, which supports automatic filling of all contents of a form. The method comprises: displaying a first interface; the first interface comprises a form and a first control, and does not comprise a second control for inputting or triggering display of first information; the form comprises a first form label and a to-be-filled control, and the first form label is a form label associated with the to-be-filled control in the first interface. The first information comprises all information to be filled in the first interface. In response to a user operation on the first control, based on the first form label, to-be-filled content matched with the first form label is filled into the to-be-filled control. In this way, the electronic device can automatically fill the form in the first interface by one key in response to one user operation, which can greatly improve the efficiency of filling the form.

[0246] In some embodiments of the present application, during the process of performing automatic filling of the form, the electronic device can display prompt information for prompting that the user is currently filling the form automatically.

[0247] According to different filling manners, the to-be-filled controls in the form can be divided into two different types. One type is a to-be-filled control that needs to be filled by inputting, which is referred to as an input control; and the other type is a to-be-filled control that provides a drop-down menu and needs the user to select appropriate content in the drop-down menu for filling, which is referred to as a drop-down control. In some embodiments, the automatic filling process of the form can implement automatic filling of empty input controls and empty drop-down controls. That is, the above-mentioned empty to-be-filled control specifically refers to an empty to-be-filled control, and can also be an empty drop-down control. The empty to-be-filled control can also be named by other names, such as a to-be-filled control.

[0248] In embodiments of the present application, the automatic filling process of the form can be divided into the following parts: a visual detection part, a to-be-filled content acquisition part, a to-be-filled content maintenance part, and a filling execution part.

[0249] The visual detection part is used to perform visual detection on the first interface after triggering the automatic filling process of the form, to determine whether the first interface comprises a form, and to determine, when the first interface comprises a form, empty to-be-filled controls (including input controls and / or drop-down controls), display positions of the empty to-be-filled controls, a form label associated with the empty to-be-filled controls, and a display position of the form label. The to-be-filled content acquisition part is used to acquire to-be-filled content corresponding to the empty to-be-filled controls when the first interface comprises a form. The to-be-filled content maintenance part is used to maintain and update the acquired to-be-filled content during the process of performing the filling operation. For example, the mobile phone can maintain and update the to-be-filled content according to the filling progress of the current form, such as deleting to-be-filled content that has been filled, or updating coordinate information of an input control corresponding to the to-be-filled content when the form is shifted. The filling execution part is used to perform the filling operation and automatically fill the acquired to-be-filled content into the corresponding empty to-be-filled controls.

[0250] In some embodiments, the table automatic filling process in the embodiments of the present application further includes an exception handling part. The exception handling part is configured to detect whether an exception occurs during the execution of the filling operation, and perform exception handling for the occurred exception. Specifically, the exception handling part is configured to detect whether an exception occurs after the filling operation for one empty filling control is completed.

[0251] The information processing method proposed in the embodiments of the present application is described in detail below with reference to the accompanying drawings. Figure 13 The specific process of the information processing method in some embodiments is shown.

[0252] S1201. In response to a user operation, a table in a first interface is identified.

[0253] The user operation can trigger the table automatic filling process. The first interface includes the table and the first control, and does not include a second control for inputting or triggering the display of the first information. The first information includes all information to be filled in the first interface. In some embodiments, the user operation can correspond to Figure 14 The triggering operation of the one-key filling control by the user. The one-key filling control can be the first control.

[0254] In some embodiments, S1201 can specifically include that the mobile phone acquires a first interface screenshot in response to the user operation. Then, the first interface screenshot is identified to obtain the table therein. The acquisition of the first interface screenshot can be implemented in any manner.

[0255] Optionally, as an example, the mobile phone can implement the identification of the table in the first interface through a model. For example, the mobile phone identifies the first interface screenshot through a first model to obtain the table in the first interface. It should be noted that the specific implementation of obtaining the table through the first model will be described in detail in the following embodiments. It should be noted that the first model can also be named as other names, such as a visual detection model.

[0256] The mobile phone identifies the first interface, and can determine whether the first interface includes a table, and in a case where the first interface includes a table, identify each table label in the table and a fill-in control associated with each table label. Further, in a case where the first interface includes a table, the mobile phone can identify and determine a label area corresponding to each table label and a fill-in area corresponding to the fill-in control associated with each table label, to locate the display positions of the table label and the fill-in control in the first interface, to facilitate automatic filling of the table according to the display positions of the table label and the fill-in control in subsequent processes. Common label areas and fill-in areas are rectangular areas, and the rectangular area can indicate the display position of the rectangular area in the first interface by a bounding box (bbox); for example, a label box and a fill-in box. For example, in a case where the first interface includes a table, the mobile phone can obtain coordinate information of a bounding box corresponding to the label area and coordinate information of a bounding box corresponding to the fill-in area.

[0257] The mobile phone identifies the table in the first interface, and can obtain all table labels in the table in the first interface and fill-in controls associated with each table label. The specific implementation manner in which the mobile phone identifies the table in the first interface and determines the association relationship between the table label and the fill-in control can refer to the description in other embodiments or the description in related technologies. In an actual scenario, the table in the first interface can include some filled fill-in controls (i.e., filled fill-in controls) and some unfilled fill-in controls, i.e., empty fill-in controls. It can be understood that in the embodiments of the present application, the mobile phone only needs to automatically fill in the empty fill-in controls when performing the table automatic filling process. Therefore, after identifying the table in the first interface, the empty fill-in controls in the table and the first table label associated with the empty fill-in controls can be obtained, to facilitate automatic filling of the unfilled fill-in controls; for example, S1202.

[0258] S1202. Obtain an empty fill-in control in the table and a first table label associated with the empty fill-in control.

[0259] The empty fill-in control is a fill-in control to be filled in the table, and the first table label is used to indicate the type of content to be filled in the empty fill-in control associated with the first table label. In the embodiments of the present application, the empty fill-in control is specifically an input control to be filled in the table and / or an empty drop-down control.

[0260] In some embodiments, S1202 can specifically include: obtaining coordinate information of the empty fill-in control and coordinate information of the first table label associated with the empty fill-in control. The coordinate information of the empty fill-in control is specifically the coordinate information of the fill-in area corresponding to the empty fill-in control. The coordinate information of the first table label is specifically the coordinate information of the label area corresponding to the first table label. For example, the information obtained in S1202 can be represented as: fill-in control 1: coordinate information of fill-in area 1 and coordinate information of label area 1 associated with fill-in control 1; fill-in control 2: coordinate information of fill-in area 2 and coordinate information of label area 2 associated with fill-in control 2; … fill-in control n: coordinate information of fill-in area n and coordinate information of label area n associated with fill-in control n. Subsequently, based on the coordinate information of the fill-in area and the coordinate information of the label area associated with the fill-in area, the fill-in operation can be performed on the empty fill-in control.

[0261] S1203. Obtain the content to be filled from the user information library that matches the first table label.

[0262] In order to determine what the content of the empty fill-in control should be, after obtaining the coordinate information of the first table label associated with the empty fill-in control in S1202, it is necessary to identify what the first table label is, and then find the corresponding content to be filled. In some embodiments, S1203 can specifically include: identifying the label text corresponding to the first table label, and obtaining the content to be filled from the user information library that matches the first table label based on the label text.

[0263] In some embodiments, identifying the label text corresponding to the first table label can be achieved by a text recognition technology. For example, the mobile phone can identify the label text corresponding to the first table label by an optical character recognition (OCR) model.

[0264] The user information library stores some content to be filled in a form, which can be the content filled in a form commonly used by the user, such as the recipient address, identity information, and sender address, etc. For example, the user information library stores the following content: table label and label data associated with the table label. The label data is the data that needs to be filled into the empty fill-in control in the form in the form automatic fill-in process. In some embodiments, in the user information library, the table label and the label data are stored in the form of key-value pairs; the table label is the key and the label data is the value. In this embodiment, S1203 includes: obtaining the label data that matches the first table label from the user information library, and taking the matched label data as the content to be filled.

[0265] The specific implementation of obtaining the to-be-filled content matching the first table label from the user information library will be described in detail in subsequent embodiments.

[0266] S1204. Perform filling operations on each empty filling control according to the correspondence between the first table label, the empty filling control, and the to-be-filled content, to fill the to-be-filled content into the empty filling control.

[0267] In S1202 described above, the association between the empty filling control and the first table label is obtained, and the to-be-filled content is found from the user information library based on the first table label. Therefore, the mobile phone can determine the correspondence between the first table label, the empty filling control, and the to-be-filled content based on the above information. Subsequently, the mobile phone can fill each to-be-filled content into the corresponding empty filling control based on the correspondence between the three.

[0268] In some embodiments, in S1204 described above, the mobile phone performs filling operations on each empty filling control. Specifically, the filling operations can be performed on each empty filling control in sequence. Alternatively, as an example, in S1204 described above, the specific implementation logic of performing filling operations on each empty filling control in sequence is as follows: the mobile phone selects an empty filling control as a target filling control, performs a filling operation on the target filling control, then selects another empty filling control as a new target filling control, and performs a filling operation on the new target filling control, and so on, until all empty filling controls are filled with the corresponding to-be-filled content. For example, in the interface 201 shown in FIG. 2, taking the table corresponding to the editing of the recipient address as an example, the empty filling controls include four filling controls associated with the table labels "Name", "Phone", "Region", and "Detailed Address" (i.e., the first table label described above). When performing filling operations on each empty filling control in sequence, the mobile phone can first perform a filling operation on the empty filling control associated with the table label "Name", then perform a filling operation on the empty filling controls associated with the table labels "Phone" and "Region", and finally perform a filling operation on the empty filling control associated with the table label "Detailed Address". Figure 15

[0269] In some embodiments, performing a filling operation on the target filling control can specifically include: performing a simulated click on the target filling control based on the coordinate information of the filling area corresponding to the target filling control. The to-be-filled content matching the first table label associated with the target filling control is filled into the filling area corresponding to the target filling control. The coordinate information of the filling area corresponding to the target filling control can be obtained in S1202.

[0270] ​Generally, after a user performs a click operation on a fill-in control, the mobile phone displays a keyboard for the user to input, or the mobile phone enters a keyboard mode. Alternatively, in some embodiments of the present application, after a simulated click operation is performed on a target fill-in control, the mobile phone also displays a keyboard. After the content to be filled in is filled in the corresponding fill-in control, the mobile phone needs to be controlled to exit the keyboard mode, and then another empty fill-in control can be selected as a new target fill-in control to perform a fill-in operation. That is, in some embodiments, during the process of performing a fill-in operation on a target fill-in control, after the content to be filled in is filled in the fill-in area corresponding to the target fill-in control, a keyboard exit operation is also performed. Alternatively, in some other embodiments of the present application, after a simulated click operation is performed on a target fill-in control, the mobile phone enters a keyboard mode, and the keyboard can also not be displayed on the display interface.

[0271] It should be noted that the specific implementation of the keyboard exit operation can refer to the description in the related art, and will not be described here.

[0272] In some embodiments, the specific implementation of S1204 can specifically include: generating a content vector according to the obtained content to be filled in, and the first table label and the empty fill-in control. Then, the plurality of content vectors are stored in a preset data structure. The empty fill-in control in the content vector can be specifically represented by the coordinate information of the fill-in area corresponding to the empty fill-in control. Then, the mobile phone performs a fill-in operation on each empty fill-in control based on the preset data structure.

[0273] In some embodiments, the mobile phone performs a fill-in operation on each empty fill-in control based on the preset data structure, which can specifically include: taking out a content vector in the preset data structure, such as a first content vector. According to the coordinate information of the fill-in area in the first content vector, a target fill-in control is determined. The content to be filled in the first content vector is filled in the target fill-in control. The step of taking out a content vector in the preset data structure is returned. Then, based on the newly taken out content vector, a fill-in operation is performed on the next empty fill-in control until the preset data structure is empty; that is, the fill-in operation has been completed on all empty fill-in controls.

[0274] For example, the preset data structure can be a queue, such as a to-be-filled queue.

[0275] The to-be-filled queue can include a plurality of queue elements, and each queue element stores a content vector, including a first indication fill-in control, an empty fill-in control, and a content to be filled in; the first table label, the empty fill-in control, and the content to be filled in correspond one by one. For example, a queue element is composed of: (first table label, content to be filled in, coordinate information of the fill-in area corresponding to the empty fill-in control).

[0276] In some other embodiments, in order to better position the display position of the empty fill-in control, the queue element further comprises coordinate information of the first table label, for example, the constitution of a queue element is as follows: (the first table label, the content to be filled in, coordinate information of the label area, coordinate information of the fill-in area).

[0277] Further, the mobile phone selects an empty fill-in control as a target fill-in control, and performs the fill-in operation on the target fill-in control, which is implemented as follows: taking the first queue element in the queue of content to be filled in, for example, queue element A; according to the coordinate information of the fill-in area A in the queue element A, performing a simulated click operation on the fill-in area A (i.e. the empty fill-in control); filling the content to be filled in in the queue element A into the fill-in area A. It can be understood that the empty fill-in control corresponding to the fill-in area A in the queue element A is the target fill-in control for performing the fill-in operation. Since the first queue element in the queue of content to be filled in has been taken, the second queue element in the queue of content to be filled in before taking is taken as the first queue element in the queue of content to be filled in after taking. After the fill-in operation on the empty fill-in control corresponding to the fill-in area A based on the queue element A is completed, the first queue element in the queue of content to be filled in after taking, for example, queue element B, is taken again. Then, the empty fill-in control corresponding to the fill-in area B in the queue element B is taken as a new target fill-in control, and the above steps are repeated to perform the fill-in operation on the new fill-in control based on the queue element B, until the queue of content to be filled in is empty. It can be understood that in this embodiment, when the queue of content to be filled in is empty, it indicates that the fill-in operation on all empty fill-in controls has been completed.

[0278] Through the above method, the fill-in operation is performed on each empty fill-in control based on the queue of content to be filled in, and then the fill-in operation on all empty fill-in controls in the table is completed.

[0279] In the technical solution proposed in the embodiments of the present application, the mobile phone can automatically fill in the table in the first interface by one key in response to one user operation, which can greatly improve the efficiency of filling in the table.

[0280] In addition, after the fill-in operation on an empty fill-in control in the table is completed, some abnormal situations may occur before moving to the next empty fill-in control to perform the fill-in operation. In order to successfully fill in all empty fill-in controls by one key in response to one user operation, it is necessary to detect whether an abnormal situation occurs, and perform corresponding abnormal handling for the abnormal situation. The process of detecting the abnormal situation and performing the abnormal handling for the abnormal situation is described in detail below based on the implementation manner of sequentially performing the fill-in operation on each empty fill-in control based on the queue of content to be filled in. For example, Figure 14 The corresponding flow is shown.

[0281] In this embodiment, the to-be-filled queue includes n queue elements, and a queue element is composed of (k n , v n , bbox 表格标签n , bbox 填写控件n ). Wherein, k n represents the label text of the table label, v n represents the label data matched with k n , bbox 表格标签n represents the coordinate information of the label area corresponding to the table label n, and bbox 填写控件n represents the coordinate information of the fill-in area corresponding to the fill-in control n.

[0282] S1301. Take out the first queue element (k1, v1, bbox 表格标签1 , bbox 填写控件1 ), determine the display position of the fill-in control 1 according to bbox 填写控件1 , and fill v1 into the fill-in control 1.

[0283] The specific implementation of the above S1 can refer to the description of the above embodiment based on the to-be-filled queue to perform the fill-in operation on the empty fill-in control.

[0284] After taking out the first queue element (k1, v1, bbox 表格标签1 , bbox 填写控件1 ) from the to-be-filled queue, the element position in the to-be-filled queue will change, for example, the second queue element before taking out becomes the first queue element after taking out.

[0285] S1302. Detect whether an abnormal situation occurs.

[0286] In some embodiments, the abnormal situation can include that the keyboard mode is not exited. As can be known from the description of the above embodiment, after filling the to-be-filled content into the fill-in area corresponding to the target fill-in control, the mobile phone will also perform a keyboard exit operation. Therefore, under normal circumstances, after the fill-in operation on the empty fill-in control is completed, the keyboard should be exited.

[0287] Optionally, as an example, the mobile phone can detect whether the keyboard mode is not exited by the following manner: the mobile phone identifies whether the first interface includes a keyboard to determine whether the abnormal situation of the keyboard mode not being exited exists. Specifically, if the first interface includes a keyboard, it indicates that the abnormal situation of the keyboard mode not being exited exists. On the contrary, if the first interface does not include a keyboard, it indicates that the abnormal situation of the keyboard mode not being exited does not exist. Wherein, identifying whether the first interface includes a keyboard can be implemented by any manner.

[0288] As another example, in an embodiment where the phone is in the keyboard mode but does not display the keyboard, the phone can determine whether there is an abnormal situation where the keyboard mode is not exited by obtaining keyboard mode related information from the background.

[0289] In some embodiments, the abnormal situation can include that the table is shifted. The table can be shifted when a fill operation is performed on one fill control in the table. After the table is shifted, the display position of the fill control is changed. The change of the display position of the fill control can affect the automatic fill process of the table and cause the fill operation to be unable to be automatically performed continuously. For example, as shown in the example of FIG. 14A, after a simulated click operation is performed on one fill control in the table, the cursor is activated, i.e., the phone displays the cursor in the fill area corresponding to the fill control. The cursor is used to indicate the position of the current operation, and the position of the cursor moves with the filled content. For example, as shown in the example of FIG. 14B, after the fill operation performed on the fill control is completed, the position of the cursor is position 1402. In another scenario, as shown in the example of FIG. 15A, when the cursor is activated, the position of the cursor is position 1501. After the cursor is activated, the table is shifted. In this example, after the fill operation performed on the fill control is completed, the position of the cursor is position 1502. Figure 14 Figure 15

[0290] Optionally, as an example, the phone can detect whether the table is shifted by obtaining a first cursor position when the cursor is activated and a second cursor position after the fill operation performed on the current fill control is completed. The difference between the vertical coordinates of the first cursor position and the second cursor position is calculated. If the difference is greater than or equal to an error threshold, it is considered that the table is shifted. The error threshold can be set according to actual conditions. Specifically, the error threshold is a number greater than 0. The current fill control refers to the fill control that is currently selected and on which the fill operation is completed, i.e., the fill control on which the cursor is located.

[0291] As an example in FIG. 14B, the first cursor position is position 1401, and the second cursor position is position 1402. The vertical coordinates of the two positions are the same, and the difference is less than the error threshold. That is, in the example of FIG. 14A, the table is not shifted. Figure 15 As an example in FIG. 15B, the first cursor position is position 1501, and the second cursor position is position 1502. The vertical coordinates of the two positions are different. The difference is greater than or equal to the error threshold. That is, in the example of FIG. 15A, the table is shifted. Figure 14 Figure 16 Figure 17

[0292] ​​​​​If the result of the determination of S1302 is no, it indicates that no abnormal situation occurs, and S1304 can be directly executed. If the result of the determination of S1302 is yes, it indicates that an abnormal situation occurs, and S1303 is executed first, and then S1304 is executed.

[0293] S1303. Abnormal situation handling is performed.

[0294] The main purpose of the abnormal situation handling for the keyboard mode not being exited is to control the mobile phone to exit the keyboard mode.

[0295] In the case where the mobile phone is in the keyboard mode, a common operation for exiting the keyboard is to click the area corresponding to the non-filling control in the first interface, other than the keyboard. For example, the area corresponding to the table label in the table in the first interface or the empty area in the first interface. In some embodiments, the abnormal situation handling for the keyboard mode not being exited can include determining the first table label associated with the current filling control, obtaining the coordinate information of the label area corresponding to the first table label, and performing a simulated click operation in the label area based on the coordinate information of the label area.

[0296] In the case where the mobile phone is in the keyboard mode, another common operation for exiting the keyboard is to click the control corresponding to the complete key or the enter key in the keyboard area in the first interface. For example, the control of the complete key can be displayed as the word "complete" / "next item" / …, or as the symbol "√" in the keyboard as shown in FIG. 5B, and the like. Similarly, the control of the enter key can be displayed as the word "enter" / "enter" / …, or as the enter symbol. In other embodiments, the abnormal situation handling for the keyboard mode not being exited can include obtaining the coordinate information of the preset control in the keyboard area in the first interface, and performing a simulated click operation on the preset control based on the coordinate information of the preset control. The preset control can be the control corresponding to the complete key or the enter key. Figure 18 In the case where the abnormal situation of the table being shifted occurs, the display position of the empty filling control which has not performed the filling operation is changed, and the filling queue generated according to the content to be filled before this time cannot correspond to the shifted table, and the empty filling control which has not performed the filling operation cannot complete the automatic filling based on the filling queue. Therefore, the main purpose of the abnormal situation handling for the table being shifted is to continue to perform the filling operation on the empty filling control behind.

[0297]

[0298] ​In some embodiments, the abnormality handling for the table displacement includes: after the table displacement, re-acquiring the to-be-filled content in the displaced table, and generating a new to-be-filled queue. That is, after determining that the table is displaced, the displaced table is re-identified, and the empty filling control in the displaced table and the table label (denoted as the second table label) associated with the empty filling control are obtained. Then, the new to-be-filled content can be re-acquired from the user database based on the second table label in the displaced table, and a new to-be-filled queue is generated. Finally, the mobile phone can fill the new to-be-filled content into the empty filling control in the displaced table based on the new to-be-filled queue. The specific implementation of identifying the displaced table, obtaining the to-be-filled content, and filling the to-be-filled content into the empty filling control is consistent with the corresponding operation specific implementation in the above-mentioned embodiments, and will not be repeated here.

[0299] In other embodiments, the abnormality handling for the table displacement includes: after the table displacement, updating the to-be-filled queue based on the displaced table. Generally, in the case of table displacement, the displaced table and the table before displacement itself only have the display positions of the table labels and the filling controls changed, and the table labels and the filling controls, and the association relationship between the table labels and the filling controls will not change. Among them, only the display positions of the empty filling controls that have not performed the filling operation when the table is displaced may affect the execution of the subsequent filling operation. As can be known from the description of the above-mentioned embodiments, the to-be-filled elements in the to-be-filled queue, the k n The label text of the table label. Therefore, the above-mentioned updating the to-be-filled queue based on the displaced table can specifically include: re-identifying the empty filling controls (denoted as the remaining empty filling controls) in the table, and the table labels (denoted as the remaining table labels) associated with the remaining empty filling controls, obtaining the coordinate information of the remaining empty filling controls, and the label text of the remaining table labels. Then, based on the label text of the remaining table labels, the coordinate information of the remaining empty filling controls is updated to the corresponding queue element of the to-be-filled queue.

[0300] It should be noted that the new to-be-filled queue generated based on the displaced table, and the updated to-be-filled queue correspond to the empty filling controls that have not performed the filling operation after the table displacement.

[0301] In some embodiments, the abnormal situation can include that the keyboard mode is not exited and the table is shifted. Alternatively, as an example, when detecting whether there is an abnormal situation, the keyboard mode can be first detected whether it is not exited, and then the table is detected whether it is shifted when the keyboard mode is exited. When there is an abnormal situation that the keyboard mode is not exited, the abnormal processing for the keyboard mode not exited is first performed, and then the keyboard mode is detected whether it is exited again after the abnormal processing for the keyboard mode not exited is performed. If the keyboard mode is exited after the abnormal processing for the keyboard mode not exited is performed, the table is detected whether it is shifted. If the keyboard mode is not exited after the abnormal processing for the keyboard mode not exited is performed, the automatic form filling process of the table can be ended, and the table is not detected whether it is shifted.

[0302] Further, when the keyboard mode is first detected whether it is exited, if it is detected that the keyboard mode is exited, the table can be further detected whether it is shifted. Then, if the table is shifted, the abnormal processing for the table shifted can be performed, the new to-be-filled queue can be regenerated, or the to-be-filled queue can be updated according to the manner in the above embodiments, and then the filling operation for the next empty filling control can be continued, that is, S1304 is performed. If it is detected that the keyboard mode is exited and the table is not shifted, the filling operation for the next empty filling control can be directly performed, that is, S1304 is performed.

[0303] In the embodiments in which the abnormal situation can include that the keyboard mode is not exited and the table is shifted, the result of S1302 is yes, which corresponds to the following scenario: when the abnormal situation is first detected, the keyboard mode is exited, and the table is not shifted. S1302 above first detects whether the keyboard mode is not exited, and if yes, S1303 is performed to perform the abnormal processing for the keyboard mode not exited. Then, whether the keyboard mode is not exited is judged, and if the keyboard mode is not exited, the automatic form filling process of the table is directly ended. If the keyboard mode is exited, S1302 is performed again to detect whether the table is shifted. If the table is shifted, S1303 is performed again to perform the abnormal processing for the table shifted. It should be noted that after the abnormal processing for the table shifted is performed, S1304 can be directly performed.

[0304] S1304. The first queue element (k2, v2, bbox 表格标签2 , bbox 填写控件2 ) is taken out, the display position of the filling control 2 is determined according to bbox 填写控件2 , and v2 is filled into the filling control 2.

[0305] In the technical solution provided in the embodiments of the present application, after the filling operation on one empty filling control is completed, it is detected whether an abnormal situation occurs. When an abnormal situation occurs, the mobile phone automatically performs corresponding abnormal processing without user operation. In this way, it can be ensured that the mobile phone can automatically complete one-key filling of the entire form.

[0306] In some embodiments, several parts of the above form automatic filling process can be implemented by state machines respectively. The form of state machine is convenient to manage and can realize efficient and accurate form automatic filling. The state machines corresponding to the parts of the form automatic filling process are described in detail below.

[0307] Each state machine can include two parts, one part is state running action, and the state running action part is used for the state machine to execute related logic. The other part is state transition condition, and the state transition condition is used to determine the next operation to be performed according to the corresponding judgment condition, such as jumping to another state machine or ending the form automatic filling process.

[0308] 1. The state machine corresponding to the visual detection part, such as state machine 1 or visual detection state machine, the specific implementation of the state machine 1 is as follows Figure 19 .

[0309] State running action: taking a screenshot of the first interface to obtain a first interface screenshot. The first model is used to recognize the first interface screenshot, detect several categories of table elements included in the first interface, and give their respective corresponding bbox. In this embodiment, the table elements are divided into five categories, including: ① table label L t ; ② unfilled input control (i.e. empty filling control) I u ; ③ filled input control I f ; ④ unfilled drop-down control (i.e. empty drop-down control) D u ; and ⑤ filled drop-down control D f .

[0310] In addition, when the mobile phone uses the first model to recognize the first interface screenshot, the table label and the associated input control or drop-down control are given in the form of a sequence of paired bbox . Each sequence element represents the bbox of a pair of filling controls and the bbox of a table label. The bbox of the input control is specifically the bbox of the filling area corresponding to the input control; the bbox of the table label is specifically the bbox of the label area corresponding to the table label. It should be noted that this part corresponds to S1201 and S1202 described above. The following takes the input control as an example for description.

[0311] For example, the bbox of the input control is represented as Its upper left corner coordinates are The coordinates of the bottom right corner are The bbox of the table label is represented as Its upper left corner coordinates are The coordinates of the bottom right corner are Paired bbox sequences The input control portion in the code can be represented as:

[0312]

[0313] in:

[0314]

[0315]

[0316] Understandable, paired bbox sequences The dropdown control portion is represented similarly to the input control.

[0317] State transition condition: First determine the paired bounding box sequence. Is it empty? If the paired bbox sequence If empty, it means there are currently no fill controls in the table, and the automatic form filling process ends. If the paired bbox sequence... If not empty, then check if the queue Q to be filled is empty. If the paired bounding box sequence... If the field is not empty and the queue Q is empty, it means that no content matching the empty fill control has been retrieved yet; in this case, we can jump to state machine 2. If the paired bbox sequence... If the field is not empty and the queue Q is not empty, it means that a field matching the empty field control has been found; at this point, the process can jump to state machine 3.

[0318] 2. The state machine corresponding to the part where the content to be filled is to be retrieved, such as state machine 2 or the content to be filled retrieval state machine. The specific implementation of state machine 2 is as follows: Figure 20 .

[0319] State execution action: For the empty input control I identified in state machine 1 using visual inspection. u Related table label (the first table label mentioned above) L t and empty dropdown control D u Associated table label L t Combine OCR model to obtain table label L t The corresponding label text. And, the table label L... tThe corresponding label text is taken as a key (K), and the label data matched with the table label Lt is queried from the user information library as a value (V).

[0320] The bbox of an empty input control I u and the bbox of the table label L u associated with the empty input control I t are combined into a key-value pair vector, and the key-value pair vector is combined into a to-be-filled queue Q. The to-be-filled queue Q including the to-be-filled content corresponding to the empty input control can be expressed as:

[0321]

[0322] Similarly, the bbox of an empty drop-down control D u and the bbox of the table label L u associated with the empty drop-down control D t are combined into a key-value pair vector.

[0323] The state running action of the state machine 2 specifically corresponds to S1203 in the above embodiment.

[0324] State transition condition: it is judged whether the to-be-filled queue Q is empty. If the to-be-filled queue Q is not empty, it indicates that the filling operation can be performed; at this time, it is jumped to the state machine 4. If the to-be-filled queue Q is empty, it indicates that the label data matched with the table label L t corresponding label text is not found, the automatic filling of the table cannot be realized, and the automatic filling process of the table is ended.

[0325] 3. The state machine corresponding to the to-be-filled content maintenance part, such as state machine 3 or to-be-filled content maintenance state machine, the specific implementation of the state machine 3 is as follows: Figure 21 .

[0326] State running action: the bbox of the empty filling control re-detected by the visual detection state machine is obtained. The bbox of the empty filling control in the to-be-filled queue Q is updated to be consistent with the bbox of the empty filling control re-detected by the visual detection state machine. This part corresponds to the process of executing the exception handling when there is an exception in the table in S1302 and S1303 in the above embodiment.

[0327] State transition condition: after the state running action is executed, it can be jumped to the state machine 4.

[0328] 4. The state machine corresponding to the filling execution part, such as state machine 4 or filling execution state machine, the specific implementation of the state machine 4 is as follows: Figure 22 .

[0329] Status operation action: First, check if the queue to be filled Q is empty. If the queue to be filled is empty, it means that all empty fill controls have been automatically filled, and the automatic form filling process ends.

[0330] Conversely, if the queue Q to be filled is not empty, it means that there are still empty input controls in the first interface that need to be automatically filled. The following explanation uses the empty input control corresponding to the first queue element as an example. Retrieve the first queue element (k) from the queue Q to be filled. i ,v i bbox Iu,i bbox Lt,i ), based on the bbox in the first queue element Iu,i Determine the empty input control I u The coordinates of the top left corner of the bbox and the coordinates of the bottom right vertex Calculate the empty input control I u Center coordinates:

[0331]

[0332] For bbox Iui center coordinates Perform a simulated click to activate the cursor. Obtain the cursor's position (i.e., the initial cursor position mentioned above). and the value v in the first queue element i Inject the empty input control I u The cursor position can specifically be the coordinates of the top-left corner of the cursor. Finally, control the phone to exit keyboard mode.

[0333] It should be noted that if the first queue element in the queue to be filled Q corresponds to an empty drop-down control, its implementation is similar to the specific implementation of the filling process of an empty input control, and will not be elaborated here.

[0334] The state operation action of state machine 4 corresponds to the process of performing a filling operation on an empty filling control in S1204 of the above embodiment.

[0335] State transition condition: Check again whether the queue Q to be filled is empty. If the queue Q to be filled is empty, it means that all empty fill controls have been automatically filled, and the form autofill process ends. Otherwise, jump to state machine 5.

[0336] 5. State Machine for Exception Handling. The following explanation uses exceptions such as failure to exit keyboard mode and table shifting as examples. The state machine for exception handling includes state machines 5, 6, and 7.

[0337] The state machine 5, also called a keyboard exit abnormality detection state machine, is used to detect whether there is an abnormality of keyboard mode not being exited. In this embodiment, the case where the phone displays the keyboard after the keyboard mode is not exited is taken as an example. The specific implementation of the state machine 5 is as follows: Figure 23A :

[0338] State running action: Detect whether the keyboard exists.

[0339] The state running action of the state machine 6 corresponds to the process of detecting whether there is an abnormality of keyboard mode not being exited in the above embodiment.

[0340] State transition condition: Determine whether the keyboard exists. If the keyboard exists, it indicates that there is an abnormality of keyboard mode not being exited, and the state machine 7 is jumped to. If the keyboard does not exist, it indicates that there is no abnormality of keyboard mode not being exited, and the state machine 6 is jumped to.

[0341] The state machine 6, also called a table shift abnormality processing state machine, is used to detect whether there is an abnormality of table shift, and determine the next execution action when the abnormality of table shift occurs. The specific implementation of the state machine 6 is as follows: Figure 23B :

[0342] State running action: Obtain the position of the cursor (i.e., the second cursor position mentioned above) According to the position of the cursor obtained in the state machine 4 and the position of the cursor obtained in the state machine 4 the vertical offset value of the cursor is calculated

[0343] The state running action of the state machine 6 corresponds to the process of detecting whether the table has been shifted in the above embodiment.

[0344] State transition condition: Determine whether the vertical offset value of the cursor is greater than or equal to the error threshold value. If It is indicated that the vertical offset value of the cursor is less than the error threshold value ε, and the automatic table filling process can continue, i.e., the state machine 4 is jumped to. If the vertical offset value of the cursor is greater than or equal to the error ε, it indicates that the abnormality of table shift occurs after the filling control is filled. When the abnormality of table shift occurs, the phone can re-identify the shifted table, and thus the state machine 1 is jumped to.

[0345] The state machine 7, also called a keyboard exit abnormality processing state machine, is used to perform the keyboard exit operation again for the keyboard that fails to exit. The specific implementation of the state machine 7 is as follows: Figure 23A :

[0346] State running action: Perform the keyboard exit operation.

[0347] The way of performing the keyboard exit operation includes way one and / or way two, wherein:

[0348] Method one: obtaining the position of the cursor According to the position Determine the target click position. Then perform a simulated click operation at the target click position to exit the keyboard.

[0349] For example, the mobile phone can calculate the center coordinates of the bbox of the table label associated with the current fill-in control according to the bbox of the fill-in control and the bbox of the table label, and take the center coordinates of the table label as the target click position. The center coordinates of the bbox of the table label As follows:

[0350]

[0351] Method two: locate the display position of the corresponding control such as the complete key or the enter key on the keyboard through the first model, and then perform a simulated click operation on the display position of the corresponding control such as the complete key or the enter key. This way two is to exit the keyboard by confirming the input.

[0352] In some embodiments, when an abnormal situation of the keyboard mode not being exited occurs, the failed keyboard to exit can be a secure keyboard. The state running action of the state machine 7 corresponds to the process of handling the abnormal situation of the keyboard mode not being exited in the above-mentioned embodiments.

[0353] State transition condition: detect again whether the keyboard exists. If it still exists, the automatic form filling process ends. If the keyboard does not exist after handling the abnormal situation of the keyboard mode not being exited, jump to the state machine 6.

[0354] In the above-mentioned embodiments, the information processing method is implemented through the state machine, which can efficiently and accurately implement one-key automatic filling of the form. One-key filling of multiple fill-in controls in the form can improve the form filling efficiency.

[0355] Further, after completing the form filling, the content in the filled form can be extracted and updated to the user information library. Therefore, in some embodiments, the above-mentioned method can further include:

[0356] In response to a user operation, the first model is used to identify the first interface screenshot, detect several categories of form elements included in the form in the first interface, and give their respective corresponding bbox. It should be noted that the form in the first interface has completed form filling. Since the content extraction is performed on the filled form in this method, the main object is the filled input control I f and the filled drop-down control D fand its respective associated table label are extracted. Moreover, in the process of identifying the table, it is also necessary to determine the association relationship between the table label and the filled-in input control I f and the filled-in drop-down control D f . For example, in the output result of the first model, the association relationship between the table label and the filled-in input control I f and the association relationship between the table label and the filled-in drop-down control D f can be represented by key-value pairs.

[0357] Further, based on the OCR model, the text of the filled-in input control I f and the label text of the table label L t associated therewith are obtained. Further, the corresponding key-value pair of the filled-in input control I f can be represented as:

[0358] KV If,i ={(K i ,V i ,bbox If,i ,bbox Lt,i )|i=1,2,…,n}

[0359] Similarly, based on the OCR model, the text of the filled-in drop-down control D f and the label text in the table label L t associated therewith can also be obtained. Further, the corresponding key-value pair of the filled-in drop-down control D f can be represented as:

[0360] KV Df,i ={(K i ,V i ,bbox Df,i ,bbox Lt,i )|i=1,2,…,m}

[0361] In order to avoid extracting duplicate table content, after obtaining the corresponding key-value pair of the filled-in input control I f and the corresponding key-value pair of the filled-in drop-down control D f , the mobile phone can also perform a set union operation on the two key-value pairs, and the set union is taken as the final extracted table content. Specifically, the set union operation on the two key-value pairs can be a set union based on the bbox of the table label: only one associated filled-in control, input control or drop-down control, is retained for the bbox of a table label. Specifically, the set union of formula (6) and formula (7) can be represented as:

[0362] KV total =KVIf ∪KV Df

[0363] KV total = {(K i , V i , bbox If,i , bbox Df,i , bbox Lt,i ) | i = 1, 2, …, n + m}

[0364] After obtaining the KV total , the mobile phone can store the KV total to the user information base. Optionally, as an example, the mobile phone stores the KV total to the user information base, and specifically can directly store the KV total to the user information base. As another example, before the mobile phone stores the KV total to the user information base, the mobile phone can also compare the KV total extracted this time with the data in the user information base. If there is duplicate data, only the data in the KV total extracted this time that has been updated can be stored to the user information base.

[0365] In the technical solutions proposed in the embodiments of the present application, the screenshot recognition through the first interface can extract the control text of the filled-in fill-in control and the label text of the table label associated with the filled-in fill-in control, and further generate the key-value pair of the table label and the fill-in control. The key-value pair can help update the user information base and maintain the richness and accuracy of the data in the user information base.

[0366] In addition, the embodiments of the present application also propose an information processing method, which is specifically used for content extraction of a filled-in form. For example, the method can be applied after the user manually fills in the form. The specific implementation of the method is described in detail above with reference to the embodiments of the present application for extracting the content of the filled-in form.

[0367] Based on the exemplary information processing method of the above embodiments, in the process of automatically filling in the form, the electronic device needs to obtain the user information from the user information base and fill the user information into the correct position of the form. Based on this, the electronic device needs to know the position of the fill-in box in the form and the type of the form associated with the fill-in box, so as to obtain the correct user information.

[0368] Exemplarily, the embodiment of the present application provides a table visual detection method, an electronic device, a computer readable storage medium and a computer program product. With the aid of a table visual detection module and a non-maximum suppression (NMS) algorithm, the electronic device can not only position and classify table labels and fill-in boxes in a table, but also pair the table labels with the fill-in boxes in the table, thereby realizing automatic filling of the table.

[0369] The electronic device can include, but is not limited to, a mobile phone, a tablet computer, a notebook computer, an ultra-mobile personal computer (UMPC), a netbook, a personal digital assistant (PDA), a mobile phone, a wearable device, a vehicle-mounted device, an augmented reality (AR) / virtual reality (VR) device, a smart television, and the like, and the embodiment of the present application does not limit the electronic device.

[0370] The table can have various styles, such as the various style tables shown in Figure 23B and Figure 23A . The table can be displayed in an interface, and the table has a table structure, that is, the table. The table can include, but is not limited to, a table in an application program, a web form in a browser, and the like, such as an online registration form, filling in a survey questionnaire, filling in a courier information form, and the like.

[0371] The table can include one or more targets, and the target category of the target can include a table label and a fill-in box, as shown in Figure 23B and Figure 23A .

[0372] The table label generally refers to an identifying word or symbol in the table for describing the content, attribute or category of the table. For example, the table label can include a label name, or a label name and a label box. The label name can be "name" as shown in Figure 23B and Figure 23A . The label box can be a box that encloses "name" as shown in Figure 23B and Figure 23A .

[0373] The label box mentioned in the embodiment of the present application can also be referred to as a prompt box or a prompt control. The label name mentioned in the embodiment of the present application can also be referred to as a label text.

[0374] The fill-in box generally refers to an area where a user can input or edit data. The fill-in box mentioned in the embodiment of the present application can also be referred to as a fill-in control.

[0375] Next, the multiple specific implementation manners of the fill-in box are described.

[0376] In some embodiments, as shown in Figure 23B and Figure 23A , the fill-in box can include: an unfilled input box, a filled input box, an unfilled drop-down box, and a filled drop-down box.

[0377] In some embodiments, as shown in Figure 23B and Figure 23A , the fill-in box can include: an input box and a drop-down box. The input box includes an unfilled input box and a filled input box. The drop-down box includes an unfilled drop-down box and a filled drop-down box.

[0378] The input box generally refers to an interactive control that allows users to input text, numbers, and other characters. The drop-down box, also known as a drop-down list or combo box, generally refers to an interactive control that provides limited options for users to choose from. The input box mentioned in the embodiments of the present application can also be referred to as an input control. The drop-down box mentioned in the embodiments of the present application can also be referred to as a drop-down control.

[0379] In some embodiments, as shown in Figure 23B and Figure 24 , the fill-in box can include: an unfilled fill-in box and a filled fill-in box. The unfilled fill-in box includes an unfilled input box and an unfilled drop-down box. The filled fill-in box includes a filled input box and a filled drop-down box.

[0380] As can be seen, in the table, the target category of the target can include five categories, namely table label, unfilled input box, filled input box, unfilled drop-down box, and filled drop-down box.

[0381] In addition, the fill-in box can also include: an information element. When the fill-in control is not filled with content, the information element can be a prompt content, such as Figure 24 “Please enter your name here” as shown. When the fill-in control has been filled with content, the information element can be a filled content, such as Figure 24 “Zhao Yiyi” as shown. The prompt content mentioned in the embodiments of the present application can also be referred to as a prompt text. The filled content mentioned in the embodiments of the present application can also be referred to as a filled text.

[0382] Of course, the target category of the target can also include two categories, namely table label and fill-in box.

[0383] Alternatively, the target category of the target can include three categories, namely table label, input box, and drop-down box.

[0384] Alternatively, the target category of the target can include three categories, namely table label, unfilled fill-in box, and filled fill-in box.

[0385] Referring to Figure 24 , Figure 24 Part of a software structure of an electronic device is provided in an embodiment of the present application. As shown in Figure 23A , a first model in the electronic device can include an image preprocessing module 2411, a table visual detection module 2412, and an NMS algorithm module 2413.

[0386] The first model is configured to perform table visual detection on a table in an interface, and determine a pair of targets having a pairing relationship in the table, so that based on one of the pair of targets, the other target in the pair of targets can be accurately determined to be filled with user information.

[0387] The image preprocessing module 2411 can preprocess parameters such as size, dimension, pixel, or grayscale of the interface to obtain a preprocessed interface. The interface includes a table. For example, Figure 23B , the interface can be the first interface described below. The format of the preprocessed interface is consistent with the input format of the table visual detection module 2412.

[0388] In some embodiments, the image preprocessing module 2411 can adjust the size or space occupied by the table interface, thereby reducing the storage space and speeding up the transmission. And / or, the image preprocessing module 2411 can perform normalization processing on the table interface, so that the pixel value of the interface is adjusted to a certain specific interval, facilitating the subsequent use of the interface and ensuring the performance and stability of the interface processing.

[0389] Therefore, the image preprocessing module 2411 can transmit the preprocessed interface to the table visual detection module 2412.

[0390] The image preprocessing module 2411 is a software code or a software module. The image preprocessing module 2411 can be set in a memory in the electronic device and called by a processor such as a central processing unit (CPU) or a graphics processing unit (GPU) in the electronic device.

[0391] Based on the preprocessed interface, the table visual detection module 2412 can perform detection such as positioning, classification, and pairing on a plurality of targets in the table to obtain a plurality of detection results.

[0392] Each detection result corresponds to a group of paired bounding boxes. Each group of paired bounding boxes can include a plurality of bounding boxes, such as two bounding boxes. The plurality of bounding boxes correspond to targets of different target categories in the table.

[0393] For example, if one of the two bounding boxes has a target category of table label, then the other bounding box can have a target category of fill-in box, or the other bounding box can have a target category of drop-down box or input box, or the other bounding box can have a target category of unfilled fill-in box or filled fill-in box, or the other bounding box can have a target category of unfilled input box, filled input box, unfilled drop-down box, or filled drop-down box.

[0394] Each detection result can include relevant feature information of each bounding box in the set of paired bounding boxes, such as a position and a confidence of each target category. Each target category here refers to each target category of the target.

[0395] In some examples, the table visual detection module 2412 can extract a feature map from the preprocessed interface. The feature map F can be represented as Generally, R represents a real number set, C is a number of channels, H is a height of the feature map, and W is a width of the feature map.

[0396] The table visual detection module 2412 can apply a plurality of convolution layers on the feature map, that is, the table visual detection module 2412 performs a convolution operation on the feature map, which can reduce the number of channels C to an output dimension required by the table visual detection module 2412.

[0397] In some embodiments, the table visual detection module 2412 can represent the process of the convolution operation described above by a tensor C in Formula Seven:

[0398] C = Conv2D(F, 2 x (4 + K)) Formula Seven

[0399] where Conv2D represents a two-dimensional convolution operation, and 2 x (4 + K) represents that two bounding boxes are predicted at each spatial position of the feature map, each bounding box includes 4 coordinate values and confidence of K categories, and the 4 coordinate values represent a position of the corresponding bounding box, such as coordinates and dimensions. For example, the 4 coordinate values include a coordinate of a top-left corner of the bounding box, a width of the bounding box, and a height of the bounding box. K is a number of target categories.

[0400] After the convolution operation, the size of the tensor C output by the table visual detection module 2412 is (2 x (4 + K)) x H x W, that is, the table visual detection module 2412 can output relevant feature information of a plurality of paired bounding boxes at each spatial position of the feature map. In this way, the table visual detection module 2412 can realize positioning, classification, and pairing of each target in the table.

[0401] Therefore, the table visual detection module 2412 can transmit the related feature information of the multiple sets of paired bounding boxes to the NMS algorithm module 2413.

[0402] In some embodiments, the table visual detection module 2412 can include a feature extraction module 24121, a feature post-processing module 24122, and a table label paired detection head module 24123. Figure 24

[0403] The feature extraction module 24121 can extract features from the preprocessed interface to generate a feature map. The feature extraction module 24121 can transmit the feature map to the feature post-processing module 24122. The feature post-processing module 24122 performs post-processing and feature selection on the feature map to obtain a processed feature map. Therefore, the feature post-processing module 24122 can transmit the processed feature map to the table label paired detection head module 24123.

[0404] The table label paired detection head module 24123 can locate, classify, and pair each target in the table according to the processed feature map to obtain multiple detection results, i.e., the related feature information of the multiple sets of paired bounding boxes. That is, the table label paired detection head module 24123 can detect each target in the table and determine the target category, position, and pairing relationship of each target.

[0405] Therefore, the table label paired detection head module 24123 transmits the related feature information of the multiple sets of paired bounding boxes to the NMS algorithm module 2413.

[0406] In the table visual detection module 2412, the feature extraction module 24121 and the feature post-processing module 24122 can be different software modules or can be integrated in the same software module. The feature extraction module 24121 and the feature post-processing module 24122 can use a conventional feature extraction algorithm, such as a you only look once (YOLO) algorithm. The feature extraction module 24121 can use an algorithm such as a convolutional neural network. The feature post-processing module 24122 can use an algorithm such as a feature pyramid network (FPN).

[0407] The table label paired detection head module 24123 is a software module. The table label paired detection head module 24123 can be separately arranged from the feature extraction module 24121 and the feature post-processing module 24122 or can be integrated with the feature extraction module 24121 and the feature post-processing module 24122. ​

[0408] The table label matching detection head module 24123 needs to be trained in order to locate, classify, and match the various targets in the table. In some embodiments, the table label matching detection head module 24123 can be trained alone, or it can be trained together with the feature extraction module 24121 and the feature post-processing module 24122, that is, the entire table visual detection module 2412 is trained.

[0409] In some instances, training samples are typically used to train the table visual detection module 2412, enabling the table label matching detection head module 24123 in the table visual detection module 2412 to accurately locate, classify, and match each target in the table.

[0410] The training samples include multiple tables, and each table can be labeled with attributes such as location, target category, and pairing relationship.

[0411] Additionally, when the target category is a fill box, the target's style and / or the number of words in the target's prompt can also be tagged, which helps to distinguish whether the target category is a table label or a fill box.

[0412] Additionally, when the target category is a dropdown list, a symbol is used to indicate that the target is a dropdown list, such as... Figure 25 and Figure 25 The symbol ">" shown can also be used for labeling, which helps to distinguish whether the target category is an input box or a drop-down box.

[0413] In addition, when the target category is a fill box, the status of whether the target is filled can also be tagged, which helps to distinguish whether the target category is an unfilled fill box or a filled fill box.

[0414] It should be noted that the embodiments of this application do not limit the information of the labeled targets in the training samples.

[0415] In other instances, by combining a multi-task loss function and training samples to train the table visual detection module 2412, the table label matching detection head module 24123 in the table visual detection module 2412 can more accurately locate, classify, and match each target in the table, taking into account the losses caused by localization, classification, and matching.

[0416] The NMS algorithm module 2413 can filter out paired bounding boxes with high confidence and low overlap from multiple paired bounding boxes based on the relevant feature information of multiple paired bounding boxes, remove duplicate and low-quality paired bounding boxes, and obtain filtered multiple paired bounding boxes, thereby ensuring that these paired bounding boxes do not overlap excessively in the same category.

[0417] In this way, the NMS algorithm module 2413 can determine each target in the table and the target having a pairing relationship with the corresponding target according to the screened multiple sets of paired bounding boxes, that is, multiple sets of paired targets.

[0418] For example, for each table label in the table, the NMS algorithm module 2413 can determine the fill-in box having a pairing relationship with each table label according to the screened multiple sets of paired bounding boxes.

[0419] Therefore, the NMS algorithm module 2413 can obtain the pairing relationship and related feature information such as the position and the confidence of the target category of the multiple sets of paired targets in the table.

[0420] The NMS algorithm module 2413 is a software code or a software module. The NMS algorithm module 2413 can be set in a memory in the electronic device and be called by a processor such as a CPU or a GPU in the electronic device.

[0421] In the following, the embodiment of the present application will take the structure shown in Figure 25 The structure as an example, combined with the drawings and application scenarios, the table visual detection method provided by the embodiment of the present application is described in detail.

[0422] Please refer to Figure 26A , Figure 26B The flowchart of a table visual detection method provided by the embodiment of the present application. As Figure 26A shown, the table visual detection method provided by the embodiment of the present application can include:

[0423] S2501, acquire the first interface, the first interface includes multiple targets of the table.

[0424] In the case that the first interface includes multiple targets of the table, based on the filling intention of the user, the electronic device can use multiple ways to acquire the first interface. In some embodiments, the electronic device can intercept the image displayed on the display after receiving the operation on the first control or the operation on the hardware button.

[0425] Among them, the first control is a virtual control on the image, and the operation on the first control can include but is not limited to: single click, double click, sliding touch operation, etc. The hardware button is a physical entity in the electronic device, and the operation on the hardware button can include but is not limited to: single click, double click, etc. Physical pressing operation.

[0426] When the format of the image conforms to the input format of the table visual detection module 2412, the electronic device can determine the image as the first interface.

[0427] When the format of the image does not conform to the input format of the table visual detection module 2412, the electronic device can pre-process the image through the image pre-processing module 2411, process the format of the image to conform to the input format of the table visual detection module 2412, and obtain a pre-processed interface. Thus, the electronic device can determine the pre-processed interface as the first interface.

[0428] In summary, the electronic device can obtain the first interface based on the filling intention of the user.

[0429] S2502, detecting a plurality of targets of the table to obtain a plurality of detection results, the detection results including a position of a set of paired bounding boxes and a confidence of each target category, and the set of paired bounding boxes including a plurality of bounding boxes having a pairing relationship.

[0430] The electronic device can position, classify, and pair the targets in the table according to the first interface to obtain a plurality of detection results.

[0431] Each detection result includes relevant feature information of a set of paired bounding boxes. The set of paired bounding boxes can include a plurality of bounding boxes, and the plurality of bounding boxes have a pairing relationship. In some embodiments, the plurality of bounding boxes can be two bounding boxes, or three or more bounding boxes.

[0432] A detection result includes relevant feature information of all bounding boxes in the paired bounding boxes. The relevant feature information of a bounding box can include a position of the bounding box and a confidence of each target category of the bounding box. The confidence of a target category refers to a probability that the target belongs to the target category.

[0433] For example, the position of a bounding box can be represented by a coordinate of a vertex of the bounding box, a width of the bounding box, and a height of the bounding box. Alternatively, the position of a bounding box can also be represented by coordinates of four vertices of the bounding box. Alternatively, the position of a bounding box can also be represented by coordinates of intersection points of two diagonals of the bounding box, a width of the bounding box, and a height of the bounding box.

[0434] In a set of paired bounding boxes, the target categories of the plurality of bounding boxes are different. For example, when the plurality of bounding boxes include a first bounding box and a second bounding box, the target categories of the first bounding box and the second bounding box are different.

[0435] When the target category of the first bounding box is the table label, the target category of the second bounding box can be any one of the aforementioned unfilled input box, the filled input box, the unfilled drop-down box, and the filled drop-down box. Alternatively, the target category of the second bounding box can be the aforementioned unfilled fill-in box or the filled fill-in box. Alternatively, the target category of the second bounding box can be the aforementioned input box or drop-down box. Alternatively, the target category of the second bounding box can be the aforementioned fill-in box.

[0436] For example, when the multiple bounding boxes include a first bounding box, a second bounding box, and a third bounding box, at least two of the target categories corresponding to the first bounding box, the second bounding box, and the third bounding box are different.

[0437] S2503, according to the multiple detection results, pairing the multiple targets of the table to obtain one or more pairing results, the pairing result including a set of paired target positions and target category confidence, and the set of paired targets including multiple targets having a pairing relationship.

[0438] The electronic device can obtain all the paired bounding boxes in the multiple detection results. Among the all paired bounding boxes, there can be a target in the table corresponding to multiple sets of paired bounding boxes. Therefore, the electronic device can determine, by the NMS algorithm module 2413, a set of paired bounding boxes corresponding to each target in the table from all the paired bounding boxes according to the multiple detection results.

[0439] Therefore, the electronic device can determine a set of paired bounding boxes corresponding to each target in the table. In this way, the electronic device can determine which targets in the table have a pairing relationship. Among the two targets having a pairing relationship, the two targets are referred to as paired targets.

[0440] The electronic device can also determine the relevant feature information of each target in the table. The relevant feature information of the target can include the position of the target and the target category of the target.

[0441] For example, in the set of paired bounding boxes corresponding to the target 1, the first bounding box corresponds to the target 1, and the second bounding box corresponds to the target 2 having a pairing relationship with the target 1.

[0442] Therefore, the target 1 and the target 2 can be referred to as paired targets. The position of the first bounding box can be regarded as the position of the target 1, and the target category of the first bounding box can be regarded as the category of the target 1. The position of the second bounding box can be regarded as the position of the target 2, and the target category of the second bounding box can be regarded as the category of the target 2.

[0443] In summary, the electronic device can obtain the relevant feature information of one or more sets of paired targets in the table.

[0444] Taking one of the bounding boxes in each pair of bounding boxes as the target class corresponding to the table label, and the other bounding box as the target class corresponding to the input box / drop-down box as an example, a corresponding pair of results can be represented as:

[0445] The label box bbox, the class confidence of the label box, the input box / drop-down box bbox, and the class confidence of the input box / drop-down box.

[0446] The label box bbox refers to the position of the table label. The class confidence of the label box refers to the confidence of the table label. The input box / drop-down box bbox refers to the position of the input box / drop-down box. The class confidence of the input box / drop-down box refers to the confidence of the input box / drop-down box. Bbox is usually an abbreviation of bounding box, i.e., bounding box.

[0447] In some embodiments, the electronic device can store the relevant feature information of each pair of targets as a queue element in the to-be-filled queue. In the process of filling in the form, the electronic device can automatically fill in the form according to the queue elements in the to-be-filled queue and the user information in the user information library.

[0448] The table visual detection method of the embodiments of the present application can obtain a first interface through an electronic device, and can obtain a plurality of targets of a table in the first interface. The electronic device can detect the plurality of targets of the table to locate, classify, and pair each target, and then obtain a plurality of detection results, each detection result including the position of each pair of bounding boxes and the confidence of each target class, so that each target corresponds to one or more pairs of bounding boxes. The electronic device can pair the plurality of targets of the table according to the plurality of detection results to obtain one or more pair results, which can filter out the pair of bounding boxes with high repetition and low confidence, so that each target corresponds to a pair of bounding boxes. Then, the electronic device can determine one or more pairs of targets in the plurality of targets, and the electronic device can also obtain the position and class of each pair of targets according to the position of the pair of bounding boxes and the confidence of each target class, that is, the electronic device can determine which target or targets each target is paired with from the plurality of targets, as well as the position and target class of the target.

[0449] Therefore, the electronic device can obtain the position and target class of one or more pairs of targets in the table, facilitate the recommendation of the user information to be filled in by another target in the pair of targets based on one of the pair of targets in the automatic filling process, complete the process of writing while filling, or directly fill in the user information to be filled in by another target in the pair of targets in the interface, and complete the filling process at one time.

[0450] In this way, the pairing of multiple targets is realized for the table in the interface, so that the electronic device can automatically fill in various tables according to the pairing relationship of the targets, is not limited to a specific table style, can comprehensively cover all styles of tables, avoids filling errors, frequent filling and high filling requirements, and improves the filling experience and filling efficiency of the user.

[0451] Based on the description of the above embodiments, taking the representation of the position of the bounding box as an example, the coordinates of one vertex of the bounding box, the width of the bounding box, and the height of the bounding box, and a set of paired bounding boxes including the first bounding box and the second bounding box, in some embodiments, the multiple detection results obtained by S2501 are represented by the tensor O in Formula One;

[0452]

[0453] Wherein, B is the number of the first interface; N is the total number of the multiple detection results; (x1, y1, w1, h1) is the coordinates and size of one of the bounding boxes in the set of paired bounding boxes; (x2, y2, w2, h2) is the coordinates and size of the other bounding box in the set of paired bounding boxes; is the confidence of the K categories of one of the bounding boxes; is the confidence of the K target categories of the other bounding box; K is the number of target categories.

[0454] In summary, the electronic device can process one interface separately to determine the paired targets in the table included in the interface. Alternatively, the electronic device can also process multiple interfaces in batches to determine the paired targets in the table included in each interface.

[0455] Based on the description of the above embodiments, in S2502, the electronic device can combine the table visual detection module 2412 mentioned in the foregoing to locate and classify the multiple targets of the table.

[0456] In some embodiments, the electronic device can input the first interface into the table visual detection module 2412. The table visual detection module 2412 can locate, classify, and pair the targets of the first interface, and output multiple detection results.

[0457] In combination with the multi-task loss function L, the training sample can be used to train the table visual detection module 2412, so that the table visual detection module 2412 can take into account the loss caused by positioning, classification, and pairing, and can more accurately locate, classify, and pair the targets in the table.

[0458] Wherein, the multi-task loss function L can be represented by Formula Two:

[0459] L = aLcls + βL bbox Equation two;

[0460] wherein, L cls is a classification loss function, the classification loss function is used to represent the accuracy of the target class of the bounding box; L bbox is a paired bounding box regression loss function, the paired bounding box regression loss function is used to represent the accuracy of the paired bounding box; and α and β are weight coefficients, which are used to balance the influence of the classification loss function and the paired bounding box regression loss function.

[0461] In summary, the multi-task loss function L can include a weighted sum of the classification loss function L cls and the paired bounding box regression loss function L bbox Thus, the performance of classifying, locating and pairing each target in the table is optimized.

[0462] In some embodiments, the classification loss function can use the classification loss function L cls to measure the accuracy of classifying and locating each target in the table.

[0463] wherein, the cross-entropy loss function can be represented by equation three;

[0464]

[0465] wherein, N is the total number of samples; K is the number of target classes; y i,1,k and y i,2,k are the real labels of the paired bounding box of the i-th sample; p i,1,k and p i,2,k are the probabilities that the i-th sample belongs to the target class k; and γ is a penalty coefficient for weighting the loss of the paired bounding box.

[0466] In some embodiments, the bounding box regression loss function L bbox can use the smooth L1 loss function to measure the accuracy of pairing each target in the table.

[0467] wherein, the paired bounding box regression loss function L bbox can be represented by equation four;

[0468]

[0469] wherein, N is the total number of samples; is the real paired bounding box of the i-th sample; is the predicted paired bounding box; and the smooth L1 loss function is defined as follows:

[0470]

[0471] Based on the description of the above embodiments, in S2503, the NMS algorithm module 2413 in the electronic device can employ various ways to effectively process the paired bounding boxes output by the table visual detection module 2412, filter out duplicate and low-quality paired bounding boxes, and obtain one or more pairing results.

[0472] Next, combined with Figure 26B and Figure 26A , the detailed process of filtering out duplicate and low-quality paired bounding boxes according to the confidence and overlap of the bounding box will be specifically introduced. The confidence of the bounding box refers to the probability that the target corresponding to the bounding box is a target of a certain target category. The overlap of the bounding box refers to the degree or area of overlap in position with other bounding boxes.

[0473] Please refer to Figure 26B and Figure 26A , Figure 26B a flowchart of a table visual detection method provided by an embodiment of the present application, Figure 27A a schematic diagram of the intersection over union provided by an embodiment of the present application. As Figure 27A shown, the table visual detection method of the present application can include:

[0474] S2601, determining the weighted confidence and weighted overlap of all paired bounding boxes in the plurality of detection results.

[0475] The electronic device can obtain the relevant feature information of all paired bounding boxes in the plurality of detection results. For any group of paired bounding boxes, the electronic device can determine the confidence of each target type of each bounding box in the group of paired bounding boxes. Thus, the electronic device can perform weighted processing on the confidence of each target type of all bounding boxes in each group of paired bounding boxes to obtain the weighted confidence of each group of paired bounding boxes.

[0476] In some embodiments, taking a group of paired bounding boxes including two bounding boxes as an example, the weighted confidence ω of each group of paired bounding boxes can be represented by Formula Five;

[0477]

[0478] where λ is a weighting coefficient; λ is used to control the contribution of one of the paired bounding boxes in the confidence; and 1-λ is used to control the contribution of the other paired bounding box.

[0479] For any group of paired bounding boxes, the electronic device can determine the position of each bounding box in the group of paired bounding boxes. Thus, the electronic device can perform weighted processing on the positions of all bounding boxes in each group of paired bounding boxes to obtain the weighted overlap of each group of paired bounding boxes.

[0480] In some embodiments, taking an example of a set of paired bounding boxes including two bounding boxes, the weighted overlap degree IoUφ of each set of paired bounding boxes can be represented by Formula Six.

[0481]

[0482] wherein, is the weighted intersection over union IoU value of one of the paired bounding boxes B1 and the first reference bounding box ; and is the weighted intersection over union IoU value of the other of the paired bounding boxes B2 and the second reference bounding box ; μ is a weighted coefficient; and μ is used to control the contribution of one of the bounding boxes B1 to the weighted intersection over union IoU value; and 1-μ controls the contribution of the other of the bounding boxes B2 to the weighted intersection over union IoU value.

[0483] In the process of calculating the weighted overlap degree, the concept of intersection over union (IoU) is used. The concept of IoU originates from mathematics in the field of sets and is used to describe the relationship between two sets A and B. It is equal to the number of elements contained in the intersection of the two sets, divided by the number of elements contained in the union of the two sets. The specific calculation formula is as follows:

[0484]

[0485] The concept of intersection over union (IoU) is used by the electronic device to describe the overlap or coincidence between two bounding boxes. Among them, the two bounding boxes can be regarded as two sets of pixels. The IoU of the two bounding boxes is equal to the area of the overlapping part of the two bounding boxes divided by the area of the combined two bounding boxes.

[0486] For example, as shown in FIG. 1, the intersection area 1 in the intersection is the overlapping area of the two bounding boxes, and the diagonal area 2 in the union is the combined area of the two bounding boxes. The area of the intersection area 1 divided by the area of the diagonal area 2 can obtain the intersection over union between the two bounding boxes. Figure 27A

[0487] S2602, according to the weighted confidence and the weighted overlap degree of all paired bounding boxes, determining a first detection result from the plurality of detection results, the first detection result including one or more detection results, and the targets corresponding to the paired bounding boxes in different detection results being different.

[0488] ​The electronic device filters out repeated and low-quality paired bounding boxes from the multiple detection results according to the weighted confidence and the weighted overlap degree of all paired bounding boxes, and selects paired bounding boxes with high confidence and low overlap degree, so as to ensure that each target corresponds to a group of paired bounding boxes. Therefore, the electronic device can determine the first detection result from the multiple detection results.

[0489] S2603, according to each detection result in the first detection result, determining the position of each group of paired targets in the multiple targets as the position of a group of paired bounding boxes in the corresponding detection result, and the confidence of the target category of each group of paired targets as the target category with the highest confidence of a group of paired bounding boxes in the corresponding detection result.

[0490] According to the first detection result, the electronic device can determine each group of paired targets in the multiple targets and the related feature information of each group of paired targets.

[0491] For any one detection result, the position of each group of paired targets is the position of a group of paired bounding boxes in the detection result, that is, the position of each target in each group of paired targets is the position of the corresponding bounding box in the group of paired bounding boxes. The target category of each group of paired targets is the target category with the highest confidence of a group of paired bounding boxes in the detection result, that is, the position of each target in each group of paired targets is the target category with the highest confidence of the corresponding bounding box in the group of paired bounding boxes.

[0492] In summary, the electronic device can effectively process the paired bounding boxes output by the table visual detection module 2412, select paired bounding boxes with high confidence and low overlap degree, and remove repeated and low-quality paired bounding boxes, so that each target in the table can correspond to a group of paired bounding boxes.

[0493] Based on this, in S2602, the electronic device can first consider the confidence and then consider the overlap. Alternatively, the electronic device can first consider the overlap and then consider the confidence. Alternatively, the electronic device can consider the confidence and the overlap at the same time. Therefore, the electronic device can obtain the first detection result.

[0494] Taking the selection principle of first considering the confidence and then considering the overlap as an example, please refer to Figure 27B , Figure 27B A flowchart of a table visual detection method provided by an embodiment of the present application. As shown in Figure 27B , the table visual detection method of the embodiment of the present application can include:

[0495] S2701, the electronic device calculates the weighted confidence ω of each group of paired bounding boxes.

[0496] S2702, the electronic device calculates the weighted overlap degree IoUφ of each group of paired bounding boxes.

[0497] S2703, the electronic device ranks all the paired bounding boxes according to the weighted confidence ω of each group of paired bounding boxes.

[0498] The ranking manner of all the paired bounding boxes can be in descending order of the weighted confidence or in ascending order of the weighted confidence, and the embodiments of the present application do not limit this.

[0499] S2704, the electronic device performs NMS operation, and can filter out paired bounding boxes with high confidence and low overlap, to obtain a first detection result.

[0500] Next, combined with Figure 27B , the specific implementation process of the electronic device performing S2703 and S2704 will be described in detail.

[0501] Please refer to Figure 28 , Figure 28 for a flowchart of a table visual detection method provided by the embodiments of the present application. As shown in Figure 29 , the table visual detection method of the embodiments of the present application can include:

[0502] S2801, all the paired bounding boxes are ranked according to the order from large to small of the weighted confidence.

[0503] The electronic device can rank all the paired bounding boxes according to the order from large to small of the weighted confidence. . Among them, the weighted confidence of the ranked paired bounding boxes satisfies the following conditions:

[0504]

[0505] S2802, all the paired bounding boxes are ranked according to the order from large to small of the weighted confidence.

[0506] Among the ranked all paired bounding boxes, the electronic device compares whether the difference between the weighted overlap of the second group of paired bounding boxes and the first group of paired bounding boxes is less than the first threshold value.

[0507] ​If the difference between the weighted overlap of the second set of paired bounding boxes and the first set of paired bounding boxes is greater than or equal to a first threshold, the second set of paired bounding boxes is deleted. If the difference between the weighted overlap of the second set of paired bounding boxes and the first set of paired bounding boxes is less than the first threshold, the first set of paired bounding boxes is deleted, and the second set of paired bounding boxes is updated to match the first set of paired bounding boxes.

[0508] Next, the electronic device compares whether the difference between the weighted overlap of the third set of paired bounding boxes and the first set of paired bounding boxes is less than a first threshold. The comparison process of the weighted overlap between the third set of paired bounding boxes and the first set of paired bounding boxes can be found in the description of the weighted overlap between the second set of paired bounding boxes and the first set of paired bounding boxes, and will not be repeated here.

[0509] By analogy, the electronic device can iterate through and compare the last pair of paired bounding boxes in all sorted pairs. Thus, the electronic device can determine the first detection result as the final selected paired bounding boxes.

[0510] In one specific embodiment, NMS operation may include the following process:

[0511] An electronic device can initialize a list of results for a control. Results list Used to store the paired bounding boxes for the final filtering.

[0512] Next, the electronic device sorts all the paired bounding boxes. Perform iterations:

[0513] If the results list If empty, the electronic device will set the pairing bounding box. Add to results list middle.

[0514] If the results list If not empty, then for each pair of bounding boxes Electronic devices can calculate bounding boxes and bounding box Do not match the results list The weighted IoU of the selected bounding boxes, and whether the weighted IoU is less than the first threshold, can be represented by the following expression:

[0515]

[0516] if The electronic device will then set the pairing bounding box Add to results list middle.

[0517] Among them, IoU threshis a first threshold value.

[0518] In summary, the electronic device can screen out the paired bounding boxes with high confidence and low overlap, remove the repeated and low-quality paired bounding boxes, ensure that the paired bounding boxes in the first detection result do not overlap excessively in the same target category, and have the highest weighted confidence, so that each target in the table can correspond to a group of paired bounding boxes.

[0519] Based on the above exemplary information processing method, during the process in which the user uses any application program of the electronic device, the electronic device can display a to-be-entered interface of the application program. The to-be-entered interface is the first interface mentioned above. The to-be-entered interface is used to display information to be filled in by the user, that is, the to-be-entered interface is an interface in which the user information needs to be entered in the application program. The to-be-entered interface can include a plurality of filling controls and name controls corresponding to the filling controls, that is, the name controls correspond one-to-one to the filling controls. The filling control is used to display the user information filled or selected by the user. The name control is the table label mentioned above. The name control is used to display the information name to which the filling control belongs. Then, in response to the filling operation of the user on any filling control in the to-be-entered interface, the electronic device displays the filling content of the user. That is, the user needs to fill in the user information in the to-be-entered interface one by one to complete the information entry.

[0520] For example, the above-mentioned application program can be an application program (such as a shopping APP) installed on the electronic device, can be a web version of the application program, can be a small program in the application program, and the like, and the specific implementation is not limited. The web version of the application program can be an application found in the browser. The small program in the application program can be an application that can be used without downloading and installing. The information name to which the above-mentioned filling control belongs can include at least one of a user name, a name, a login password, a gender, a certificate type, a contact method, and a delivery address.

[0521] However, if the user needs to fill in the same information on different interfaces (such as account registration interfaces of different application programs), that is, needs to input the repeated user information multiple times, it can cause the information filling process to be more cumbersome, waste a long filling time, and make the user experience poor.

[0522] Therefore, in order to simplify the information filling process and improve the information filling efficiency, the embodiment of the present application provides an information input method. In the method, the electronic device displays a to-be-input interface, wherein the to-be-input interface includes a plurality of filling controls and name controls corresponding to the filling controls. Then, in response to a triggering operation of a user on any filling control in the to-be-input interface, the electronic device acquires a label name of the name control corresponding to the triggered filling control. Then, the electronic device selects candidate user information corresponding to the label name of the name control from a user information library. The user information library includes a plurality of user information, which is collected by the electronic device from a plurality of application programs. Then, the electronic device displays the candidate user information in the to-be-input interface.

[0523] In the embodiment of the present application, after detecting the triggering operation of the user on any filling control in the to-be-input interface, the electronic device can select the corresponding candidate user information from the user information library according to the label name of the name control corresponding to the triggered filling control, and display the candidate user information in the to-be-input interface. In this way, intelligent input of information can be realized. The user only needs to select the corresponding user information for filling according to the actual demand, without manually filling the user information, thereby simplifying the information filling process, improving the filling efficiency of the user information, and improving the user experience. In addition, since the candidate user information is the user information collected by the electronic device from a plurality of application programs, that is, the electronic device can select the corresponding candidate user information from all the user information collected before, and is not limited to the user information collected by the application to which the to-be-input interface belongs, the comprehensive display of the user information can be realized, the situation that the user repeatedly fills the same user information in different applications is reduced, and the convenience condition for the user to quickly input information is provided.

[0524] For example, the electronic device described above can be a mobile phone, a tablet, a desktop, a laptop, a handheld computer, a notebook computer, an ultra-mobile personal computer (UMPC), a netbook, a cellular phone, a personal digital assistant (PDA), an augmented reality (AR) \ virtual reality (VR) device, and other devices capable of installing and using an application program that requires input of user information. The specific form of the electronic device is not specially limited in the embodiment of the present application.

[0525] Figure 29 FIG. 28 is a schematic diagram of a hardware structure of an electronic device 2800 provided by the embodiment of the present application, as shown in Figure 29As shown, the electronic device 2800 can include a processor 2810, an external memory interface 2820, an internal memory 2821, a universal serial bus (USB) interface 2830, a charge management module 2840, a power management module 2841, a battery 2842, an antenna 1, an antenna 2, a mobile communication module 2850, a wireless communication module 2860, an audio module 2870, a speaker 2870A, a receiver 2870B, a microphone 2870C, a headset jack 2870D, a sensor 2880, a key 2890, a motor 2891, an indicator 2892, a camera 2893, a display screen 2894, and a subscriber identification module (SIM) card interface 2895, etc.

[0526] It can be understood that the structure shown in the embodiments of the present application does not constitute a specific limitation on the electronic device 2800. In some other embodiments of the present application, the electronic device 2800 can include more or fewer components than those shown, or combine certain components, or split certain components, or different arrangement of components. The components shown can be implemented in hardware, software, or a combination of software and hardware.

[0527] The processor 2810 can include one or more processing units, for example: the processor 2810 can include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Among them, different processing units can be independent devices, or can be integrated in one or more processors.

[0528] In some embodiments, the electronic device 2800 can complete the information input method provided by the present application through the processor 2810.

[0529] The wireless communication function of the electronic device 2800 can be realized by the antenna 1, the antenna 2, the mobile communication module 2850, the wireless communication module 2860, the modem processor, and the baseband processor, etc.

[0530] The electronic device 2800 implements a display function through a GPU, a display screen 2894, and an application processor, etc. The GPU is a microprocessor for image processing, connected to the display screen 2894 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 2810 can include one or more GPUs that execute program instructions to generate or change display information.

[0531] The display screen 2894 is used to display images, videos, etc. In some embodiments, the electronic device 2800 can display a to-be-entered interface through the display screen 2894, and display candidate user information corresponding to any fill-in control in the to-be-entered interface.

[0532] The electronic device 2800 can implement a shooting function through an ISP, a camera 2893, a video codec, a GPU, a display screen 2894, and an application processor, etc.

[0533] The electronic device 2800 can implement an audio function through an audio module 2870, a speaker 2870A, a receiver 2870B, a microphone 2870C, an earphone interface 2870D, and an application processor, etc. For example, music playing, recording, etc. In some embodiments, the electronic device 2800 can collect a voice instruction input by a user through the microphone 2870C. Then, the electronic device 2800 can determine an information entry mode of the to-be-entered interface according to the voice instruction. The information entry mode includes a one-key entry mode and a separate entry mode. The one-key entry mode is used to represent that the electronic device 2800 displays candidate user information corresponding to all fill-in controls at one time. The separate entry mode is used to represent that the electronic device 2800 displays candidate user information corresponding to a fill-in control triggered by a user according to the fill-in control.

[0534] The keys 2890 can include a power key, a volume key, etc. The keys 2890 can be mechanical keys. They can also be touch keys. The electronic device 2800 can receive key inputs, generate key signal inputs related to user settings and function control of the electronic device. The motor 2891 can generate a vibration prompt. The motor 2891 can be used for incoming call vibration prompts, and can also be used for touch vibration feedback. The indicator 2892 can be an indicator light, which can be used to indicate a charging state, a power change, and can also be used to indicate a message, a missed call, a notification, etc. The SIM card interface 2895 is used to connect a SIM card. The SIM card can be inserted into or pulled out of the SIM card interface 2895 to realize contact and separation with the electronic device.

[0535] For example, the software system of the aforementioned electronic device 2800 can adopt a layered architecture, event-driven architecture, microkernel architecture, microservice architecture, or cloud architecture. This application embodiment uses a layered Android system as an example to illustrate the software structure of the electronic device 2800.

[0536] Figure 29 This is a software structure block diagram of an electronic device 2800 according to an embodiment of this application.

[0537] A layered architecture divides software into several layers, each with a clear role and function. Layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into four layers, from top to bottom: the application layer (application layer), the application framework layer (framework layer), the Android runtime and system libraries, and the kernel layer (or driver layer). The application layer can include a series of application packages.

[0538] like Figure 30 As shown, the application package may include SMS, gallery, notes, form filling assistant, information management module, function settings module, etc.

[0539] The form-filling assistant described above is used to determine the name control corresponding to the triggered fill-in control in the input interface. In some embodiments, in response to a user's triggering operation on any fill-in control in the input interface, the form-filling assistant determines the name control corresponding to the triggered fill-in control. The input interface includes multiple fill-in controls and their corresponding name controls. That is, there is a one-to-one correspondence between the fill-in control and the name control.

[0540] The aforementioned information management module is used to receive user-defined operations on user information in the user information database. These operations include information query operations and / or information adjustment operations. The information query operation causes the electronic device to display the user information the user wants to query. The information adjustment operation causes the electronic device to adjust the user information in the user information database. For example, the information adjustment operation could be a deletion operation on any user's information in the user information database, a modification operation on any user's information in the user information database, or an addition operation on information in the user information database, etc., and is not specifically limited.

[0541] The function setting module is configured to set the collection source (i.e., the preset collection application) of the user information in the user information library. In some embodiments, if the switch control with the automatic collection function in the function setting interface (or the information collection setting interface) is turned on by the user, it indicates that the electronic device is allowed to collect the user information saved by all the applications, and the function setting module can collect the user information saved by any application in the electronic device. In other embodiments, if the switch control with the automatic collection function in the function setting interface is not turned on by the user, and any application belongs to the switch control turned on by the user, the function setting module can collect the user information saved by the application corresponding to the turned-on control.

[0542] The application framework layer provides an application programming interface (API) and a programming framework for the applications in the application layer. The application framework layer includes some pre-defined functions.

[0543] As shown in Figure 31 , the application framework layer can include a window manager (WMS), a content provider, a view system, an image memory module, etc.

[0544] The window manager is configured to manage the window program. The window manager can acquire the size of the display screen, determine whether there is a status bar, lock the screen, and capture the screen, etc.

[0545] The content provider is configured to store and acquire data, and make the data accessible by the applications. The data can include videos, images, audios, dialed and received calls, browsing history and bookmarks, phone books, etc.

[0546] The view system includes visual controls, such as a control for displaying text, a control for displaying pictures, etc. The view system can be used to build an application. A display interface can be composed of one or more views. For example, a display interface including a short message notification icon can include a view for displaying text and a view for displaying pictures.

[0547] The image memory module is configured to determine at least one candidate user information matched with a target label name. The target label name is obtained by standardizing a label name of a name control corresponding to the triggered fill-in control. In some embodiments, upon receiving the information query indication sent by the form-filling assistant, the image memory module can standardize the label name of the name control corresponding to the triggered fill-in control to obtain the target label name. Then, the image memory module can select at least one user information matched with the target label name from the user information library, and display the at least one candidate user information in the to-be-entered interface for the user to select.

[0548] Optionally, after obtaining the at least one candidate user information, the image memory module can sort the at least one candidate user information according to a preset display rule to obtain an information candidate list. The preset display rule can include at least one of an order of information update time, a descending order of information entry times, and an association with historical user information. The information update time refers to the time when the candidate user information is added to the user information library. The information entry times refer to the number of times the candidate user information is matched, i.e., the number of times the candidate user information is selected by the user. The historical user information refers to the user information that has been entered in the to-be-entered interface. Then, the image memory module can send the information candidate list to the form-filling assistant, so that the electronic device displays the candidate user information in the information candidate list in the order.

[0549] The Android runtime includes a core library and a virtual machine. The Android runtime is responsible for scheduling and managing the Android system. The core library contains two parts: one part is the function function that the java language needs to call, and the other part is the core library of Android.

[0550] The application layer and the application framework layer run in the virtual machine. The virtual machine executes the java files of the application layer and the application framework layer into binary files. The virtual machine is used to manage the life cycle of objects, stack management, thread management, security and exception management, and garbage collection.

[0551] The system library can include a plurality of functional modules. For example: surface manager, three-dimensional graphics processing library (for example: openGL ES), two-dimensional graphics engine (for example: SGL), media library (media libraries) and the like.

[0552] The surface manager is used to manage the display subsystem and provides a fusion of 2D and 3D layers for multiple applications. The 3D graphics processing library is used to implement 3D graphics drawing, image rendering, composition, and layer processing. The 2D graphics engine is a drawing engine for 2D drawing. The media library supports playback and recording of various commonly used audio, video formats, and static image files. The media library can support various audio and video encoding formats, such as: MPEG4, H.264, MP3, AAC, AMR, JPG, PNG, etc.

[0553] The kernel layer is a layer between hardware and software. The kernel layer at least includes display drivers, camera drivers, audio drivers, sensor drivers, etc.

[0554] For example, the software structure of the electronic device 2800 described above can also include a hardware abstraction layer. The hardware abstraction layer is a packaging of the Linux kernel driver, which provides an interface upward, which hides the hardware interface details of the specific platform, and provides a virtual hardware platform for the operating system. In the embodiments of the present application, the hardware abstraction layer can include camera HAL, audio HAL, GPS HAL, Wi-Fi HAL, etc.

[0555] It can be understood that, Figure 31 The layers in the structure shown and the components included in each layer do not constitute a specific limitation on the electronic device 2800, i.e., the mobile phone. In other embodiments of the present application, the structure can include more or fewer layers than shown, and each layer can include more or fewer components, which are not limited by the present application.

[0556] Next, based on the foregoing example of the information processing method, in the automatic form filling process or the form automatic filling process, the content to be filled can be obtained from the following user information library to be displayed in the electronic device to be entered interface. That is, the user information in the user information library can be used as the source of the content to be filled in the automatic filling process.

[0557] Next, the specific implementation of the foregoing process is described in detail. For example, the present application provides an information entry method. The method can enter the user information in the user information library into the to-be-entered interface. Next, taking a mobile phone as an example, the method of the present application is described. Specifically, as shown in the figure, Figure 31 The information entry method can include S3001-S3014.

[0558] S3001, the form filling assistant in the mobile phone receives a trigger operation of the user for any filling control in the to-be-entered interface.

[0559] The to-be-entered interface refers to an interface in an application program that needs to enter user information. The to-be-entered interface can include a plurality of fill-in controls and name controls corresponding to the fill-in controls, that is, the name controls correspond one-to-one to the fill-in controls. The fill-in control is used to display user information filled or selected by the user. The name control is used to display the information name to which the fill-in control belongs. For example, as shown in Figure 32 The to-be-entered interface displayed by the mobile phone is an add shipping address interface (that is, interface (a) in Figure 32 The add shipping address interface can include a "recipient" control 3111, a fill-in control 3121 corresponding to the "recipient" control 3111, a "mobile phone number" control 3112, and a fill-in control 3122 corresponding to the "mobile phone number" control 3112. The "recipient" control 3111 and the "mobile phone number" control 3112 are both name controls.

[0560] For example, the trigger operation refers to an operation performed by the user's finger touching the display screen of the mobile phone. The trigger operation can be a single-click operation on the fill-in control, or a double-click operation on the fill-in control, and the like, which is not limited.

[0561] It should be noted that the application program to which the to-be-entered interface belongs can be an application program installed on the mobile phone, a web-based application program, or a small program in any application program on the mobile phone, and the like, which is not limited.

[0562] S3002, in response to the trigger operation of the user on any fill-in control in the to-be-entered interface, the form-filling assistant sends an information query instruction to the portrait memory module in the mobile phone.

[0563] Specifically, after receiving the trigger operation of the user on any fill-in control in the to-be-entered interface, the form-filling assistant can send an information query instruction to the portrait memory module according to the triggered fill-in control. The information query instruction carries attribute information of the name control corresponding to the triggered fill-in control.

[0564] In some embodiments, the form-filling assistant can determine the triggered fill-in control according to the trigger position of the user and the display position of the fill-in control in the to-be-entered interface. The trigger position of the user is determined based on the touch point generated when the user's finger touches the display screen of the mobile phone. Then, the form-filling assistant can determine the name control corresponding to the triggered fill-in control according to the position mapping relationship between the fill-in control and the name control in the to-be-entered interface, and obtain the attribute information of the name control.

[0565] In this embodiment, the attribute information of the name control can include a label name of the name control, an application of an interface to which the name control belongs (or an interface of an application to which the name control belongs), and a title name of the interface to which the name control belongs (or an interface title name). The label name of the name control refers to the information name displayed by the name control in the interface to be filled in. For example, if the name control is a "recipient" control, the label name is the recipient. The application of the interface refers to the application to which the interface to be filled in belongs. The interface title name is used to represent the purpose of filling in the user information in the interface to be filled in. For example, the interface title name can be the interface information displayed at the top of the interface to be filled in. For example, if the interface to be filled in shown in (a) is an interface to be filled in, the interface title name can be "add shipping address". Figure 33

[0566] S3003, the image memory module, in a case where the information query instruction sent by the form filling assistant is received, detects a label name in attribute information of a name control corresponding to the triggered fill-in control according to the attribute information carried by the information query instruction, and obtains a detection result.

[0567] Specifically, after receiving the information query instruction sent by the form filling assistant, the image memory module can detect the label name of the name control according to the attribute information of the name control corresponding to the triggered fill-in control carried by the information query instruction, to determine whether the label name can be standardized. That is, the detection result is used to indicate whether the label name of the name control can be standardized. It can be understood that the standardization refers to converting the label name into a label name stored in the user information library, so that the mobile phone can determine the candidate user information according to the label name, and thus improve the accuracy of information recommendation.

[0568] In some embodiments, the mobile phone can detect the label name by determining whether the attribute information of the name control satisfies a preset condition. That is, the preset condition is used to evaluate whether the label name can be standardized. Specifically, if the attribute information of the name control satisfies the preset condition, it means that the label name can be standardized, and the image memory module can perform S3004 to standardize the label name. If the attribute information of the name control does not satisfy the preset condition, it means that the label name cannot be standardized, and therefore the image memory module can perform S3011 to send the label name to the cloud service, to provide a basis for subsequent standardization of the label name.

[0569] ​The preset condition can include that the attribute information of the name control satisfies any mapping relationship in a preset mapping rule, and / or the similarity between the label name in the user information library and the label name of the name control is greater than a preset similarity. It should be noted that the specific judgment process of the preset condition will be described in detail below.

[0570] S3004, in the case where the detection result indicates that the label name can be standardized, the image memory module standardizes the label name to obtain a target label name.

[0571] Specifically, after the detection result indicates that the label name of the name control can be standardized, the image memory module standardizes the label name of the name control to obtain a target label name of the name control. The target label name is any label name existing in the user information library. For example, if the label name is recipient, the image memory module can standardize the recipient to name, i.e., the target label name is name. For another example, if the label name is mobile phone number, the image memory module can standardize the mobile phone number to mobile phone number, i.e., the target label name is mobile phone number.

[0572] The standardization process of the label name will be described in detail below.

[0573] S3005, the image memory module determines whether there is user information matching the target label name in the user information library.

[0574] In some embodiments, after obtaining the target label name, the image memory module can determine whether there is user information matching the target label name in the user information library. If there is user information matching the target label name in the user information library, it indicates that the user has filled in the user information of the target label name before, and therefore the image memory module can execute S3007. If there is no user information matching the target label name in the user information library, it indicates that the user has not filled in the user information of the target label name before, and therefore the image memory module can execute S3006.

[0575] In an example, the user information library can include multiple pieces of user information of the machine owner and the label name of each piece of user information. For example, taking the user information library including the gender of the machine owner as "female" as an example, the label name is gender, and the user information is female. In another example, the user information library can also include multiple pieces of user information of at least one associated user and the label name of each piece of user information. The associated user refers to a user who has a contact with the machine owner. For example, the associated user can be a relative of the machine owner (such as a child, a parent, etc.), a friend of the machine owner, a customer of the machine owner, etc., which is not limited in particular.

[0576] In one implementation, the user information in the aforementioned user information database can be pre-filled by the user in the information editing interface of the settings application. Specifically, upon detecting a user's information entry event, the phone can store the user-entered information as new user information in the user information database. This information entry event can be the user's entry operation in the information editing interface. This provides a basis for subsequent retrieval of user information.

[0577] In some embodiments, in response to a user's information entry operation on the information editing interface, the mobile phone can obtain the newly entered user information. The mobile phone can then store the newly entered user information in a user database for later retrieval. For example, such as... Figure 33 As shown, the mobile phone is displaying the personal information interface shown in interface (a). If the user clicks the add control 3201 in the personal information interface, the mobile phone can display... Figure 33 The information editing interface shown in (b) is shown below. Subsequently, in response to the user's input of the username information control 3202 in the information editing interface, the mobile phone can obtain and display the input content of the information control 3202, "Zhao Yiyi".

[0578] In another implementation, the user information in the aforementioned user information database can also be extracted by the mobile phone from a preset collection application. This preset collection application is an application with information collection capabilities. That is, before extracting user information stored by any application on the phone, the phone can determine whether the application has information collection capabilities. Only if the application has information collection capabilities can the phone extract information from that application to obtain the user information stored therein.

[0579] In some embodiments, the aforementioned preset collection application may be a mobile phone pre-configured, that is, a default setting on the mobile phone. For example, please see Figure 33 The phone is displaying Figure 34 Interface (a) shows the information collection settings interface, which includes a switch control 3301A for automatic collection, a switch control 3302 for the SMS application, a switch control 3303 for the Gallery application, and a switch control 3304 for the Notes application. It can be seen that the automatic collection switch control 3301, the SMS application switch control 3302, and the Gallery application switch control 3303 are all in the off state, while the Notes application switch control 3304 is in the on state. This means that the phone only defaults to the Notes application as the pre-set collection application.

[0580] In some embodiments, the preset collection application can be set by the user in advance. That is, the phone can determine the application in which the user turns on the information collection function as the preset collection application. For example, as shown in FIG. 32A, if the switch control 3301A of the automatic collection function in the information collection setting interface is turned on by the user, it means that the user allows the phone to extract the user information from any application in the phone. Therefore, the phone can regard all the applications installed in the phone as the preset collection application, that is, the phone can extract the user information saved by each application in the phone. Figure 34 Then, the phone can display the target setting interface shown in (b) of FIG. 32B, which includes the switch control 3301B corresponding to the automatic collection function. The switch control 3301B is used to represent that the automatic collection function is turned on. Figure 34

[0581] In the case where the switch control 3301A of the automatic collection function in the information collection setting interface is not turned on by the user, if the switch control 3302 corresponding to the SMS application in the information collection setting interface 3300 is clicked by the user, it means that the user allows the phone to extract the user information from the SMS application. Therefore, the phone can regard the SMS application as the preset collection application, that is, the phone can extract the user information saved by the SMS application.

[0582] It can be understood that if the preset collection application is the SMS application, the phone can extract the user information from the SMS saved by the SMS application. The SMS saved by the SMS application can be the SMS received by the phone or the SMS edited by the user. If the preset collection application is the gallery application, the phone can extract the user information from the picture saved by the gallery application. The picture saved by the gallery application can be obtained based on the shooting operation of the user or downloaded and cached from any application (such as a social application) in the phone. If the preset collection application is the note application, the phone can extract the user information from the note saved by the note application. The note saved by the note application is the note edited by the user. If the preset collection application is the third-party application (such as a social application), the phone can extract the user information from the information input interface of the third-party application. The information input interface can be any interface in the application that can input the user information. The information input interface can be an account registration interface, an interface for adding travel information, an interface for adding a shipping address, etc., which is not limited.

[0583] ​In some embodiments, the process of the mobile phone extracting user information from different applications can be automated. For example, taking a gallery application as a preset collection application, when a new image is detected in the gallery application, the mobile phone can extract information from the new image to obtain new user information and add it to the user information database. This achieves automatic information extraction without requiring user intervention, simplifying the process and facilitating the subsequent display of candidate user information.

[0584] In some embodiments, the process of the mobile phone extracting user information from different applications can also be achieved by the mobile phone extracting information based on the user's information extraction operation. Specifically, in response to the user's information extraction operation on the display interface of a preset collection application, the mobile phone can extract information from the display interface to obtain new user information. The display interface may include at least one piece of user information. The information extraction operation can be a user's click operation on an information extraction control, or a voice command from the user regarding information extraction, etc., and is not specifically limited. For example, the voice command could be to extract user information. Afterwards, the mobile phone can add the new user information to the user information database. This allows for accurate extraction of user information, confirms with the user before extraction, reduces the risk of information security issues due to excessive extraction, ensures the security of the extraction environment, and provides convenient conditions for subsequently displaying candidate user information.

[0585] For example, such as Figure 34 As shown, taking the aforementioned preset collection application as an example, the phone is displaying... Figure 34 The image display interface shown in (a) includes an ID card image 3410. Subsequently, in response to the user's wake-up operation for the voice assistant, the phone can display as shown... Figure 34 The interface shown in (b) displays a "Save to Personal Information" control, a "Copy ID Number" control, a "Copy Address" control, and a voice assistant prompt card 3420 overlaid on the image display interface. The wake-up operation can be initiated by the user uttering a wake-up phrase for the voice assistant, such as "Hello Youyou." The "Save to Personal Information" control is used to extract user information from the ID card image 3410 and save it to the user information database. It can be understood that the types of controls overlaid on the phone are determined by the content displayed on the phone when the voice assistant is activated.

[0586] After that, the user can input the dialogue content in the case that the voice assistant is in the wake-up state. The dialogue content can be in the form of voice, such as inputting voice "save user information" or the like. Alternatively, the dialogue content can also be in the form of text, such as inputting the text content "save user information" or the like in the prompt card 3420 of the voice assistant. After that, in the case that the dialogue content input by the user is detected, the mobile phone can perform an operation corresponding to the dialogue content, that is, save the user information in the ID card picture 3410 to the user information library. Alternatively, in response to the click operation of the user on the "save to personal information" control, the mobile phone can save the user information in the ID card picture 3410 to the user information library.

[0587] In an implementation manner, the mobile phone needs to perform identity authentication on the user before performing information extraction on the display interface, that is, only in the case that the identity authentication is successful (that is, the user is the machine owner), the mobile phone can perform information extraction on the display interface. Specifically, the mobile phone can perform identity authentication on the user to obtain an identity authentication result. The identity authentication result is used to indicate whether the user is the machine owner. After that, in the case that the identity authentication result indicates that the user is the machine owner, the mobile phone can perform information extraction on the display interface to obtain new user information. In the case that the identity authentication result indicates that the user is not the machine owner, that is, the user is a non-machine owner, the mobile phone does not perform the information extraction operation, that is, does not perform information extraction on the display interface. In this way, the case that the non-machine owner extracts user information can be reduced, the security of information extraction is improved, and the use experience of the user is improved.

[0588] The mobile phone can perform identity authentication on the user by face recognition, or perform identity authentication on the user by fingerprint recognition, or perform identity authentication on the user by inputting a password, and the like, which is not limited. For example, please refer to Figure 34 If the "save to personal information" control in the (b) interface is clicked by the user, the mobile phone can display the security verification interface shown in (c) of Figure 34 to perform identity authentication on the user. The security verification interface is a fingerprint recognition interface. After that, in the case that the identity authentication of the user is successful, the mobile phone can save the user information in the ID card picture 3410 to the user information library.

[0589] In another implementation, upon obtaining new user information, the mobile phone can retrieve multiple user information entries corresponding to the user to whom the new user information belongs from the user information database. Then, the mobile phone can determine whether user information with that tag name exists among the multiple user information entries based on the tag name corresponding to the new user information. If the tag name does not exist among the multiple user information entries, it means that the user information database does not store user information with that tag name. Therefore, the mobile phone can save the new user information to the user information database and display the personal information interface. This personal information interface includes the new user information. This allows for accurate saving of user information, reducing the occurrence of missed or incorrectly saved user information, and providing convenient conditions for timely retrieval of user information later.

[0590] For example, such as Figure 34 As shown, the phone is displaying... Figure 34 In interface (c), if the user's identity authentication is successful but the user information in ID card image 3410 is not stored in the user information database, the mobile phone can display the personal information interface of the user to whom ID card image 3410 belongs, that is, display... Figure 34 The (d) interface is shown in the image. This personal information interface includes the newly added ID card (i.e., ID card 3411). Optionally, if the user clicks on ID card 3411, it indicates that the user wants to view an image of the ID card. The phone can then display an image of ID card 3410 for the user to view and verify the information.

[0591] If user information with the tag name exists among the multiple user information entries, it means that user information with the tag name has already been stored in the user information database. Therefore, in order to improve the accuracy of information storage in the user information database, the mobile phone can use the user information with the tag name among the multiple user information entries as the original user information and determine whether the original user information is the same as the new user information. If the original user information is the same as the new user information, it means that the new user information has been saved to the user information database. Therefore, the mobile phone does not need to save the new user information again, so as to avoid wasting too much storage space due to duplicate saving of user information and improve the utilization rate of storage resources.

[0592] If the original user information differs from the new user information, it indicates an error in either. Therefore, the phone can display an information update interface for the user to determine whether an update is necessary. This update interface may include at least one of the following: original user information, new user information, a prompt about "updating user information," a back control, and an update control. If the user clicks the back control, it means they do not wish to update the user information, and the phone can continue to save the original information. If the user clicks the update control, it means they want to update the user information, and the phone can replace the original information with the new information in the user database. This ensures accurate user information updates, reduces errors, and improves the user experience.

[0593] For example, such as Figure 31 As shown, the phone is displaying... Figure 31 In interface (c), if the user's identity authentication is successful and the user's ID number is stored in the user information database, but the ID number stored in the user information database is different from the ID number in ID card image 3410, the mobile phone can display... Figure 31 The information update interface shown in (e) includes a prompt (i.e., "Update card information?"), the current number (i.e., the original user information), the new number (i.e., the new user information), a "Save Current Information" control (i.e., a return control), and an "Update" control. If the user clicks the "Save Current Information" control, the phone can continue to save the original ID number in the user information database. If the user clicks the "Update" control, the phone can replace the ID number in ID image 3410 with the original ID number and save it to the user information database.

[0594] S3006, if there is no user information in the user information database that matches the target tag name, the profile memory module sends a matching result instruction to the form filling assistant.

[0595] The matching result indicates that there is no user information in the user information database that matches the target tag name, meaning that the profile memory module did not find any user information that matches the tag name.

[0596] In some embodiments, when the form-filling assistant receives a matching result instruction sent by the profile memory module, the mobile phone can continue to display the input interface, that is, it does not display the recommended user information (i.e., candidate user information) on the input interface, so that the user can fill in the information for the aforementioned triggered input control.

[0597] S3007, in a case where the user information database contains user information matching the target tag name, the portrait memory module determines the user information matching the target tag name as candidate user information.

[0598] Specifically, in a case where the user information database contains user information matching the target tag name, the portrait memory module can display the user information matching the target tag name as candidate user information in the to-be-entered interface.

[0599] In some embodiments, the portrait memory module can determine the number of information of the candidate user information. In a case where the number of information of the candidate user information is 1, the portrait memory module can directly execute S3008. In a case where the number of information of the candidate user information is not 1, that is, the user information database contains multiple user information matching the target tag name, the portrait memory module can sort the multiple candidate user information according to a preset display rule to obtain an information candidate list. The information candidate list is used to represent the arrangement order of the multiple candidate user information.

[0600] The preset display rule can include at least one of the following: the order of information update time, the order of information entry times, and association with historical user information. The information update time refers to the time when the candidate user information is added to the user information database. The information entry times refer to the number of times the candidate user information is selected by the user. The historical user information refers to the user information that has been entered in the to-be-entered interface.

[0601] In an implementation manner, the portrait memory module can preferentially place the candidate user information close to the information filling time in the front of the information candidate list, and place the candidate user information far from the information filling time in the back of the information candidate list. Specifically, the portrait memory module can obtain the information update time of each candidate user information. Then, the portrait memory module can sort the multiple candidate user information according to the order of information update time to obtain the information candidate list. For example, the candidate user information is A user, B user and C user, and the information update time of A user is T1, the information update time of B user is T2, and the information update time of C user is T3. T1, T2 and T3 include but are not limited to year, month, date, clock and minute, etc. Therefore, the portrait memory module can place C user in the first position of the information candidate list, place B user in the second position of the information candidate list, and place A user in the third position of the information candidate list, that is, the information candidate list is C user, B user and A user.

[0602] In another implementation manner, the image memory module can preferentially place the candidate user information with a large number of information query times in front of the information candidate list, and place the candidate user information with a small number of information query times at the back of the information candidate list. Specifically, the image memory module can obtain the information query times of each candidate user information. Then, the image memory module can sort the plurality of candidate user information in descending order of the information query times to obtain the information candidate list. For example, the candidate user information are A user, B user and C user respectively, and the information query times of A user is 10, the information query times of B user is 5, and the information query times of C user is 1. Therefore, the image memory module can place A user at the first position of the information candidate list, place B user at the second position of the information candidate list, and place C user at the third position of the information candidate list, that is, the information candidate list is A user, B user and C user.

[0603] In another implementation manner, the image memory module can preferentially place the candidate user information associated with the historical user information in front of the information candidate list, and place the candidate user information not associated with the historical user information at the back of the information candidate list. Specifically, the image memory module can obtain the associated user information of each candidate user information in the user information library. Then, the image memory module can sequentially determine whether the associated user information of the historical user information is the same as the associated user information of the candidate user information. If the associated user information of the historical user information is the same as the associated user information of any candidate user information, the image memory module can place the candidate user information at the first position of the information candidate list. For example, the candidate user information are the mobile phone number “189****4045” of A user and the mobile phone number “189****4123” of C user, and the historical user information is A user. It can be seen that the image memory module can determine that the mobile phone number “189****4045” is the candidate user information associated with the historical user information, and therefore the image memory module can place the mobile phone number “189****4045” at the first position of the information candidate list and place the mobile phone number “189****4123” at the second position of the information candidate list, that is, the information candidate list is A user and C user.

[0604] It should be noted that the historical user information described above can be carried by the information query indication described above, that is, after receiving the trigger operation of the user on any fill-in control in the to-be-entered interface, the form filling assistant can send the information query indication carrying the historical user information and the attribute information of the name control corresponding to the triggered fill-in control to the image memory module, thereby providing a basis for subsequent accurate display of recommended information. Then, the image memory module can store the historical user information in the user information library, thereby providing a basis for subsequent calling of user information.

[0605] In some cases, before storing the historical user information into the user information library, the portrait memory module can perform mapping processing on the label name to which the historical user information belongs, to obtain a mapped label name. Then, the portrait memory module can store the mapped label name as the label name of the historical user information and the historical user information into the user information library. In this way, the label names in the user information library can all be standardized label names, that is, the label names corresponding to the same type of user information are all the same, so as to facilitate subsequent accurate calling of user information and improve the calling efficiency of user information.

[0606] It can be understood that the historical user information can have certain rules (such as the format of the mobile phone number is 11 digits, the email number contains an "@" symbol), therefore, the portrait memory module can perform mapping processing on the label name to which the historical user information belongs according to the historical user information and a preset information rule, to obtain a mapped label name. The preset information rule includes but is not limited to at least one of the following: the historical user information includes an "@" character, the total number of characters of the historical user information is 11 digits, and the number of characters of the historical user information is within a preset number interval. For example, the historical user information is "189****4045", and the label name to which the historical user information belongs is a contact method. Since the preset format of the mobile phone number is 11 digits, the portrait memory module can determine that the mapped label name to which the historical user information belongs is a mobile phone number, that is, the portrait memory module can standardize the contact method to a mobile phone number. For another example, the historical user information is "Zhao Yi Yi", and the label name to which the historical user information belongs is a sender. Since the number of characters of the name is generally 2-4, that is, the preset number interval is 2-4, the portrait memory module can determine that the mapped label name to which the historical user information belongs is a name, that is, the portrait memory module can standardize the sender to a name.

[0607] S3008, the portrait memory module sends the candidate user information to the form filling assistant.

[0608] In some embodiments, after obtaining the above-mentioned candidate user information, the portrait memory module can send the candidate user information to the form filling assistant. For example, if the number of information of the candidate user information is 1, the portrait memory module can directly send the candidate user information to the form filling assistant. For another example, if the number of information of the candidate user information is not 1, the portrait memory module can send the above-mentioned information candidate list to the form filling assistant, that is, send the arranged candidate user information to the form filling assistant.

[0609] S3009, the form filling assistant displays the candidate user information in the to-be-entered interface in the case of receiving the candidate user information sent by the portrait memory module.

[0610] In some embodiments, after receiving the candidate user information sent by the image memory module, the form-filling assistant can display the candidate user information in the to-be-entered interface. For example, if the number of pieces of candidate user information is 1, the form-filling assistant can directly display the candidate user information. For another example, if the number of pieces of candidate user information is not 1, the form-filling assistant can display the candidate user information in the to-be-entered interface in the order of arrangement in the information candidate list. In this way, the user can be facilitated to select the corresponding user information, and the user's experience can be improved.

[0611] In an implementation manner, the form-filling assistant can display the candidate user information in a suspended manner below the triggered fill-in control. For example, as shown in FIG. 13, Figure 31 the phone is displaying the interface (a) in FIG. 13, Figure 31 if the fill-in control 3121 corresponding to the "recipient" control is triggered by the user, the phone can display the candidate user information of the "recipient" in a suspended manner below the fill-in control 3121, that is, the phone can display the interface (b) in FIG. 13. Figure 31 The candidate user information of the "recipient" can include Zhao Yiyi and Zhao Sanzhi.

[0612] In another implementation manner, the form-filling assistant can display the candidate user information in a drop-down box manner on the basis of the triggered fill-in control. The drop-down box is used to enable the user to select user information from the candidate user information for filling in.

[0613] It can be understood that, in order to simplify the display content of the user information, if the number of pieces of candidate user information is not 1, the form-filling assistant can display only a preset number of pieces of candidate user information located at a front position in the information candidate list. The preset number of pieces can be preset according to actual conditions, for example, the preset number of pieces can be 1, 3, or the like, and is not limited specifically. For example, if the information candidate list includes 10 pieces of candidate user information, the form-filling assistant can obtain a preset number (for example, 3) of pieces of candidate user information located at a front position in the information candidate list, and display the preset number of pieces of candidate user information, that is, the phone can display only the first 3 pieces of candidate user information in the information candidate list. In this way, the accurate display of the user information can be implemented, the situation that the interface is cluttered due to too much display of the user information can be reduced, the neatness of the interface can be improved, and the user is facilitated to select the corresponding user information subsequently.

[0614] S3010, in response to a selection operation of the user on any piece of candidate user information, the form-filling assistant displays the selected candidate user information in the triggered fill-in control.

[0615] In some embodiments, after the candidate user information is displayed in the interface to be filled in, if any candidate user information is selected by the user, the form-filling assistant can display the selected candidate user information in the triggered fill-in control. For example, referring to Label name , the mobile phone is displaying the interface shown in (b) of Target label name , if the candidate user information "Zhao Yiyi" is clicked by the user, the mobile phone can directly display "Zhao Yiyi" in the fill-in control 3121, that is, display the interface shown in (c) of Contact . In this way, intelligent filling of user information can be achieved, and the user only needs to select the corresponding user information according to the actual needs to fill in, without manually filling in the user information, simplifying the information filling process, improving the filling efficiency of the user information, and thus improving the user's use experience.

[0616] In some embodiments, if the candidate user information displayed in the interface to be filled in does not meet the user's needs, the user can also not select any candidate user information, and the user can directly fill in the corresponding user information in the triggered fill-in control, that is, in response to the user's fill-in operation on the triggered fill-in control, the form-filling assistant displays the filled-in user information in the triggered fill-in control.

[0617] It can be understood that in the case where the triggered fill-in control displays user information, that is, displays selected candidate user information or filled-in user information, the form-filling assistant can send the user information to the portrait memory module, so that the portrait memory module stores the user information in the user information library. In this way, not only can the user information be saved in time, improving the storage efficiency of the user information, but also the mobile phone can determine the input content of the blank fill-in control in the current interface (i.e., the interface to be filled in), so that the input content is associated with the user information, that is, belongs to the user information of the same user, improving the selection efficiency of the candidate user information, and thus improving the user's use experience.

[0618] S3011, in the case where the detection result indicates that the label name cannot be standardized, the portrait memory module sends the label name to a name management module in the cloud service.

[0619] In some embodiments, after the detection result indicates that the label name cannot be standardized, the portrait memory module can send the label name to a name management module in the cloud service (or server) for subsequent standardized learning of the label name, thereby improving the standardization efficiency of the attribute name.

[0620] The cloud service can include a name management module for standardized learning of the label name of the name control.

[0621] In some embodiments, after the detection result indicates that the label name cannot be standardized, the portrait memory module can send the detection result to the form filling assistant. Then, the form filling assistant continues to display the above-mentioned to-be-entered interface, that is, does not display any candidate user information in the to-be-entered interface, in a case where the form filling assistant receives the detection result sent by the portrait memory module. Then, in response to the information entering operation of the user on the triggered fill-in control, the form filling assistant can display the entered user information in the triggered fill-in control. It can be understood that the entered user information is obtained based on the information entering operation of the user. That is, if the label name cannot be standardized, it means that the mobile phone cannot query the corresponding user information from the user information library, that is, the mobile phone cannot display the candidate user information on the to-be-entered interface, so the mobile phone can continue to display the to-be-entered interface to remind the user to manually fill in the user information.

[0622] Then, the form filling assistant can send the above-mentioned entered user information to the portrait memory module as historical user information. Then, the portrait memory module, in a case where the portrait memory module receives the historical user information sent by the form filling assistant, maps the label name of the historical user information according to the historical user information and the above-mentioned preset information rule, to obtain a mapped label name. Then, the portrait memory module can store the mapped label name as the label name of the historical user information and the historical user information in the user information library. In this way, the label names in the user information library are all standardized label names, that is, the label names corresponding to the same type of user information are all the same, so as to facilitate subsequent accurate calling of user information and improve the calling efficiency of user information.

[0623] S3012, the name management module, in a case where the name management module receives the label name sent by the portrait memory module, standardizes the label name to obtain a target attribute name.

[0624] Specifically, after the name management module receives the label name sent by the portrait memory module, the name management module can save the label name. Then, the name management module can standardize the label name every interval of a preset time to obtain a target label name. In this way, the accuracy of determining the target label name can be improved, thereby providing a basis for subsequent determination of the target label name.

[0625] In some embodiments, the operation and maintenance personnel can standardize the above-mentioned label name to obtain a target label name. In this way, the accuracy of determining the target label name can be improved by artificial standardization, thereby providing a basis for subsequent determination of the target label name.

[0626] In some embodiments, the name management module can normalize the label name according to a name adjustment model to obtain a target label name. Specifically, the name management module can input the label name into the name adjustment model to obtain the target label name. The name adjustment model is a supervised learning algorithm. The name adjustment model is trained based on a label name set, which includes a plurality of label names and the real label name carried by each label name. That is, the name management module can input the label name into the name adjustment model to be trained to obtain a predicted label name. Then, the name management module can adjust the parameters of the name adjustment model according to the preset label name and the real label name to obtain a trained name adjustment model. In this way, the accuracy of determining the target label name can be improved by machine learning, without the need for manual standardization learning, thereby improving the standardization efficiency of the label name and providing a basis for subsequent determination of the target label name.

[0627] S3013, the name management module sends the mapping relationship between the target label name and the label name to the portrait memory module.

[0628] S3014, the portrait memory module adds the mapping relationship between the target label name and the label name to the preset mapping rule after receiving the mapping relationship between the target label name and the label name sent by the name management module.

[0629] Specifically, the portrait memory module can add the mapping relationship to the preset mapping rule after receiving the mapping relationship between the target label name and the label name sent by the name management module, so as to standardize the label name subsequently. The preset mapping rule includes a plurality of mapping relationships between the label name and the target label name.

[0630] In an implementation manner, the portrait memory module can display the preset mapping rule through a mapping table. The mapping table is used to represent the mapping relationship between the plurality of label names and the target label name.

[0631] In some embodiments, the mapping table can include a plurality of target label names and the label name corresponding to each target label name. That is, the target label name in the preset mapping rule can be determined based on the label name.

[0632] Exemplarily, as shown in Table 1, the mapping table can include a label name "contact", a target label name "name" corresponding to the label name "contact", a label name "recipient", a target label name "name" corresponding to the label name "recipient", a label name "contact number", a target label name "mobile number" corresponding to the label name "contact number", a label name "mobile number", and a target label name "mobile number" corresponding to the label name "mobile number". That is, the label name "contact" and the label name "recipient" both correspond to the target label name "name", and the label name "contact number" and the label name "mobile number" both correspond to the target label name "mobile number".

[0633] Table 1

[0634] Name Recipient Name Contact number Mobile phone number Mobile phone Mobile phone number Interface belonging application Interface title name Label name

[0635] The mapping table can further include an interface title name corresponding to each target label name and an application to which the interface belongs. That is, the target label name in the preset mapping rule can be determined based on the interface title name, the application to which the interface belongs, and the label name. The interface title name is interface information displayed at the top of the mobile display interface. It can be understood that the interface title name is used to represent the purpose of filling in user information in the mobile display interface. In this way, a basis for subsequent accurate determination of the target label name is provided.

[0636] Exemplarily, as shown in Table 2, the mapping table includes a target label name "mobile number" corresponding to an application "application 1" to which the interface belongs, an interface title name "add sender information", and a label name "contact information", a target label name "email number" corresponding to the application "application 1" to which the interface belongs, an interface title name "apply for invoice", and a label name "contact information", a target label name "file number" corresponding to an application "application 2" to which the interface belongs, an interface title name "add student file", and a label name "ID number", and a target label name "ID number" corresponding to an application "application 3" to which the interface belongs, an interface title name "add travel information", and a label name "ID number".

[0637] Table 2

[0638] Target label name Application 1 Add sender information Contact information Mobile phone number Application 1 Apply for invoice Contact information Email number Application 2 Add student file Certificate number File number Application 3 Add travel information Certificate number ID number Figure 29 Figure 35 Figure 29

[0639] The above describes a specific implementation process of how the mobile phone displays corresponding candidate user information according to the triggered filling control for the user to select. Next, the process of how to standardize the label name will be described in detail in combination with the structure shown in Figure 36 and the standardization process of the label name shown in Figure 33 .

[0640] S3501, the image memory module invokes a preset mapping rule.

[0641] Specifically, after obtaining the attribute information of the name control, the image memory module can invoke the preset mapping rule. It can be understood that the plurality of mapping relationships included in the preset mapping rule can be obtained from the name management module in the cloud service. For example, the image memory module can obtain new mapping relationships within a preset period from the name management module every interval of the preset period. For another example, the name management module can directly send new mapping relationships to the image memory module in the case of generating new mapping relationships.

[0642] S3502, the image memory module determines whether the attribute information of the name control satisfies any mapping relationship in the preset mapping rule.

[0643] In some embodiments, after invoking the preset mapping rule, the image memory module can determine whether the attribute information of the name control satisfies any mapping relationship in the preset mapping rule. If the attribute information of the name control satisfies any mapping relationship in the preset mapping rule, it means that the attribute information of the name control satisfies the preset condition, that is, the label name can be standardized, and therefore the image memory module can perform S3503. If the attribute information of the name control does not satisfy any mapping relationship in the preset mapping rule, it means that the label name can not be standardized, and therefore, in order to further determine whether the label name can be standardized, the image memory module can perform S3504.

[0644] In an implementation manner, if the mapping table representing the preset mapping rule only includes the label name and the target label name, that is, the target label name can be directly determined based on the label name, therefore, the image memory module can only determine whether there is a label name same as the label name of the name control in the preset mapping rule. If there is a label name same as the label name of the name control in the preset mapping rule, the image memory module can determine that the attribute information of the name control satisfies the preset condition. If there is no label name same as the label name of the name control in the preset mapping rule, that is, the label name is different from the label name in the preset mapping rule, the image memory module can determine that the attribute information of the name control does not satisfy the preset condition.

[0645] For example, if the label name of the name control is "recipient" and the mapping table shown in Table 1 also includes the label name "recipient", it indicates that the mapping table includes the same label name as the label name "recipient" of the name control, and thus the image memory module can determine that the attribute information of the name control satisfies the preset condition, i.e., the label name "recipient" of the name control can be standardized. If the label name of the name control is "consignee" and the mapping table shown in Table 1 does not include "consignee", it indicates that the mapping table does not include the same label name as the label name "consignee" of the name control, and thus the image memory module can determine that the attribute information of the name control does not satisfy the preset condition, i.e., the label name "consignee" of the name control cannot be standardized.

[0646] In another implementation, if the mapping table representing the preset mapping rule includes the application to which the interface belongs, the interface title name, the label name, and the target label name, i.e., the target label name is determined based on the interface title name, the application to which the interface belongs, and the label name, the image memory module needs to determine whether the preset mapping rule includes a mapping relationship that is the same as the application to which the interface belongs, the interface title name, and the label name included in the attribute information. If the preset mapping rule includes a mapping relationship that is the same as the application to which the interface belongs, the interface title name, and the label name included in the attribute information, the image memory module can determine that the attribute information of the name control satisfies the preset condition. If the preset mapping rule does not include a mapping relationship that is the same as the application to which the interface belongs, the interface title name, and the label name included in the attribute information, i.e., any content included in the attribute information is different from the mapping relationship in the preset mapping rule, the image memory module can determine that the attribute information of the name control does not satisfy the preset condition.

[0647] For example, referring to Table 2, if the application to which the interface information belongs is application 1, the interface title name is add sender information, and the label name of the name control is contact information, and the mapping table shown in Table 2 includes a mapping relationship that simultaneously satisfies the application to which the interface belongs "application 1", the interface title name "add sender information", and the label name of the name control "contact information", the image memory module can determine that the attribute information of the name control satisfies the preset condition, that is, the label name "contact information" of the name control can be standardized. If the application to which the attribute information belongs is application 2, the interface title name is add sender information, and the label name of the name control is contact information, and the mapping table shown in Table 2 does not include a mapping relationship that simultaneously satisfies the application to which the interface belongs "application 2", the interface title name "add sender information", and the label name of the name control "contact information", the image memory module can determine that the attribute information of the name control does not satisfy the preset condition, that is, the label name "contact information" of the name control cannot be standardized.

[0648] In S3503, the image memory module takes the target label name included in the mapping relationship satisfied by the attribute information as the target label name of the name control.

[0649] It can be understood that if the attribute information of the name control satisfies any mapping relationship in the preset mapping rule, it means that the label name of the name control has been standardized, and therefore the image memory module can directly take the target label name in the mapping relationship satisfied by the attribute information of the name control as the target label name of the name control. In this way, not only can the loss of resources caused by repeated standardization be reduced, but also the utilization rate of mobile resources can be improved, and the standardization efficiency of the label name can be improved, thereby speeding up the efficiency of information display and providing convenient conditions for subsequent selection of corresponding information by the user.

[0650] In S3504, the image memory module determines whether there is a label name in the user information library that has a similarity greater than a preset similarity with the label name of the name control.

[0651] It can be understood that if the attribute information of the name control does not satisfy any mapping relationship in the preset mapping rule, it means that the label name of the name control has not been standardized, and therefore, in order to further determine whether the label name of the name control can be standardized, the image memory module can continue to determine whether there is a label name in the user information library that has a similarity greater than a preset similarity with the label name of the name control. The preset similarity can be set in advance according to actual conditions. In this way, multi-angle standardization of the label name can be achieved, that is, the label name can be standardized in different ways, thereby providing a basis for subsequent accurate display of candidate user information.

[0652] In some embodiments, if there is a label name in the user information library that has a similarity greater than a preset similarity with the label name of the name control, it indicates that the label name of the name control is more likely to be the same as the label name in the user information library, that is, the label name of the name control can be standardized, and therefore the image memory module can perform S3505. If there is no label name in the user information library that has a similarity greater than a preset similarity with the label name of the name control, that is, the similarity between each label name in the user information library and the label name of the name control is less than or equal to the preset similarity, that is, the label name of the name control cannot be standardized, and therefore the image memory module can perform S3506.

[0653] Specifically, for each label name in the user information library, the image memory module can calculate the similarity between the label name and the label name of the name control. The similarity is used to represent the similarity of the label name of the name control and the label name being the same label name. Then, in the case that there is any label name in the user information library that has a similarity greater than a preset similarity with the label name of the name control, the image memory module can determine that the attribute information of the name control meets the preset condition, that is, the label name of the name control can be standardized. In the case that the similarity between each label name in the user information library and the label name of the name control is less than or equal to the preset similarity, the image memory module can determine that the attribute information of the name control does not meet the preset condition, that is, the label name of the name control cannot be standardized.

[0654] The similarity can be determined based on a semantic vector (or referred to as a first semantic vector) of the label name of the name control and a semantic vector (or referred to as a second semantic vector) of the label name in the user information library. Specifically, the image memory module can perform feature extraction on the label name of the name control according to a semantic vector model to obtain the first semantic vector. The semantic vector model is used to map words to vectors, so that each word corresponds to a semantic vector. Then, the image memory module can obtain the second semantic vector of each label name in the user information library. The second semantic vector is obtained by feature extraction based on the semantic vector model. Then, for the second semantic vector of each piece of user information in the user information library, the image memory module can calculate the similarity between the second semantic vector and the first semantic vector, that is, the similarity between the label name of the user information and the label name.

[0655] For example, if the semantic vector of the username is [0.99, 0.12, 0.03, 0.8], the semantic vector of the name in the user information library is [0.79, 0.42, 0.08, 0.7], and the semantic vector of the mobile phone number in the user information library is [0.19, 0.22, 0.48, 0.7], the image memory module calculates the similarity between the semantic vector of the username and the semantic vector of the name as 0.88, and the similarity between the semantic vector of the username and the semantic vector of the mobile phone number as 0.60. It can be seen that the similarity between the semantic vector of the username and the semantic vector of the name is 0.88, which is greater than the preset similarity 0.85, and therefore the image memory module can determine that the label name "username" of the name control meets the preset condition.

[0656] In S3505, the image memory module determines the label name of the name control as the target label name if the similarity between the label name and the label name of the name control is greater than the preset similarity.

[0657] It can be understood that if there is a label name in the user information library that has a similarity greater than the preset similarity with the label name of the name control, it means that the label name in the user information library is similar to the label name of the name control, and therefore the image memory module can directly determine the label name with the similarity greater than the preset similarity as the target label name of the name control. In this way, the target label name can be quickly and accurately determined, which provides a basis for subsequent determination of candidate user information.

[0658] In some embodiments, if there are multiple label names in the user information library that have a similarity greater than the preset similarity with the label name of the name control, the image memory module can determine all the multiple label names as the target label name of the name control. For example, if the label name of the name control is certificate number, and the label names in the user information library can include ID number, student number, and filing number, the similarity between the ID number and the certificate number is 1, the similarity between the student number and the certificate number is 2, and the similarity between the filing number and the certificate number is 3, all of which are greater than the preset similarity, the image memory module can determine the ID number, the student number, and the filing number as the target label name of the name control. In this way, the situation of missing display of user information can be reduced, and the accuracy of information display can be improved.

[0659] In S3506, the image memory module uploads the label name of the name control to the cloud service.

[0660] Specifically, after determining that the similarity between the label name and the label name of the name control in the user information library is greater than the preset similarity, it is determined that the label name of the name control cannot be directly standardized. Therefore, the image memory module can upload the label name of the name control to the cloud service. Then, the name management module in the cloud service can perform standardized learning on the label name of the name control. In this way, a basis for subsequent comprehensive standardization of the label name is provided.

[0661] It should be noted that the above specific implementation process of standardizing the label name is described based on the condition that the attribute information of the name control satisfies any mapping relationship in the preset mapping rule, and the similarity between the label name and the label name of the name control in the user information library is greater than the preset similarity. However, if the preset condition only includes that the attribute information of the name control satisfies any mapping relationship in the preset mapping rule, the mobile phone can only perform the processes of S3501-S3503, and in the case that the attribute information of the name control does not satisfy any mapping relationship in the preset mapping rule, the label name of the name control is uploaded to the cloud service. If the preset condition only includes that the similarity between the label name and the label name of the name control in the user information library is greater than the preset similarity, the mobile phone can only perform the processes of S3504-S3506.

[0662] The above describes the specific implementation process of how the image memory module in the mobile phone standardizes the label name of the name control. Next, the process of how to obtain user information from the preset collection application to store in the user information library will be described in detail in combination with the structure shown in Figure 33 and the process of obtaining user information in the user information library shown in Figure 33 S3601-S3606:

[0663] S3601, the function setting module in the mobile phone receives a trigger operation of a user on any switch control in the setting interface.

[0664] The above setting interface (or information collection setting interface) is an interface for setting whether the application program in the mobile phone can collect user information. For example, the setting interface can be the interface shown in (a) of Figure 33

[0665] S3602, in response to the trigger operation of the user on any switch control in the setting interface, the function setting module sends a function setting instruction to the image memory module.

[0666] ​In some embodiments, if any of the switch controls in the setting interface is triggered by the user, it indicates that the user wants to adjust the information collection function of the application to which the switch control belongs, that is, the user wants to turn on or turn off the information collection function. Therefore, the function setting module can send a function setting instruction to the portrait memory module. The function setting instruction carries a setting result of the information collection function. The setting result is used to indicate whether the information collection function of the application to which the triggered switch control belongs is turned on.

[0667] It can be understood that the application program to which each switch control belongs is pre-set. For example, in the (a) interface in FIG. 33, the application program to which the switch control 3301A belongs is all the applications in the mobile phone, that is, as long as the switch control 3301A is turned on by the user, the mobile phone can collect the user information saved by all the applications. The application program to which the switch control 3302 belongs is the message application in the mobile phone, that is, as long as the switch control 3302 is turned on by the user, the mobile phone can collect the user information saved by the message application. Figure 29

[0668] S3603, in a case where the function setting instruction sent by the function setting module is received, the portrait memory module updates the collection application list according to the setting result carried by the function setting instruction. The collection application list includes at least one preset collection application.

[0669] Specifically, after receiving the function setting instruction, the portrait memory module can update the collection application list according to the setting result carried by the function setting instruction. In this way, not only the privacy of the user can be protected, but also a basis for subsequent acquisition of user information is provided.

[0670] In an implementation manner, if the setting result indicates that the information collection function of the application to which the triggered switch control belongs is turned on, the portrait memory module can add the application to which the triggered switch control belongs to the collection application list to obtain a new collection application list.

[0671] In another implementation manner, if the setting result indicates that the information collection function of the application to which the triggered switch control belongs is turned off, the portrait memory module can delete the application to which the triggered switch control belongs from the collection application list to obtain a new collection application list.

[0672] S3604, the portrait memory module judges whether the application to which the triggered switch control belongs is an application program in the collection application list.

[0673] ​In some embodiments, after receiving the function setting instruction, the portrait memory module can determine whether the application to which the triggered switch control belongs is an application in the collection application list. If the application to which the triggered switch control belongs is an application in the collection application list, it indicates that the user wants to turn off the information collection function of the application. Therefore, the portrait memory module can perform S3605. If the application to which the triggered switch control belongs is not an application in the collection application list, it indicates that the user wants to turn on the information collection function of the application. Therefore, the portrait memory module can perform S3606.

[0674] S3605, in the case where the application to which the triggered switch control belongs is an application in the collection application list, the portrait memory module deletes all user information associated with the application from the user information library.

[0675] Specifically, if the application to which the triggered switch control belongs is an application in the collection application list, the portrait memory module can delete all user information associated with the application from the user information library. In this way, all user information associated with the application can be deleted while turning off the information collection function of the application, thereby protecting user privacy and improving the security of mobile phone use.

[0676] In some embodiments, in the case where the application to which the triggered switch control belongs is an application in the collection application list, it indicates that the user does not want to disclose the user information entered by the application. Therefore, the portrait memory module can directly delete all user information associated with the application from the user information library. In this way, the information privacy of the user can be protected, and security assurance can be provided for the user.

[0677] In another embodiment, in the case where the application to which the triggered switch control belongs is an application in the collection application list, the mobile phone can display a pop-up interface. The pop-up interface can include prompt information, a delete control, and a retention control. The prompt information is used to represent whether the user information about the application to which the triggered switch control belongs saved in the user information library is retained. Then, in response to the user's triggering operation on the delete control in the pop-up interface, the portrait memory module in the mobile phone can delete all user information associated with the application from the user information library. Alternatively, in response to the user's triggering operation on the retention control in the pop-up interface, the portrait memory module in the mobile phone can retain all user information associated with the application in the user information library.

[0678] For example, as shown in FIG. 33B, if the switch control 3304 to which the note application belongs is clicked by the user, it indicates that the user wants to turn off the information collection function of the note application. Therefore, the mobile phone can display a pop-up interface 3306 as shown in FIG. 33C. Figure 37 Figure 38 ​The pop-up interface shown in (c) is a pop-up interface based on the setting interface. The pop-up interface can include prompt information about whether to retain the collected data of the note, a deletion control, and a retention control. If the deletion control is clicked by the user, the portrait memory module can delete all user information associated with the note application from the user information library. If the retention control is clicked by the user, the portrait memory module can retain all user information associated with the note application in the user information library.

[0679] S3606, in a case where the application to which the triggered switch control belongs is not an application in the collection application list, the portrait memory module acquires all user information saved by the application and stores the user information in the user information library.

[0680] Specifically, in a case where the application to which the triggered switch control belongs is not an application in the collection application list, the portrait memory module can acquire all user information saved by the application and store the user information in the user information library. In this way, all user information saved by the application can be acquired while the information collection function of the application is enabled, thereby providing a basis for subsequent display of recommended information.

[0681] It can be understood that the steps S3605 and S3606 are parallel solutions, that is, in a case where the application to which the triggered switch control belongs is an application in the collection application list, the portrait memory module can execute the steps of S3605 without executing the steps of S3606; in a case where the application to which the triggered switch control belongs is not an application in the collection application list, the portrait memory module can execute the steps of S3606 without executing the steps of S3605.

[0682] In an implementation manner, the process of updating the collection application list by the portrait memory module and the process of determining, by the portrait memory module, whether the application to which the triggered switch control belongs is an application in the collection application list can be executed simultaneously or in a sequence. For example, the portrait memory module can execute the steps of S3603 first and then execute the steps of S3604-S3606. For another example, the portrait memory module can execute the steps of S3604-S3606 first and then execute the steps of S3603.

[0683] The above describes the specific implementation process of how the portrait memory module in the mobile phone acquires user information from the preset collection application to store in the user information library. Next, how to query and adjust the user information in the user information library will be described in detail in combination with the structure shown in Figure 38 and the process of processing user information shown in Figure 38 , which can include S3701-S3708. S3701, the portrait memory module in the mobile phone acquires all user information in the user information library. S3702, the portrait memory module in the mobile phone determines whether the user information in the user information library is valid. S3703, in a case where the user information in the user information library is valid, the portrait memory module in the mobile phone retains the user information in the user information library. S3704, in a case where the user information in the user information library is invalid, the portrait memory module in the mobile phone deletes the user information in the user information library. S3705, the portrait memory module in the mobile phone determines whether the user information in the user information library is valid. S3706, in a case where the user information in the user information library is valid, the portrait memory module in the mobile phone retains the user information in the user information library. S3707, in a case where the user information in the user information library is invalid, the portrait memory module in the mobile phone deletes the user information in the user information library. S3708, the portrait memory module in the mobile phone determines whether the user information in the user information library is valid.

[0684] S3701, the information management module in the mobile phone receives a query operation of a user on the user information library.

[0685] The query operation can be a click operation on any query control in the mobile phone with an information display function. That is, if the query control in the mobile phone is clicked by the user, the mobile phone can directly display the user information in the user information library.

[0686] In an example, the query control can be a query control for all user information in the user information library. For example, referring to Figure 38 , the mobile phone is displaying the setting interface shown in (a) of Figure 38 , which includes a "personal information" control (i.e., a query control). In another example, the query control can also be a query control for the user information of any user in the user information library. For example, referring to Figure 38 , the mobile phone is displaying the information list interface shown in (b) of Figure 38 , which includes information query controls (i.e., query controls) of users, such as an information query control of the owner user A, an information query control of the user B, and the like. In yet another example, the query control can also be a query control for any user information in the user information library, and the like, without limitation.

[0687] S3702, in response to the query operation of the user on the user information library, the information management module sends an information calling instruction to the portrait memory module.

[0688] In some embodiments, after detecting the query operation of the user on the user information in the user information library, the information management module can send an information calling instruction to the portrait memory module, so that the portrait memory module calls the user information from the user information library for display. The information calling instruction carries a label name of the user information to be queried. The user to be queried is determined based on the query control corresponding to the query operation of the user. For example, if the query control is a query control for all user information in the user information library, the user to be queried is all users in the user information library. For another example, if the query control is a query control for the user information of any user in the user information library, the user to be queried is the user corresponding to the query control.

[0689] S3703, the portrait memory module, upon receiving the information calling instruction sent by the information management module, calls target user information from the user information library according to the label name of the user to be queried carried by the information calling instruction.

[0690] Specifically, after receiving the information calling instruction, the image memory module can call the user information matching the label name of the user to be queried from the user information library, that is, call the target user information. In this way, a basis for subsequent display of the target user information is provided.

[0691] In an example, if the user to be queried is user A and the label name is a mobile phone number, the image memory module can call the mobile phone number of user A from the user information library and take the mobile phone number of user A as the target user information. In another example, if the user to be queried is all users and the label name is a user name, the image memory module can call the user names of all users from the user information library and take the user names of all users as the target user information.

[0692] S3704, the image memory module sends the target user information to the information management module.

[0693] S3705, the information management module displays the target user information upon receiving the target user information sent by the image memory module.

[0694] In some embodiments, the information management module can display the target user information upon receiving the target user information. In this way, the user information query function can be realized, the accuracy of user information display can be improved, and the user query experience can be improved.

[0695] For example, as shown in Figure 32 If the "personal information" control in the setting interface (a) is clicked by the user, the mobile phone can display the user names of all users in the user information library (such as the owner user A, user B, etc.), that is, display the user list interface shown in (b) in Figure 38 If the information query control to which the owner user A belongs in the user list interface is clicked by the user, the mobile phone can display all user information of the owner user A (such as the mobile phone numbers "131****5613", "189****7894", etc.), that is, display the information detail interface shown in (c) in Figure 32 .

[0696] S3706, the information management module receives an information adjustment operation of the user on the user information library.

[0697] The information adjustment operation can be an information deletion operation for any user in the user information library, an information modification operation for any user information in the user information library, or an information addition operation for the user information library, and the like, without limitation. It can be understood that the information deletion operation, the information modification operation, and the information addition operation are all pre-set according to actual conditions. For example, the information deletion control can be a sliding operation for the query control. The information modification operation can be a long-press operation for the query control, which refers to an operation corresponding to a pre-set time when a user's finger presses the query control for a time reaching the pre-set time. The information addition operation can be a single-click operation for the addition control, which is used to add new user information to the user information library. The new user information can be user information of a new user or new user information of an original user in the user information library, without limitation.

[0698] In an example, the user list interface shown in interface (a) is taken as an example. The information deletion operation can be a sliding operation (for example, a sliding operation from right to left) for the information query control to which the owner user A belongs. The information addition operation can be a single-click operation for the addition control 3201. In another example, the information detail interface shown in interface (c) is taken as an example. The information modification operation can be a long-press operation for the student number control. Figure 32 Figure 39

[0699] S3707, in response to the information adjustment operation of the user for the user information library, the information management module sends an information adjustment instruction to the portrait memory module.

[0700] Specifically, after detecting the adjustment operation of the user for the user information library, the information management module can send an information adjustment instruction to the portrait memory module, so as to subsequently adjust the user information in the user information library accordingly. The information adjustment instruction carries the information adjustment operation.

[0701] ​​In some embodiments, if the information adjustment operation is an information addition operation, it indicates that the user wants to add new user information, and thus the information adjustment instruction only needs to carry the information addition operation. If the information adjustment operation is an information deletion operation or an information modification operation, it indicates that the user wants to process the user information already saved in the user information library, and thus the information adjustment instruction can also carry the label name corresponding to the user information to be processed (or referred to as the to-be-processed information) in order to accurately determine the user information to be processed (i.e., the user information that the user wants to process). For example, if the information adjustment operation is an information deletion operation for the owner user A, the information adjustment instruction carries the information deletion operation, the owner user A, and all label names corresponding to the owner user A. For another example, if the information adjustment operation is an information modification operation for the mobile phone number of the owner user A, the information adjustment instruction carries the information modification operation, "131****5613", and the mobile phone number corresponding to "131****5613".

[0702] In S3708, the image memory module adjusts the user information library according to the information adjustment instruction sent by the information management module.

[0703] In some embodiments, after receiving the information adjustment instruction, the image memory module can directly process the user information in the user information library according to the content carried by the information adjustment instruction. In this way, the adjustment function (i.e., the addition function, the deletion function, and the modification function) of the user information can be realized, the user information in the user information library is added based on the actual needs of the user, the necessity of each piece of user information is ensured, and the user experience is improved.

[0704] In an implementation manner, in the case where the information adjustment instruction carries an information addition operation, the image memory module can add the newly filled user information to the user information library. For example, as shown in FIG. 13A, the mobile phone is displaying the personal information interface shown in interface (a), and if the addition control 501 in the personal information interface is clicked by the user, the mobile phone can display the information editing interface shown in interface (b) in FIG. 13B. Then, in response to the filling operation of the user on any information control (such as the information control 3202 of the user name) in the information editing interface, the mobile phone can display the filling content (such as "Zhao Yiyi") of the information control. Then, if the "save" control 3203 in the information editing interface is clicked by the user, the image memory module in the mobile phone can add the newly filled user information to the user information library. The newly filled user information is respectively the user name "Zhao Yiyi", the nationality "Han", and the mobile phone number "132****7788". Figure 39 Figure 38

[0705] ​​In another implementation, when the aforementioned information adjustment instruction carries an information deletion operation and the tag name corresponding to the user information of the user to whom the information deletion operation belongs, the profile memory module can delete the user information corresponding to the tag name from the user information database, that is, delete the tag name of the aforementioned information to be processed. For example, such as... Figure 37 As shown, the phone is displaying the personal information interface shown in interface (a). Responding to the user's swipe gesture on the information query control for user B, indicating that the user wants to delete all user information for user B, the phone's profile memory module can delete all user information for user B from the user information database. Afterwards, the phone can display... Figure 40 The personal information interface shown in (b) is the personal information interface that does not include the information query control belonging to user B.

[0706] In another implementation, when the aforementioned information adjustment instruction carries an information modification operation and the tag name corresponding to the user information of the user to whom the information modification operation belongs, the profile memory module can modify the user information corresponding to the tag name and replace the modified user information with the modified user information, saving it to the user information database. For example, as shown... Figure 41 As shown, in response to the user's long press operation on the student ID control in interface (c), indicating that the user wants to modify the student ID of the owner user A, the profile memory module in the phone can modify the student ID "000***147" of the owner user A based on the newly entered student ID content, and save the modified student ID to the user information database.

[0707] It should be noted that, Figure 41 The execution order of the information query and information adjustment processes shown is not limited. For example, the mobile phone can first execute steps S3701 to S3705, and then execute steps S3706 to S3708; or, the mobile phone can first execute steps S3706 to S3708, and then execute steps S3701 to S3705. Alternatively, the mobile phone can execute only steps S3701 to S3705, or only steps S3706 to S3708.

[0708] It should be understood that the corresponding steps performed by each module in the aforementioned mobile phone can also be performed by other modules in the mobile phone. For example, the process by which the profile memory module detects the label name of the name control can also be performed by the name management module in the cloud service. As another example, the process by which the profile memory module adjusts user information in the user information database can also be performed by the information management module; no specific limitations are imposed.

[0709] The process of displaying corresponding candidate user information based on the triggered fill-in control is introduced above. The specific implementation process of how to display the candidate user information corresponding to all fill-in controls in the to-be-entered interface will be introduced below. As shown in Figure 41 , the process is as follows:

[0710] S4001, the form filling assistant in the mobile phone receives a one-key fill-in operation of the user for the to-be-entered interface.

[0711] For example, the one-key fill-in operation can be a triggering operation of the user for the "one-key fill-in" control in the to-be-entered interface (such as the interface shown in (a) of Figure 35 ), or the user can output a voice instruction including "one-key fill-in" when the mobile phone displays the to-be-entered interface, and the specific implementation is not limited.

[0712] S4002, in response to the one-key fill-in operation of the user for the to-be-entered interface, the form filling assistant obtains an attribute information list. The attribute information list includes the attribute information of each name control in the to-be-entered interface, and the attribute information of the name control includes the label name.

[0713] In an implementation manner, the form filling assistant can obtain the attribute information of the name control in sequence according to the display order of the name control in the to-be-entered interface, and compose the attribute information of the plurality of name controls obtained in sequence into the attribute information list. That is, the attribute information of the name control in the attribute information list is sorted according to the display order. In this way, a basis is provided for subsequent display of the recommended information, and the situation that the display effect is affected because the recommended information does not correspond to the label name is reduced.

[0714] For example, as shown in Figure 41 , the mobile phone is displaying the to-be-entered interface shown in (a) of Figure 41 , and the display order of the name controls in the to-be-entered interface is "recipient" control, "mobile phone number" control and "detailed address" control respectively. Therefore, the above attribute information list can include the attribute information of the "recipient" control, the attribute information of the "mobile phone number" control and the attribute information of the "detailed address" control, that is, the attribute information of the control is the recipient, the mobile phone number and the detailed address.

[0715] S4003, the form filling assistant sends the attribute information list to the portrait memory module in the mobile phone.

[0716] Specifically, after obtaining the attribute information list, the form filling assistant can send the attribute information list to the portrait memory module, so that the portrait memory module can determine the corresponding candidate user information in sequence according to the attribute information in the attribute information list, thereby improving the accuracy of the determination of the candidate user information.

[0717] S4004, the image memory module, in the case of receiving the attribute information list sent by the form filling assistant, standardizes the label name of each name control in the attribute information list to obtain the target label name of each name control.

[0718] In some embodiments, after receiving the attribute information list, the image memory module can detect the label name of each name control in the attribute information list to obtain a detection result. Then, in the case that the detection result indicates that the label name of the name control can be standardized, the image memory module can standardize the label name of the name control to obtain the target label name of the name control.

[0719] The detection process and the standardization process of the label name of the name control are similar to the above-described standardization process of the label name of the name control, and will not be described here. Figure 41

[0720] S4005, the image memory module divides the target label names of the plurali...

Claims

1. An information processing method, characterized in that, include: The first interface is displayed. The first interface is a form filling interface. The first interface includes a first fill box and multiple label boxes. Each label box has a corresponding label name. Different label boxes have different label names. In response to a first operation applied to the first input box in the first interface, a first label box matching the first input box is determined from the plurality of label boxes; the first label box has a first label name; The first content text is displayed in the associated area of ​​the first input box; the tag attribute of the first content text matches the first tag name.

2. The information processing method according to claim 1, characterized in that, The step of displaying the first content text in the associated area of ​​the first fill box includes: At least one optional content text matching the first label name is displayed in the associated area of ​​the first fill box; the at least one optional content text includes the first content text; In response to a user's touch operation on the first content text, the first content text is displayed in the first input box.

3. The information processing method according to claim 1 or 2, characterized in that, The first interface also includes a second input box, which is a different form frame from the first input box; The information processing method further includes: In response to a second operation applied to the second fill box in the first interface, a second label box matching the second fill box is determined from the plurality of label boxes; The second label box is a different table frame from the first label box, and the second label box has a second label name; The second content text is displayed in the associated area of ​​the second fill box; the tag attribute of the second content text matches the second tag name.

4. The information processing method according to any one of claims 1-3, characterized in that, The information processing method further includes: Obtain the cursor coordinates of the cursor corresponding to the first operation in the first interface; The first input box is determined based on the cursor coordinates; the coordinate range of the first input box within the table frame of the first interface includes the cursor coordinates.

5. The information processing method according to claims 1-4, characterized in that, Before determining the first label box that matches the first fill box from the plurality of label boxes, the information processing method further includes: Identify multiple table frames on the first interface, and identify multiple text information on the first interface; the multiple table frames include fill boxes and label boxes; The table frame containing each text message is determined based on the coordinates of the text box for each text message and the coordinate range of the table frames of the plurality of table frames; Determine the label name for each label box.

6. The information processing method according to any one of claims 1-5, characterized in that, The step of determining the first label box that matches the first fill box from the plurality of label boxes includes: Using a pre-determined Gaussian distribution model, a first label box matching the first fill box is determined from the plurality of label boxes; the Gaussian distribution model is constructed based on the relative positions of multiple table frame group samples, each table frame group sample includes a sample label box and a sample fill box, and the relative positions of the sample label box and the sample fill box are different in different table frame group samples.

7. The information processing method according to claim 6, characterized in that, The step of determining the first label box that matches the first fill box from the plurality of label boxes using a pre-determined Gaussian distribution model includes: Calculate the first distance between each label box and the first fill box; Filter candidate label boxes whose first distance is greater than a first threshold; Using the Gaussian distribution model, the matching probability between each candidate label box and the first fill box is calculated; The candidate label box with the highest matching probability is determined as the first label box.

8. The information processing method according to any one of claims 1-7, characterized in that, Before displaying the first content text in the associated area of ​​the first fill box, the information processing method further includes: The table database is used to search for the first content text that matches the first tag name; the table database includes multiple sets of associated data, each set of associated data including mutually matching tag names and content text.

9. The information processing method according to claim 8, characterized in that, Before searching the table database for the first content text that matches the first tag name, the information processing method further includes: Identify the filled form boxes on the first interface; the filled form boxes are fill boxes with text that has been filled in. Each associated data in the table database also includes multiple tag names and content text matching each tag name; the step of searching the table database for the first content text matching the first tag name includes: From the table database, search for first associated data including the filled-in text content, and from the first associated data, search for first content text that matches the first tag name.

10. The information processing method according to any one of claims 1-9, characterized in that, The information processing method further includes: In response to a user's click action on the first control, the currently displayed first interface is identified as a form-filling interface; the first control is used to trigger automatic form filling on the form-filling interface.

11. The information processing method according to any one of claims 1-9, characterized in that, The information processing method further includes: Identify the change in cursor coordinates on the first interface; If the change in cursor coordinates is greater than the second threshold, the currently displayed first interface is determined to be a form-filling interface.

12. An electronic device, characterized in that, The electronic device includes a touch screen, a memory, and a processor, wherein the touch screen and the memory are both coupled to the processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory, causing the electronic device to perform the information processing method as described in any one of claims 1 to 11.

13. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when run on a computer, causes the computer to perform the information processing method as described in any one of claims 1 to 14.

14. A computer program product, characterized in that, It includes a computer program, which, when executed by a processor, implements the information processing method as described in any one of claims 1 to 11.

Images