Information filling method and device, medium and electronic equipment

Abstract

The invention belongs to the technical field of artificial intelligence, and relates to an information filling method and device, a medium and electronic equipment. The method comprises the steps that when a user operation instruction is received, a screen image is captured, and a to-be-filled item is displayed on the screen image; and in response to a user operation instruction, filling the to-be-filled item. According to the information filling method and device, the filling limitation of a single input box is overcome, the operation complexity of a user is reduced, the information filling time is shortened, the information identification and filling accuracy is improved, the filling mode is not limited to the adaptation of a platform and a scene, the cross-platform and multi-scene information filling requirements can be met, and the user experience is improved. And the scene application coverage rate is improved.

Description

Technical Field

[0001] The present disclosure belongs to the field of artificial intelligence technology, and in particular relates to an information filling method, an information filling device, a computer-readable storage medium, and an electronic device. Background Art

[0002] With the development of technology, users often need to fill out various online forms. In scenarios where existing filling services are adapted, they can support intelligent recognition of clipboard contents, enabling one-click copying and filling of a single input field. Alternatively, they can provide input suggestions and fill in a single input field based on user input, historical forms on smart connected devices, user account information, and other information.

[0003] However, the applicable scenarios for filling in a single input box are relatively simple, and users are required to find the correct filling location on the form page. As a result, users have reported that filling in the form pages one by one is too complicated, the operation is not smart enough, and the versatility is poor. Summary of the Invention

[0004] To overcome the problems existing in the related art, the present disclosure provides an information filling method, an information filling device, a computer-readable storage medium, and an electronic device.

[0005] According to a first aspect of an embodiment of the present disclosure, there is provided an information filling method, comprising: When receiving a user operation instruction, capturing a screen image, wherein the screen image displays the items to be filled in; In response to the user operation instruction, the to-be-filled item is filled.

[0006] Optionally, the method further includes: Displaying a next image of the screen image, wherein the next image displays other items to be filled that have not been fully displayed; In response to the user operation instruction, continue to fill in the other items to be filled in; or In response to another user operation instruction, continue filling in the other items to be filled in.

[0007] Optionally, the method further includes: When a user operation instruction is received, the first large language model is used to extract the entity to be filled in the user operation instruction.

[0008] Optionally, after extracting the to-be-filled entity in the user operation instruction using the first large language model, the method further includes: A context association algorithm is used to normalize the entity to be filled in, so as to obtain a normalized entity to be filled in.

[0009] Optionally, after capturing the screen image, the method further includes: A large-scale visual language model is used to determine items to be filled in the screen image.

[0010] Optionally, after determining the to-be-filled items in the screen image using the large-scale visual language model, the method further includes: The to-be-filled item is normalized using a reference resolution algorithm to obtain a normalized to-be-filled item.

[0011] Optionally, filling the to-be-filled item includes: Entity attribute information corresponding to the to-be-filled entity is retrieved, and the to-be-filled item is filled with the entity attribute information.

[0012] Optionally, filling the entity attribute information in the to-be-filled item includes: Using a second large language model, a first mapping relationship is established between the item to be filled and the entity attribute information, and a second mapping relationship is generated between a position to be filled and the entity attribute information, wherein the position to be filled is determined by the large-scale visual language model; The first mapping relationship and the second mapping relationship are converted into operation instructions, so as to fill the entity attribute information into the to-be-filled position of the to-be-filled item according to the operation instructions.

[0013] According to a second aspect of an embodiment of the present disclosure, there is provided an information filling device, including: a data acquisition module configured to capture a screen image when receiving a user operation instruction, wherein the screen image displays items to be filled; The information filling module is configured to fill in the items to be filled in response to the user operation instruction.

[0014] Optionally, the device further comprises: an image display module configured to display a next image of the screen image, wherein the next image displays other items to be filled that have not been fully displayed; A first filling module is configured to continue filling the other to-be-filled items in response to the user operation instruction; or The second filling module is configured to continue filling the other to-be-filled items in response to another user operation instruction.

[0015] Optionally, the device further comprises: The model extraction module is configured to extract the to-be-filled entity in the user operation instruction by using the first large language model when receiving the user operation instruction.

[0016] According to a third aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided, on which computer program instructions are stored. When the program instructions are executed by a processor, the steps of any one of the information filling methods provided in the first aspect of the present disclosure are implemented.

[0017] According to a fourth aspect of the embodiments of the present disclosure, there is provided an electronic device, including: processor; a memory for storing processor-executable instructions; The processor is configured to: execute the executable instructions to implement the steps of any one of the information filling methods provided in the first aspect of the present disclosure.

[0018] The technical solutions provided by the embodiments of the present disclosure may have the following beneficial effects: In the method and apparatus provided by the exemplary embodiments of the present disclosure, the information filling process is triggered by user operation instructions, which greatly reduces the tediousness of the operation and the degree of dependence on the accuracy of the user operation compared to the interactive method of user confirmation of each field. Furthermore, by filling in the items to be filled in the screen image in response to the user's operation instructions, a "one-sentence filling" information filling method is provided, which reconstructs the human-computer interaction method in the information filling scenario, overcomes the filling limitations of a single input box, greatly shortens the time for filling information, and improves the accuracy of information recognition and filling. Moreover, this filling method has no restrictions on the adaptation of platforms and scenarios, and can meet the information filling needs across platforms and multiple scenarios, and achieves a leap-forward improvement in scenario application coverage.

[0019] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0021] Figure 1 The following schematically shows a flow chart of an information filling method in an exemplary embodiment of the present disclosure; Figure 2 The following schematically illustrates a flow chart of a method for filling entity attribute information in an exemplary embodiment of the present disclosure; Figure 3 A schematic flow chart of a method for continuously filling in other items to be filled in an exemplary embodiment of the present disclosure is shown; Figure 4 The following schematically illustrates an architecture diagram of an information filling method in an application scenario in an exemplary embodiment of the present disclosure; Figure 5The following schematically shows an interface diagram for displaying items to be filled in an application scenario in an exemplary embodiment of the present disclosure; Figure 6 Schematically illustrates an interface diagram applicable to the information filling method in an application scenario in an exemplary embodiment of the present disclosure; Figure 7 Schematically illustrates another interface diagram applicable to the information filling method in an application scenario in an exemplary embodiment of the present disclosure; Figure 8 A schematic structural diagram of an information filling device in an exemplary embodiment of the present disclosure is shown schematically; Figure 9 A schematic diagram schematically illustrates the structure of another information filling device in an exemplary embodiment of the present disclosure; Figure 10 The following schematically shows the structure of another information filling device in an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION

[0022] Exemplary embodiments will be described in detail herein, with examples illustrated in the accompanying drawings. In the following description, when referring to the drawings, identical numerals in different figures represent identical or similar elements, unless otherwise indicated. The embodiments described in the following exemplary embodiments are not intended to represent all possible embodiments consistent with the present disclosure. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the present disclosure, as detailed in the appended claims.

[0023] It should be noted that all actions of acquiring signals, information or data in the present disclosure are carried out in compliance with the corresponding data protection laws and policies of the country where they are located and with the authorization given by the owner of the corresponding device.

[0024] With the development of technology, users often need to fill out various online forms. In scenarios where the Fill service is suitable, it supports intelligent recognition of clipboard contents, enabling one-click copying and filling of a single input field. Alternatively, it can provide input suggestions and fill a single input field based on user input, historical forms on smart connected devices, user account information, and other information.

[0025] In traditional form-filled page recognition technology, two subtasks are usually required: OCR (Optical Character Recognition) and Grounding (a task of matching natural language words with specific areas in an image). These two subtasks are related tasks in traditional computer vision.

[0026] Traditional OCR technology is limited by its ability to recognize structured information and process complex layouts. Traditional Grounding technology relies on manually designed rules or single-modal feature extraction, making it difficult to achieve accurate matching of image areas and text descriptions, and lacks the ability to reason about global semantics.

[0027] Therefore, the applicable scenarios for filling in a single input box are relatively simple, and users are required to find the correct filling location on the form filling page, resulting in user feedback that the method of filling in the form filling page one by one is too complicated, the operation is not smart enough, and the versatility is poor.

[0028] In order to solve the problems existing in the related art, the present invention provides an information filling method. Figure 1 is a flow chart showing a method for filling information according to an exemplary embodiment. Figure 1 As shown, the method may include at least the following steps: Step S110. When a user operation instruction is received, a screen image is captured, and the screen image displays items to be filled in.

[0029] Step S120: Fill in the items to be filled in in response to the user's operation instruction.

[0030] In an exemplary embodiment of the present disclosure, the information filling process is triggered by user operation instructions, which greatly reduces the tediousness of the operation and the degree of dependence on the accuracy of the user operation compared to the interactive method of user confirmation of each field. Furthermore, by filling in the items to be filled in the screen image in response to the user operation instructions, a "one-sentence filling" information filling method is provided, which reconstructs the human-computer interaction method in the information filling scenario, overcomes the filling limitations of a single input box, greatly shortens the time for filling information, and improves the accuracy of information recognition and filling. In addition, this filling method has no restrictions on the adaptation of platforms and scenarios, and can meet the information filling needs across platforms and multiple scenarios, and achieves a leapfrog improvement in scenario application coverage.

[0031] The following describes in detail the various steps of the information filling method.

[0032] In step S110 , when a user operation instruction is received, a screen image is captured, and the screen image displays items to be filled in.

[0033] In the exemplary embodiment of the present disclosure, the user operation instruction is an instruction indicating a request to fill in information, which may be in voice or text form, and this exemplary embodiment does not specifically limit this. Generally, the user operation instruction may be "Fill in my information" or "Fill in my father's ID number", etc.

[0034] In an optional embodiment, when a user operation instruction is received, the first large language model is used to extract the entity to be filled in the user operation instruction.

[0035] After receiving a user operation instruction, the first large language model is used to extract the to-be-filled entity in the user operation instruction. The form-filling operation requirement implicit in the user operation instruction can also be determined. The form-filling operation requirement can include "yes" and "no." Only when the form-filling operation requirement is "yes" is the first large language model used to extract the to-be-filled entity in the user operation instruction.

[0036] The entity to be filled in includes two parts: "relationship" and "keyword." Other entities may also be included, and this exemplary embodiment does not specifically limit this. The "relationship" entity to be filled in represents the relationship between the role and the user to be filled in the user operation instruction, and the "keyword" entity to be filled in represents the specific information to be filled in the user operation instruction.

[0037] In an optional implementation, a context association algorithm is used to normalize the entity to be filled, so as to obtain a normalized entity to be filled.

[0038] Since the representation of user operation instructions is related to personal behaviors such as the user's speaking habits, in order to accurately retrieve the entity attribute information corresponding to the item to be filled in, the extracted entity to be filled in can be normalized so that the corresponding entity attribute information can be retrieved using the standard entity to be filled in, avoiding the problem of inaccurate information filling.

[0039] Generally, contextual association algorithms can be used to normalize entities to be filled in. Contextual association algorithms determine relationships between texts based on contextual relevance when processing text. This technology can be used in fields such as natural language processing, machine translation, and search engine optimization. When calculating contextual text associations, algorithms such as cosine similarity and Jaccard similarity are often used to calculate similarity between texts, or fuzzy matching can be used for normalization.

[0040] When the user operation instruction is received, the current screen image may be captured to determine the items to be filled in displayed in the current screen image.

[0041] In an optional embodiment, a large-scale visual language model is used to determine the items to be filled in the screen image.

[0042] Among them, the large-scale visual language model can be QwenVL (Vision Language) architecture, LlaVA (Large Language and Vision Assistant, a multimodal large model), MiniCPM-V (a terminal-side multimodal large model), GLM-4v (General Language Model 4 Visual‌, a general language model) and InternVL (a multimodal large language model), etc. This exemplary embodiment does not specifically limit this.

[0043] Specifically, the screen image is input into a pre-trained large-scale visual language model, so that the pre-trained large-scale visual language model outputs a to-be-filled item. The to-be-filled item is a form structure description of the form filling operation requirement, which can support the joint understanding of the text, icons and layout structure in the screen image.

[0044] Table 1 shows the form protocol definition of the items to be filled in:

[0045] Table 1 It is worth noting that it is not necessary to fill in all the fields represented by the to-be-filled items in Table 1 in the screen image, or only fill in all the fields represented by the to-be-filled items in Table 1. Fields can be added or reduced according to actual conditions, and this exemplary embodiment does not make any special restrictions on this.

[0046] In an optional embodiment, a reference resolution algorithm is used to normalize the items to be filled, thereby obtaining normalized items to be filled.

[0047] On different form filling pages, the "keyword" entity to be filled in may have different expressions. For example, "name" can be a given name, a nickname, or a full name. In order to unify the items to be filled in, a reference resolution algorithm can be used to normalize the items to be filled in.

[0048] Co-reference resolution is a technique in natural language processing that primarily addresses the problem of reference in text, specifically identifying content within a text that refers to the same entity. Co-reference resolution analyzes contextual information to associate ambiguous pronouns with the specific entities they refer to, thereby helping computers better understand the meaning of sentences.

[0049] There are two main approaches to implementing coreference resolution algorithms: rule-based and machine learning. Rule-based methods use a series of manually written rules to match pronouns with their referent entities. Machine learning methods use training models to allow computers to automatically learn the relationship between pronouns and their referent entities.

[0050] Through reference resolution, we can eliminate reference ambiguity in the text, process context-related expressions such as "myself", "user name", "relative relationship", and improve the accuracy and efficiency of the natural language processing system.

[0051] In step S120 , in response to a user operation instruction, the items to be filled in are filled in.

[0052] In an exemplary embodiment of the present disclosure, after receiving a user operation instruction and capturing a screen image, a response may be made to the user operation instruction to fill in the to-be-filled items displayed in the screen image.

[0053] In an optional embodiment, entity attribute information corresponding to the entity to be filled is retrieved, and the entity attribute information is filled in the item to be filled.

[0054] To retrieve the entity attribute information corresponding to the entity to be filled, you can connect to the personal information database and search to return the entity attribute information that matches the item to be filled. This entity attribute information can be information corresponding to "relationship" and "keyword," such as "my information" or "father's ID number." The personal information database can be set up locally on the terminal or on a server, and it can be stored in an encrypted manner.

[0055] In an alternative embodiment, Figure 2 A flow chart showing a method for filling entity attribute information is shown in FIG. Figure 2 As shown, the method may include at least the following steps: in step S210, a first mapping relationship between the item to be filled and the entity attribute information is established using the second largest language model, and a second mapping relationship between the position to be filled and the entity attribute information is generated, and the position to be filled is determined by the large-scale visual language model.

[0056] A language mapping relationship between the item to be filled and the entity attribute information is established using the second largest language model as the first mapping relationship. Furthermore, as shown in Table 1, since the screen image is input into the pre-trained large-scale visual language model, the pre-trained large-scale language model outputs not only the item to be filled but also the position to be filled in the retrieved entity attribute information. Therefore, a second mapping relationship between the position to be filled and the entity attribute can be generated. The second mapping relationship can be established using a key-value pair.

[0057] The to-be-filled item, the to-be-filled position, and the entity attribute information are linked together through the first mapping relationship and the second mapping relationship.

[0058] In step S220 , the first mapping relationship and the second mapping relationship are converted into operation instructions, so as to fill the entity attribute information into the to-be-filled position of the to-be-filled item according to the operation instructions.

[0059] After determining the first mapping relationship and the second mapping relationship, the first mapping relationship and the second mapping relationship can be converted into structured instructions as machine-parseable operation instructions, so as to use the operation instructions to fill the retrieved entity attribute information into the corresponding position to be filled, thereby completing the filling of the item to be filled.

[0060] When the captured screen image cannot display all the items to be filled in, the next screen image may be displayed to completely fill in all the items to be filled in.

[0061] In an alternative embodiment, Figure 3 A flow chart showing a method of continuing to fill in other items to be filled is shown in FIG. Figure 3 As shown, the method may at least include the following steps: in step S310, the next image of the screen image is displayed, and the next image displays other items to be filled that are not completely displayed.

[0062] After the items to be filled in the current screen image are filled, the next image that is not fully displayed can be automatically switched or slid to display. Alternatively, after the user confirms that the items to be filled in the current screen image are fully filled, the next image can be displayed through an interactive method such as sliding or pulling down. The next image displays other items to be filled in that are not fully displayed in the current screen image.

[0063] In step S320, in response to the user's operation instruction, continue to fill in other items to be filled in.

[0064] After the next image of the screen image is displayed, the user operation instruction can be continued to be responded to so as to completely fill in the other items to be filled in. The filling method of the other items to be filled in is the same as that in step S120 and will not be repeated here.

[0065] In step S330 , in response to another user operation instruction, other items to be filled in continue to be filled in.

[0066] After the next screen image is displayed, the user can continue to use another user operation instruction in voice or text form to send information filling requirements, such as "Continue to fill in my information", etc., to complete the other items to be filled in. The filling method of the other items to be filled in is the same as step S120 and will not be repeated here.

[0067] The following describes in detail the information filling method in the embodiment of the present disclosure in conjunction with an application scenario.

[0068] Figure 4The schematic diagram of the architecture of the information filling method in the application scenario is shown as follows: Figure 4 As shown, the technical architecture includes a form filling requirement parsing module, an entity memory acquisition module and an operation instruction generation module.

[0069] The form filling requirement parsing module includes an intent understanding unit and a multimodal form recognition unit. The intent understanding unit uses a first language model to perform semantic analysis to determine the implicit form filling requirement in the user's operation instruction and extract the identification parameters of the entities to be filled. The multimodal form recognition unit uses a large-scale visual language model to parse the form elements in the current screen image. The large-scale visual language model's input includes the screen image, and its output is a form structure description of the items to be filled.

[0070] The entity memory acquisition module includes an entity positioning unit and a memory database interface. The entity positioning unit uses a context association algorithm to extract entity reference expressions from user operation instructions to establish a corresponding relationship with an individual, organization or device. For example, when "dad" appears in the user operation instruction, the context association algorithm can be used to establish a corresponding relationship between "dad" and the standard entity to be filled in "father" to achieve normalization. In addition, a reference resolution algorithm is integrated to normalize the items to be filled in, and the context association of the entity reference is maintained through the session state management unit. The memory database interface is connected to the personal information library stored in encrypted form locally on the terminal, and can perform structured data retrieval based on the entity to be filled in to return the entity attribute information corresponding to the entity to be filled in.

[0071] The operation instruction generation module includes a field mapping engine and an instruction encapsulation unit. The field mapping engine uses the second language model to establish a language mapping relationship between the to-be-filled items and entity attribute information, and generates a set of key-value pairs containing the to-be-filled positions and entity attribute information. The instruction encapsulation unit encodes the mapping results of the field mapping engine into a machine-parseable sequence of operation instructions to complete the form filling.

[0072] When filling out a form that requires personal information, users typically have to find or memorize the relevant information, review the information, and then manually enter it, which is a cumbersome process. In scenarios where this information filling method is used, the user communicates with the smart terminal, allowing the smart terminal to collect the user's text or voice input and obtain the items to be filled in on the smart terminal's display interface.

[0073] Figure 5 A schematic diagram of an interface showing items to be filled in an application scenario is shown, such as Figure 5As shown, the interface showing the items to be filled in is a screenshot of the received user operation instruction. Among them, the items to be filled in in the screen image include "Chinese name", "surname", "given name", "nationality", "gender", "date of birth", "passport", "validity period of certificate" and "contact number".

[0074] Furthermore, the smart terminal's search module can retrieve relevant information based on user input, and complete the final information entry based on the retrieved information and the structured understanding of the screen. This can improve the efficiency of information entry and simplify the overall operation process. It can also be applied in scenarios including shopping, ordering meals, and booking tickets.

[0075] As can be seen, with the emergence of big model technology, the implementation of end-to-end form recognition and filling products has become possible. By identifying and filling personal information form pages based on multimodal big models and integrating them into the overall intelligent filling function, the system can determine the corresponding content based on a structured understanding of the screen. Compared to traditional filling solutions, all fillable fields on the page can be filled in at once, eliminating the need for users to manually locate the fill location one by one, and is compatible with more app (application) scenarios.

[0076] Correspondingly, in Figure 5 In one embodiment, when a large-scale visual language model is used to determine the items to be filled in the screen image, the output result of the large-scale visual language model may be as follows: [ { "tag": "Chinese name", "position": "658,696", "hintText": "Optional", "inputType": "TEXT", "uiAgentType": "TYPING" }, { "tag": "Surname (Pinyin)", "position": "658,811", "hintText": "eg: HAN", "inputType": "TEXT", "uiAgentType": "TYPING" }, { "tag": "Name (Pinyin)", "position": "658,927", "hintText": "such as: MEIMEI", "inputType": "TEXT", "uiAgentType": "TYPING" }, { "tag": "nationality", "position": "270,435", "hintText": "Please select your nationality", "uiAgentType": "SELECTION" }, { "tag": "gender", "position": "270,485", "hintText": "male", "uiAgentType": "TAP" }, { "tag": "gender", "position": "270,485", "hintText": "Female", "uiAgentType": "TAP" }, { "tag": "Date of Birth", "position": "270,535", "uiAgentType": "SELECTION" }, { "tag": "Certificate Type", "position": "270,585", "text": "Passport", "uiAgentType": "SELECTION" }, { "tag": "ID number", "position": "701,1272", "hintText": "Same as the number on the boarding pass", "inputType": "NUMBER", "uiAgentType": "TYPING" }, { "tag": "Certificate validity period", "position": "270,635", "hintText": "Please fill in the validity period", "uiAgentType": "SELECTION" }, { "tag": "domestic mobile phone", "position": "270,685", "uiAgentType": "SELECTION" }, { "tag": "Mobile Number", "position": "658,1503", "hintText": "Optional, used to receive airline notifications", "inputType": "NUMBER", "uiAgentType": "TYPING" } ] During the evolution of large-scale visual language model solutions, it was found that the core tasks of information filling are structured understanding and click positioning. The model has successively implemented the solutions of "injecting OCR positioning knowledge into VLM" → "VLM+OCR positioning and fusion" → "end-to-end VLM", and combined with data enhancement, the accuracy of the entire product evaluation set was increased from 69% to 98%.

[0077] Figure 6 A schematic diagram of an interface applicable to the information filling method in an application scenario is shown. Figure 6 As shown, the information filling method can be applied to the ticket booking scenario. In this ticket booking scenario, the user operation instruction "Fill in dad's information" can be used to require the tourist to fill in various information such as name, ID type, ID number and mobile phone number, and the effect of filling in all information with one click can be achieved.

[0078] Figure 7 Another interface diagram of the information filling method applicable to the application scenario is shown in FIG. Figure 7 As shown, the information filling method can be applied to shopping scenarios. In this shopping scenario, the user operation instruction "fill in the company delivery address" can be used to require the delivery address to be filled in, and the corresponding information can be filled in with one click.

[0079] In the exemplary embodiments of this disclosure, a screen structured understanding technology based on a multimodal large model breaks through the traditional solution's reliance on manual user positioning. Furthermore, a fully automated filling mechanism driven by voice interaction reconstructs the human-computer interaction paradigm. Furthermore, this information filling method boasts cross-platform and multi-scenario adaptability, breaking through the generalization bottleneck of traditional solutions.

[0080] Specifically, a multimodal large model enables deep semantic analysis of screen content and control location, effectively overcoming the limitations of traditional smart fill solutions, which require users to navigate the form. By combining visual features with text semantics, the system automatically identifies the hierarchical relationships and functional attributes of form fields, increasing the accuracy of form recognition in traditional solutions from the industry average of approximately 60% to 98%.

[0081] Innovatively integrating intelligent devices such as voice-controlled super assistants, users can trigger the full form auto-fill process through natural language commands. Compared to existing technologies that require users to confirm each field individually (an average of 5.4 touch operations per form), this achieves true "one-click" or "sentence" operation, significantly reducing the time spent completing complex forms.

[0082] Through multimodal pre-training, the model's structural understanding and localization capabilities for GUI interfaces are enhanced, enabling it to adaptively analyze the interface layout specifications of different apps. In current testing, the model achieved extremely high coverage for form recognition in mobile apps, a significant improvement over traditional rule-based solutions.

[0083] In addition, in an exemplary embodiment of the present disclosure, an information filling device is also provided. Figure 8 The schematic diagram of the structure of the information filling device is shown. The information filling device 800 may include: a data acquisition module 810 and an information filling module 820. The data acquisition module 810 is configured to capture a screen image when receiving a user operation instruction, wherein the screen image displays the items to be filled; The information filling module 820 is configured to fill in the items to be filled in in response to the user operation instruction.

[0084] In some embodiments of the present disclosure, the information filling device 800 is further configured to: an image display module configured to display a next image of the screen image, wherein the next image displays other items to be filled that have not been fully displayed; A first filling module is configured to continue filling the other to-be-filled items in response to the user operation instruction; or The second filling module is configured to continue filling the other to-be-filled items in response to another user operation instruction.

[0085] In some embodiments of the present disclosure, the information filling device 800 is further configured to: When a user operation instruction is received, the first large language model is used to extract the entity to be filled in the user operation instruction.

[0086] In some embodiments of the present disclosure, the information filling device 800 is further configured to: A context association algorithm is used to normalize the entity to be filled in, so as to obtain a normalized entity to be filled in.

[0087] In some embodiments of the present disclosure, the information filling device 800 is further configured to: A large-scale visual language model is used to determine items to be filled in the screen image.

[0088] In some embodiments of the present disclosure, the information filling device 800 is further configured to: The to-be-filled item is normalized using a reference resolution algorithm to obtain a normalized to-be-filled item.

[0089] In some embodiments of the present disclosure, the information filling module 820 is configured to: Entity attribute information corresponding to the to-be-filled entity is retrieved, and the to-be-filled item is filled with the entity attribute information.

[0090] In some embodiments of the present disclosure, the information filling module 820 is configured to: Using a second large language model, a first mapping relationship is established between the item to be filled and the entity attribute information, and a second mapping relationship is generated between a position to be filled and the entity attribute information, wherein the position to be filled is determined by the large-scale visual language model; The first mapping relationship and the second mapping relationship are converted into operation instructions, so as to fill the entity attribute information into the to-be-filled position of the to-be-filled item according to the operation instructions.

[0091] Regarding the apparatus in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0092] The present disclosure also provides a computer-readable storage medium having computer program instructions stored thereon. When the program instructions are executed by a processor, the steps of the information filling method provided by the present disclosure are implemented.

[0093] Figure 9 FIG1 is a block diagram showing another apparatus 900 for filling information according to an exemplary embodiment. For example, the apparatus 900 may be a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, etc.

[0094] Reference Figure 9 , the apparatus 900 may include one or more of the following components: a processing component 902 , a memory 904 , a power component 906 , a multimedia component 908 , an audio component 910 , an input / output interface 912 , a sensor component 914 , and a communication component 916 .

[0095] The processing component 902 generally controls the overall operation of the device 900, such as operations associated with display, phone calls, data communications, camera operation, and recording operations. The processing component 902 may include one or more processors 920 to execute instructions to perform all or part of the steps of the above-described method. In addition, the processing component 902 may include one or more modules to facilitate interaction between the processing component 902 and other components. For example, the processing component 902 may include a multimedia module to facilitate interaction between the multimedia component 908 and the processing component 902.

[0096] The memory 904 is configured to store various types of data to support the operations of the device 900. Examples of such data include instructions for any application or method operating on the device 900, contact data, phone book data, messages, pictures, videos, etc. The memory 904 can be implemented by any type of volatile or non-volatile storage device, or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk, or optical disk.

[0097] The power supply component 906 provides power to the various components of the device 900. The power supply component 906 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the device 900.

[0098] The multimedia component 908 includes a screen that provides an output interface between the device 900 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, slides, and gestures on the touch panel. The touch sensors can not only sense the boundaries of a touch or slide action, but also detect the duration and pressure associated with the touch or slide action. In some embodiments, the multimedia component 908 includes a front-facing camera and / or a rear-facing camera. When the device 900 is in an operating mode, such as a capture mode or a video mode, the front-facing camera and / or the rear-facing camera can receive external multimedia data. Each front-facing camera and the rear-facing camera can have a fixed optical lens system or have focal length and optical zoom capabilities.

[0099] The audio component 910 is configured to output and / or input audio signals. For example, the audio component 910 includes a microphone (MIC) that is configured to receive external audio signals when the device 900 is in an operating mode, such as a call mode, a recording mode, or a voice recognition mode. The received audio signals may be further stored in the memory 904 or transmitted via the communication component 916. In some embodiments, the audio component 910 also includes a speaker for outputting audio signals.

[0100] The input / output interface 912 provides an interface between the processing component 902 and peripheral interface modules, such as a keyboard, a click wheel, buttons, etc. These buttons may include but are not limited to: a home button, a volume button, a start button, and a lock button.

[0101] The sensor assembly 914 includes one or more sensors for providing various aspects of the status assessment of the device 900. For example, the sensor assembly 914 can detect the open / closed state of the device 900, the relative positioning of components, such as the display and keypad of the device 900. The sensor assembly 914 can also detect changes in the position of the device 900 or a component of the device 900, the presence or absence of user contact with the device 900, the orientation or acceleration / deceleration of the device 900, and temperature changes of the device 900. The sensor assembly 914 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. The sensor assembly 914 may also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 914 may also include an accelerometer, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0102] The communication component 916 is configured to facilitate wired or wireless communication between the apparatus 900 and other devices. The apparatus 900 can access a wireless network based on a communication standard, such as WiFi, 2G or 3G, or a combination thereof. In an exemplary embodiment, the communication component 916 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 916 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0103] In an exemplary embodiment, the apparatus 900 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the above method.

[0104] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 904 including instructions. The instructions can be executed by the processor 920 of the apparatus 900 to perform the above method. For example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0105] In addition to being an independent electronic device, the above-mentioned device may also be part of an independent electronic device. For example, in one embodiment, the device may be an integrated circuit (IC) or a chip, where the integrated circuit may be a single IC or a collection of multiple ICs. The chip may include, but is not limited to, the following types: GPU (Graphics Processing Unit), CPU (Central Processing Unit), FPGA (Field Programmable Gate Array), DSP (Digital Signal Processor), ASIC (Application Specific Integrated Circuit), SOC (System on Chip, SoC), etc. The above-mentioned integrated circuit or chip can be used to execute executable instructions (or code) to implement the above-mentioned information filling method. The executable instructions may be stored in the integrated circuit or chip, or obtained from other devices or equipment, such as the integrated circuit or chip including a processor, memory, and an interface for communicating with other devices. The executable instruction can be stored in the memory, and when the executable instruction is executed by the processor, the above-mentioned information filling method is implemented; alternatively, the integrated circuit or chip can receive the executable instruction through the interface and transmit it to the processor for execution, so as to implement the above-mentioned method.

[0106] In another exemplary embodiment, a computer program product is further provided. The computer program product includes a computer program that can be executed by a programmable device, and the computer program has a code portion for executing the above-mentioned information filling method when executed by the programmable device.

[0107] Figure 10 FIG. 1 is a block diagram of another apparatus 1000 for filling information according to an exemplary embodiment. For example, the apparatus 1000 may be provided as a server. Figure 10 The apparatus 1000 includes a processing component 1022, which further includes one or more processors, and a memory resource represented by a memory 1032 for storing instructions, such as an application, that can be executed by the processing component 1022. The application stored in the memory 1032 may include one or more modules, each corresponding to a set of instructions. In addition, the processing component 1022 is configured to execute the instructions to perform the above-described information filling method.

[0108] Device 1000 may also include a power supply component 1026 configured to perform power management of device 1000, a wired or wireless network interface 1050 configured to connect device 1000 to a network, and an input / output interface 1058. Device 1000 may operate based on an operating system stored in memory 1032.

[0109] Other embodiments of the present disclosure will readily occur to those skilled in the art after considering the specification and practicing the present disclosure. This disclosure is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art not disclosed herein. The description and examples are to be considered as exemplary only, with the true scope and spirit of the present disclosure being indicated by the following claims.

[0110] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes can be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. An information filling method, characterized in that: include: When receiving a user operation instruction, capturing a screen image, wherein the screen image displays the items to be filled in; In response to the user operation instruction, the to-be-filled item is filled.

2. The information filling method according to claim 1, characterized in that: The method further comprises: Displaying a next image of the screen image, wherein the next image displays other items to be filled that have not been fully displayed; In response to the user operation instruction, continue to fill in the other items to be filled in; or In response to another user operation instruction, continue filling in the other items to be filled in.

3. The information filling method according to claim 1, characterized in that: The method further comprises: When a user operation instruction is received, the first large language model is used to extract the entity to be filled in the user operation instruction.

4. The information filling method according to claim 3, characterized in that: After extracting the to-be-filled entity in the user operation instruction by using the first language model, the method further includes: A context association algorithm is used to normalize the entity to be filled in, so as to obtain a normalized entity to be filled in.

5. The information filling method according to claim 3, characterized in that: After capturing the screen image, the method further includes: A large-scale visual language model is used to determine items to be filled in the screen image.

6. The information filling method according to claim 5, characterized in that: After determining the items to be filled in the screen image using the large-scale visual language model, the method further includes: The to-be-filled item is normalized using a reference resolution algorithm to obtain a normalized to-be-filled item.

7. The information filling method according to claim 5, characterized in that: Filling the item to be filled includes: Entity attribute information corresponding to the to-be-filled entity is retrieved, and the to-be-filled item is filled with the entity attribute information.

8. The information filling method according to claim 7, characterized in that: Filling the entity attribute information in the to-be-filled item includes: Using a second large language model, a first mapping relationship is established between the item to be filled and the entity attribute information, and a second mapping relationship is generated between a position to be filled and the entity attribute information, wherein the position to be filled is determined by the large-scale visual language model; The first mapping relationship and the second mapping relationship are converted into operation instructions, so as to fill the entity attribute information into the to-be-filled position of the to-be-filled item according to the operation instructions.

9. An information filling device, characterized in that: include: a data acquisition module configured to capture a screen image when receiving a user operation instruction, wherein the screen image displays items to be filled; The information filling module is configured to fill in the items to be filled in response to the user operation instruction.

10. The information filling device according to claim 9, characterized in that: The device further comprises: an image display module configured to display a next image of the screen image, wherein the next image displays other items to be filled that have not been fully displayed; A first filling module is configured to continue filling the other to-be-filled items in response to the user operation instruction; or The second filling module is configured to continue filling the other to-be-filled items in response to another user operation instruction.

11. The information filling device according to claim 9, characterized in that: The device further comprises: The model extraction module is configured to extract the to-be-filled entity in the user operation instruction by using the first large language model when receiving the user operation instruction.

12. A computer-readable storage medium having computer program instructions stored thereon, characterized in that: When the program instructions are executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.

13. An electronic device, characterized in that: include: a memory having a computer program stored thereon; A processor, configured to execute the computer program in the memory to implement the steps of the method according to any one of claims 1 to 8.

Images