Data filling method, electronic device, storage medium, program product and chip

Abstract

The application relates to the technical field of terminals, in particular to a data filling method, an electronic device, a storage medium, a program product and a chip. In the method, an input operation of a user on an interface of the electronic device is detected, a data type corresponding to the input operation is acquired, for example, in the case of detecting that the user clicks an input box, the type of data required to be filled in the input box is acquired, the data corresponding to the data type is determined from a gallery knowledge graph, and the data is displayed on the interface of the electronic device. The gallery knowledge graph comprises a corresponding relationship between data and data types determined from pictures. In this way, the information entry complexity can be reduced by automatically filling the input box.

Description

Technical Field

[0001] This application relates to the field of terminal technology, and in particular to a data filling method, electronic device, storage medium, program product, and chip. Background Technology

[0002] Currently, information entry mainly includes manual filling and certificate scanning. In manual filling, users need to input a large amount of information to complete the entry. In certificate scanning, the electronic device must first respond to the user's action of turning on the camera, and then use the camera to scan the certificate held by the user or an image of a certificate selected by the user to complete the information entry. This results in a relatively high complexity for information entry. Summary of the Invention

[0003] To address the issue of high complexity in information entry, embodiments of this application provide a data filling method, an electronic device, a storage medium, a program product, and a chip.

[0004] In a first aspect, embodiments of this application provide a data filling method applied to a first electronic device, comprising: responding to a user's first input operation on a first interface, obtaining a first data type corresponding to the first input operation; determining first data corresponding to the first data type from a knowledge graph, wherein the knowledge graph includes a correspondence between data determined from images and data types; and displaying the first data on the first interface.

[0005] Compared to some solutions that rely on user input or scanning images of user-held or selected certificates using a camera, this application's embodiment uses data from images for intelligent filling, which reduces the complexity of information entry.

[0006] Furthermore, compared to methods that use data previously entered by the user in a specific application for data population, this application's embodiments enable cross-application use by intelligently filling data from images. Moreover, this application's embodiments can provide data for applications populating data for the first time, and can also provide data that the user has not previously entered, thus achieving both efficiency and completeness in data population.

[0007] It is understood that the first electronic device can display a first interface, which may include at least one input box. Thus, the first input operation can be a long press or a tap on one of the at least one input box. The first data type can be the data type corresponding to the input box that the user long presses or taps, such as name, age, and ID number.

[0008] In some implementations of the first aspect, determining the first data corresponding to the first data type from the graph knowledge graph includes: determining at least one data group corresponding to the first data type from the graph knowledge graph; displaying at least one data group; and determining the first data corresponding to the first data type from the first data group in response to a user's selection operation on the first data group in the at least one data group.

[0009] In some implementations, the first electronic device can determine the first data corresponding to the first data type from a knowledge graph. This first data can then be displayed in the input box that the user long-presses or clicks on in the first interface. In this way, the first data required for the first input operation can be directly determined and displayed, reducing the display resources of the first electronic device.

[0010] In other implementations, the first electronic device can determine a data group, such as a first data group, corresponding to the first data type based on the interface type corresponding to the first interface and the first data type corresponding to the first input operation from a knowledge graph. This first data group may include the first data. Then, the first electronic device can display this data group. Thus, by combining the interface type corresponding to the first interface and the first data type corresponding to the first input operation to display the data group, the accuracy of the displayed data group can be improved.

[0011] Since a knowledge graph can include multiple user information data from multiple users, in some implementations, the first electronic device can also determine multiple data groups corresponding to the first data type corresponding to the first input operation from the knowledge graph, and display multiple data groups, such as the first data group and the second data group, or the first data group and the third data group, or the first data group, the second data group, and the third data group, for the user to choose from. Thus, by displaying multiple data groups based on the first data type corresponding to the first input operation, the user's choice can be increased, thereby further improving the accuracy of the subsequently displayed data.

[0012] In some implementations of the first aspect, at least one data group further includes at least one of the following: a second data group, the second data group including second data, the data type of the second data being a first data type, the user to which the second data belongs is the same as the user to which the first data belongs, and the display priority of the second data is lower than the display priority of the first data; or a third data group, the third data group including third data, the data type of the third data being a first data type, and the user to which the third data belongs is different from the user to which the first data belongs.

[0013] It is understandable that the image corresponding to the first data was accessed later than the image corresponding to the second data. Therefore, the display priority of the second data is lower than that of the first data.

[0014] In some implementations, the first electronic device can display a first data group and a third data group, thus avoiding the use of expired or obsolete user information data and thereby improving the accuracy of data filling without being noticed.

[0015] In some implementations of the first aspect, the first data group also includes at least one fourth data associated with the first data, the fourth data being of a second data type, which is different from the first data type.

[0016] It is understandable that a knowledge graph can include multiple data sets and the correspondence between the data types of each data set. Furthermore, this data can be multiple user information data for each user in multiple associated users (e.g., family members in a family group). Therefore, each data group identified from the knowledge graph can include multiple associated data sets. Within each data group, multiple data sets belong to the same user. For example, the first data group can include the second user's former name, age, and ID number data; the second data group can include the current name, age, and ID number data; and the third data group can include the first user's name, age, and ID number data.

[0017] In this embodiment of the application, by forming a data group of user information data of the same user, user information data in multiple pictures can be associated, and multiple user information data of the same user can be centralized, so that multiple data can be filled in one click later.

[0018] In some implementations of the first aspect, displaying at least one data group includes: displaying at least one data group and an image group corresponding to each data group in the at least one data group, wherein the first data group includes a first image, the first image includes fifth data and the data type of the fifth data; the above data filling method further includes: updating the data type of the fifth data in the knowledge graph of the image library in response to the user's update operation on the data type of the fifth data on the first image.

[0019] In this embodiment of the application, by synchronizing the user's update operation on the image to the image library knowledge graph, the accuracy of the image library knowledge graph can be updated in a timely manner, thereby improving the accuracy of the data displayed subsequently.

[0020] In some implementations of the first aspect, the first interface includes a first input box, and detecting a first input operation on the first interface includes: detecting a first input operation on the first input box in the first interface.

[0021] In some implementations, the first input action can be a long press or a tap on the first input box.

[0022] In some implementations of the first aspect, the first interface also includes a second input box, the data type of which is a second data type, and the first data is displayed on the first interface, including: displaying the first data in the first input box and displaying the fourth data in the second input box.

[0023] It is understandable that a knowledge graph can include multiple data sets and the correspondence between the data types of each data set. Furthermore, these data sets can include multiple user information data sets of the same user. Therefore, when the first interface also includes a second input box, and the data type corresponding to the second input box is a second data type, the first electronic device can display first data in the first input box and fourth data in the second input box. The fourth data is of the second data type and is associated with the first data. That is, after the first electronic device displays at least one data group and detects a user's selection operation on one of the at least one data group, the first electronic device can display the first data in that data group on the first interface, while also displaying other data of the user to which the first data belongs on the first interface. In this way, by displaying the first data corresponding to the first data type and the fourth data associated with the first data, intelligent filling of the same user's data required to fill multiple input boxes can be achieved, thereby improving the completeness and convenience of data filling.

[0024] In some implementations of the first aspect, the first interface also includes a third input box, the data type of which is the first data type. Displaying the first data on the first interface includes: displaying the first data in the first input box and displaying the third data in the third input box.

[0025] In some implementations of the first aspect, the first interface also includes a first prompt information field corresponding to the first input box, and obtaining the first data type corresponding to the first input operation, including: determining the first data type corresponding to the first input box based on the first prompt information field corresponding to the first input box.

[0026] In some implementations, the first interface includes at least one input box and a corresponding prompt information field for each input box. For example, the first interface includes a first input box and a corresponding first prompt information field, where the first prompt information field for the first input box is "screen visible content". When a user long-presses or clicks the first input box, the first electronic device can determine the first data type corresponding to the first input box based on the first prompt information field. Thus, by displaying the input box and its corresponding prompt information on the first interface, the user can easily verify the accuracy of the entered data.

[0027] In some implementations of the first aspect, obtaining the first data type corresponding to the first input operation includes: taking a screenshot of the first interface to obtain a second image; determining the first prompt information field corresponding to the first input box from the second image; and determining the first data type corresponding to the first input box based on the first prompt information field corresponding to the first input box.

[0028] In some implementations, the first interface includes only at least one input box. For example, the first interface includes a first input box, where the first prompt message field of the first input box is "Content not visible on screen". When a user long-presses or taps the first input box, the first electronic device can take a screenshot of the first interface to obtain a second image. Then, the first electronic device can determine the first prompt message field corresponding to the first input box from the second image, and determine the first data type corresponding to the first input box based on the first prompt message field. In this way, by displaying only the input box on the first interface, the display resources of the first electronic device can be reduced.

[0029] In some implementations of the first aspect, the image includes an image in a first electronic device and / or an image in at least one second electronic device, wherein the first electronic device and the second electronic device are associated electronic devices.

[0030] It is understood that a first electronic device may include one or more electronic devices. When there are multiple first electronic devices, these devices can log in to the same electronic account to indicate that they are all considered the first electronic device.

[0031] Furthermore, in some implementations, multiple electronic devices (such as smartphones and tablets) can share the first shared image library. For example, a smartphone held by a first user can upload images to the first shared image library, and a tablet held by the first user can also upload images to the first shared image library.

[0032] In other implementations, some electronic devices (such as smartphones and smartwatches) can share a first shared image library, while other electronic devices (such as tablets and computers) can share a second shared image library. For example, a user's smartphone and smartwatch can upload images to the first shared image library, but the user's tablet and computer cannot upload images to the first shared image library; they can only upload images to the second shared image library.

[0033] In other implementations, each of the multiple electronic devices (such as smartphones, smartwatches, and tablets) can also use different shared image libraries. For example, a smartphone held by a first user can upload images to a first shared image library, a smartwatch held by the first user can upload images to a second shared image library, and a tablet held by the first user can upload images to a third shared image library.

[0034] Similarly, a second electronic device may include one or more electronic devices. In the case where a second electronic device includes multiple electronic devices, these electronic devices can log in to the same account to indicate that these electronic devices are all second electronic devices.

[0035] It is understandable that in some implementations, the first electronic device and the second electronic device can each log in to one of multiple associated accounts to indicate that the first electronic device and the second electronic device are associated electronic devices (i.e., the first user and the second user are associated users). For example, in a family group, the father's smartphone can log in to account 'a' in the associated accounts, and the child's tablet can log in to account 'b' in the associated accounts. Therefore, the father's smartphone and the child's tablet can both be considered associated electronic devices. Thus, the first electronic device and the second electronic device can share the same shared photo library. For example, the first user's smartphone can upload pictures to the first shared photo library, and the second user's smartphone can also upload pictures to the first shared photo library.

[0036] It is understood that the image library knowledge graph can also be generated based on image data of images in a local image library, image data of images in a shared image library, and image data of images in other image libraries that are accessible to the first electronic device (such as image libraries in one or more second electronic devices that are accessible to the first electronic device). Alternatively, it can be generated based on image data of images in a local image library and image data of images in other image libraries, or it can be generated based on image data of images in other image libraries and image data of images in a shared image library. This application embodiment does not make specific limitations.

[0037] Compared to some methods that use clipboard, contacts, calendar, or data entered by the user in the past as data sources for smart fill, the embodiments of this application can make full use of user information data in images as data sources for smart fill.

[0038] In a second aspect, embodiments of this application provide an electronic device, including: a memory for storing instructions executed by one or more processors of the electronic device, and a processor, which is one of the one or more processors of the electronic device, for executing any of the data filling methods in the first aspect or the various implementations provided in the first aspect.

[0039] Thirdly, embodiments of this application provide a readable storage medium storing instructions that, when executed on an electronic device, cause the electronic device to perform the data filling method described in the first aspect or any of the various implementations provided in the first aspect.

[0040] Fourthly, embodiments of this application provide a computer program product, which includes computer instructions. When executed by an electronic device, the electronic device performs a data filling method according to any one of the first aspects or various implementations provided in the first aspect.

[0041] Fifthly, embodiments of this application provide a chip including a processor coupled to a memory for executing computer programs or instructions stored in the memory, such that the chip implements the data filling method of any one of the first aspects or various implementations provided in the first aspect.

[0042] It is understood that the specific implementation methods and beneficial effects of the second to fifth aspects mentioned above can be found in the relevant descriptions of the data filling methods in the first aspect or any of the various implementation methods provided in the first aspect, and will not be repeated here. Attached Figure Description

[0043] Figure 1A According to some embodiments of this application, a schematic diagram of a scenario for a first data filling method is shown;

[0044] Figure 1B According to some embodiments of this application, a schematic diagram of a scenario for a second data filling method is shown;

[0045] Figure 1C According to some embodiments of this application, a schematic diagram of a scenario for a third data filling method is shown;

[0046] Figure 1D According to some embodiments of this application, a schematic diagram of a scenario for a fourth data filling method is shown;

[0047] Figure 1EAccording to some embodiments of this application, a scenario diagram of the fifth data filling method is shown;

[0048] Figure 1F According to some embodiments of this application, a scenario diagram of a sixth data filling method is shown;

[0049] Figure 2 According to some embodiments of this application, a schematic diagram of the software structure of a first electronic device is shown;

[0050] Figure 3A According to some embodiments of this application, a schematic diagram of the software structure of another first electronic device is shown;

[0051] Figure 3B According to some embodiments of this application, a schematic diagram of a standard problem is shown;

[0052] Figure 4 According to some embodiments of this application, a schematic diagram of a data population process is shown;

[0053] Figure 5A According to some embodiments of this application, a flowchart illustrating a method for updating a knowledge graph is shown;

[0054] Figure 5B According to some embodiments of this application, an interactive schematic diagram of updating a knowledge graph library is shown;

[0055] Figure 5C According to some embodiments of this application, a schematic diagram of a standard problem is shown;

[0056] Figure 6A According to some embodiments of this application, a schematic diagram of picture A is shown;

[0057] Figure 6B According to some embodiments of this application, a schematic diagram of the central area directly above an image is shown;

[0058] Figure 6C According to some embodiments of this application, a schematic diagram of the central area on the left side of an image is shown;

[0059] Figure 6D According to some embodiments of this application, a schematic diagram of picture A is shown;

[0060] Figure 6E According to some embodiments of this application, a schematic diagram of picture A is shown;

[0061] Figure 7According to some embodiments of this application, a method is shown for identifying multiple images with related relationships from a local image library and a shared image library using an image semantic association model.

[0062] Figure 8 According to some embodiments of this application, a schematic diagram of a graph knowledge graph is shown;

[0063] Figure 9 According to some embodiments of this application, a data group corresponding to image group 1 is shown;

[0064] Figure 10 According to some embodiments of this application, a schematic diagram of an information population process based on a knowledge graph library is shown;

[0065] Figure 11A According to some embodiments of this application, a method is shown for displaying recommended items in an input box that match the data type corresponding to the input box;

[0066] Figure 11B According to some embodiments of this application, another method is shown for displaying recommended items in an input field that match the data type corresponding to the input field.

[0067] Figure 11C According to some embodiments of this application, a schematic diagram showing data in a highlighted form is illustrated;

[0068] Figure 12 According to some embodiments of this application, a flowchart of a data filling method is shown;

[0069] Figure 13 According to some embodiments of this application, a schematic diagram of an interface 1310 for an onboarding registration form is shown;

[0070] Figure 14 According to some embodiments of this application, a schematic diagram of an interface 1410 for a park admission registration form is shown;

[0071] Figure 15 According to some embodiments of this application, a schematic diagram of a first electronic device displaying a first data group is shown;

[0072] Figure 16 According to some embodiments of this application, a schematic diagram of another first electronic device displaying a first data group is shown;

[0073] Figure 17A According to some embodiments of this application, a schematic diagram of a first electronic device displaying a first data group and a second data group is shown;

[0074] Figure 17BAccording to some embodiments of this application, a schematic diagram of a first electronic device displaying a first data group and a third data group is shown.

[0075] Figure 17C According to some embodiments of this application, a schematic diagram is shown of a first electronic device displaying a first data group, a second data group, and a third data group;

[0076] Figure 18 According to some embodiments of this application, a schematic diagram is shown of a first electronic device displaying multiple data groups and a group of images corresponding to each data group;

[0077] Figure 19A According to some embodiments of this application, a schematic diagram of a first electronic device displaying first data is shown;

[0078] Figure 19B According to some embodiments of this application, a schematic diagram is shown of a first electronic device displaying first data and associated data of the first data;

[0079] Figure 19C According to some embodiments of this application, a schematic diagram of a first electronic device displaying first data and third data is shown;

[0080] Figure 19D According to some embodiments of this application, a schematic diagram is shown of a first electronic device displaying first data, associated data of the first data, third data, and associated data of the third data;

[0081] Figure 20 According to some embodiments of this application, a schematic diagram of the structure of a first electronic device is shown. Detailed Implementation

[0082] The embodiments of this application include, but are not limited to, a data filling method, an electronic device, a storage medium, a program product, and a chip.

[0083] It is understood that the data filling method mentioned in the embodiments of this application can be applied to electronic devices. In some implementations, electronic devices may also be referred to as terminals, user equipment (UE), mobile stations (MS), mobile terminals (MT), etc. For example, electronic devices may be mobile phones, smart TVs, wearable devices, tablets, computers with wireless transceiver capabilities, virtual reality (VR) terminal devices, augmented reality (AR) terminal devices, wireless terminals in industrial control, wireless terminals in self-driving, wireless terminals in remote medical surgery, wireless terminals in smart grids, wireless terminals in transportation safety, wireless terminals in smart cities, wireless terminals in smart homes, and other devices with intelligent filling functions.

[0084] Furthermore, the data filling method mentioned in the embodiments of this application can be applied to scenarios that require filling in user information data in application or web page forms, such as account registration, account login, ticket purchase, registration, application, registration, and payment.

[0085] like Figure 1A As shown, when the electronic device 100 detects a user's smart fill operation, for example, when it detects that the user long-presses or clicks the account input box 111 in the application login interface 110 displayed by the electronic device 100, the electronic device 100 can determine the data to be filled in the account input box 111 from the database and display the data to be filled. Figure 1B As shown, the electronic device 100 can display the account data "123" in the account input box 111. In this way, intelligent filling of the data required for a single input box can be achieved, thereby improving the accuracy of data filling.

[0086] like Figure 1CAs shown, when the electronic device 100 detects a smart filling operation of the user, for example, detecting that the user long-presses or clicks the name input box 121 in the first web form interface 120 displayed by the electronic device 100, the electronic device 100 may determine the to-be-filled data required to be filled in the name input box 121, the to-be-filled data required to be filled in the unique identity identifier input box 122, and the to-be-filled data required to be filled in the family identity identifier input box 123 from the database, and display these to-be-filled data. As shown in Figure 1D As shown, the electronic device 100 may display the name data "A" in the name input box 121, display the unique identity identifier data "234" in the unique identity identifier input box 122, and display the family identity identifier data "345" in the family identity identifier input box 123. In this way, the smart filling of data of the same user required to be filled in multiple input boxes can be realized, so as to improve the integrity and convenience of data filling.

[0087] As shown in Figure 1E As shown, in the case where the electronic device 100 detects the smart filling operation of the user, for example, detecting that the user long-presses or clicks the student name input box 131 in the second web form interface 130 displayed by the electronic device 100, the electronic device 100 may determine the to-be-filled data required to be filled in the student name input box 131 and the to-be-filled data required to be filled in the parent name input box 132 from the database, and display these to-be-filled data. As shown in Figure 1F As shown, the electronic device 100 may display the student name data "C" in the student name input box 131, and display the parent name data "D" in the parent name input box 132. In this way, the smart filling of data of multiple users required to be filled in multiple input boxes can be realized, so as to improve the comprehensiveness and intelligence of data filling.

[0088] In order to solve the above problem of relatively high complexity of information entry, embodiments of the present application provide a data filling method. In this method, when an operation of the user clicking an input box is detected, the data type of the data required to be filled in the input box is acquired, the data corresponding to the data type is determined from the gallery knowledge graph, and the data corresponding to the data type is displayed. The gallery knowledge graph includes the correspondence between a plurality of data determined from the pictures in the gallery and the data type of each data in the plurality of data. In this way, by automatically filling the input box, the complexity of information entry can be reduced.

[0089] It can be understood that the gallery knowledge graph may be generated according to the image data of the pictures in the local gallery and the image data of the pictures in the shared gallery.

[0090] In some implementations, such as in a scenario where a first user populates information on their electronic device, the local gallery can refer to the gallery on the first user's electronic device. For ease of explanation, the first electronic device held by the first user will be referred to as the first electronic device below.

[0091] A shared image library can refer to an image library in the cloud. This shared image library may include images uploaded by a first electronic device and images uploaded by at least one second user's electronic device. The first and second users can be associated users, for example, family members in a family group. For ease of explanation, the electronic device held by the second user will be referred to as the second electronic device below.

[0092] Image data can include user information data related to the user, as well as the data type of the user information data. The data type can include names, ages, and ID numbers, etc. Image data can also include image type data related to images; for example, the image type data corresponding to an image containing a unique identifier is "Unique Identifier," the image type data corresponding to an image containing learning records is "Learning Records," and the image type data corresponding to an image containing a payment card is "Payment Card," etc.

[0093] It is understood that the first electronic device may include one or more electronic devices. In the case where the first electronic device comprises multiple electronic devices, these multiple electronic devices can log in to the same electronic account to indicate that they are all the first electronic device.

[0094] Furthermore, in some implementations, multiple electronic devices (such as smartphones and tablets) can share the first shared image library. For example, a smartphone held by a first user can upload images to the first shared image library, and a tablet held by the first user can also upload images to the first shared image library.

[0095] In other implementations, some electronic devices (such as smartphones and smartwatches) can share a first shared image library, while other electronic devices (such as tablets and computers) can share a second shared image library. For example, a user's smartphone and smartwatch can upload images to the first shared image library, but the user's tablet and computer cannot upload images to the first shared image library; they can only upload images to the second shared image library.

[0096] In other implementations, each of the multiple electronic devices (such as smartphones, smartwatches, and tablets) can also use different shared image libraries. For example, a smartphone held by a first user can upload images to a first shared image library, a smartwatch held by the first user can upload images to a second shared image library, and a tablet held by the first user can upload images to a third shared image library.

[0097] Similarly, a second electronic device may include one or more electronic devices. In the case where a second electronic device includes multiple electronic devices, these electronic devices can log in to the same account to indicate that these electronic devices are all second electronic devices.

[0098] It is understandable that in some implementations, the first electronic device and the second electronic device can each log in to one of multiple associated accounts to indicate that the first electronic device and the second electronic device are associated electronic devices (i.e., the first user and the second user are associated users). For example, in a family group, the father's smartphone can log in to account 'a' in the associated accounts, and the child's tablet can log in to account 'b' in the associated accounts. Therefore, the father's smartphone and the child's tablet can both be considered associated electronic devices. Thus, the first electronic device and the second electronic device can share the same shared photo library. For example, the first user's smartphone can upload pictures to the first shared photo library, and the second user's smartphone can also upload pictures to the first shared photo library.

[0099] It is understood that the knowledge graph of the image library can be generated based on image data of images in a local image library, or based on image data of images in a shared image library, or based on image data of images in a local image library, image data of images in a shared image library, and image data of images in other image libraries that are accessible to the first electronic device (e.g., image libraries in one or more second electronic devices that are accessible to the first electronic device). It can also be generated based on image data of images in a local image library and image data of images in other image libraries, or based on image data of images in other image libraries and image data of images in a shared image library. This application does not specifically limit the scope of the knowledge graph.

[0100] The following example illustrates the data filling method mentioned in the embodiments of this application, using a scenario where a first user fills in information on a first electronic device as an example.

[0101] It is understood that the data filling method mentioned in the embodiments of this application can be applied to the first electronic device. Furthermore, the electronic device provided in the embodiments of this application can run an operating system (OS), which can be various operating systems used in the industry, such as those based on… Developed operating systems, such as Or other operating systems, such as Mobile operating system; it can also be various open-source operating systems or their derivatives, such as Linux. This includes other embedded operating systems; it can also refer to future new operating systems, such as AI operating systems based on artificial intelligence. An operating system is a set of interconnected system software programs that manage and control the operation of electronic devices, utilize and run hardware and software resources, and provide public services to organize user interaction. In electronic devices, the operating system connects downwards to the physical devices at the hardware layer and upwards to provide a runtime environment for application software.

[0102] An operating system typically includes a kernel layer, a middleware layer, and an application layer. The application layer includes applications, which can include system applications and third-party applications. The middleware layer includes a suite of software providing various services to application developers, or frameworks providing services such as databases, multimedia, and graphics, or capabilities such as distributed scheduling and system scaling. For example, the middleware layer may include a framework layer and / or a system service layer. The framework layer provides application programming interfaces (APIs) and programming frameworks for applications in the application layer. The system service layer includes the system's core capabilities, providing services to applications through the framework layer. The kernel layer is the layer between hardware and software. The kernel layer may include hardware drivers and the operating system kernel. In addition to providing hardware drivers, the kernel layer also supports functions such as memory management and system process management.

[0103] The electronic devices we use in our daily lives come in various types and forms, and are applied in a wide range of scenarios. Therefore, based on the different forms and functions of electronic devices, different application scenarios, and different user needs, the operating systems used in these devices may also differ. The basic functions implemented by the electronic device provided in this application can be achieved through a general-purpose operating system or a dedicated operating system.

[0104] To more clearly illustrate the implementation of the embodiments of this application under a specific operating system, such as Figure 2 As shown, it illustrates Based on the architecture, those skilled in the art can deduce the implementation of the embodiments of this application under other specific operating systems, such as... Implementation under operating systems, etc.

[0105] The software architecture of electronic devices can be divided into several layers. In some embodiments, from bottom to top, these layers are: kernel layer, system service layer, framework layer, and application layer. Layers communicate with each other through software interfaces. System functions can be tailored, added, or combined at the subsystem level depending on the deployment scenario of different device forms. Each subsystem can also be tailored, added, or combined at the functional level.

[0106] Continue as Figure 2 As shown, the kernel abstract layer (KAL) provides basic kernel capabilities to the upper layers by shielding the differences between multiple kernels, including but not limited to process / thread management, memory management, file system, network management, and peripheral device management.

[0107] Kernel subsystem: Supports selecting appropriate kernels for different resource-constrained devices. Kernel, not limited to kernel, kernel, (lite operating system), etc.

[0108] Driver Subsystem: The driver framework is the foundation for the open system hardware ecosystem, providing unified peripheral access capabilities and a framework for driver development and management. The driver framework includes: display drivers, camera drivers, audio drivers, Bluetooth drivers, sensor drivers, etc.

[0109] Continue as Figure 2 As shown, the system service layer comprises the core capabilities of the system, providing services to applications through the framework layer. This layer includes, but is not limited to, the following subsystems:

[0110] The system's basic capability subsystem set provides fundamental capabilities for the operation, scheduling, and migration of distributed applications across multiple devices. This set may include distributed soft bus, distributed data management, distributed task scheduling, and Ark multi-language runtime; it may also include multi-modal input subsystem, graphics subsystem, security subsystem, and AI subsystem.

[0111] Basic software service subsystem set: provides public and general software services; the basic software service subsystem set may include event notification subsystem, telephone service subsystem, multimedia subsystem, etc.

[0112] Enhanced software service subsystem suite: Provides differentiated enhanced software services for different devices; the enhanced software service subsystem suite may include smart screen proprietary business subsystem, wearable proprietary business subsystem, IoT proprietary business subsystem, etc.

[0113] Hardware service subsystem set: Provides hardware services; the hardware service subsystem set may include location service subsystem, unified identity and access management (IAM) subsystem, wearable proprietary hardware service subsystem, biometric identification, IoT proprietary hardware service subsystem, etc.

[0114] Distributed task scheduling enables distributed service management (discovery, synchronization, registration, and invocation), supporting remote startup, remote invocation, remote connection, and migration of applications across devices.

[0115] Distributed data management enables data synchronization, data storage, data sharing, and data access across all scenarios and devices.

[0116] The distributed soft bus provides communication-related capabilities for seamless interconnection between multiple devices, including: WLAN service capabilities, Bluetooth service capabilities, soft bus, inter-process communication (remote procedure call, RPC), and StarFlash communication capabilities.

[0117] Ark Multilingual Runtime is a unified compilation runtime platform designed to support the joint compilation and execution of multiple programming languages ​​and multiple chip platforms.

[0118] Smart fill services include, for example Figure 3A The services shown include the knowledge graph service, page understanding service, and recommendation item construction service.

[0119] The image library knowledge graph service stores the image library knowledge graph and reads data changes from image libraries (such as the local and shared image libraries mentioned earlier) to update the knowledge graph. The image library knowledge graph can include at least one of the following: user relationships, relationships between image groups and images, relationships between users and image groups, relationships between image groups and data groups, non-sensitive data, and sensitive data. To ensure the security of the data in the image library knowledge graph, the service can encrypt and manage the data. For example, the service can use a relational database (RDB) provided by ArkData to encrypt and store user relationships, relationships between image groups and images, relationships between users and image groups, relationships between image groups and data groups, and non-sensitive data. Alternatively, the service can use an asset database (ASSET) to encrypt and store sensitive data.

[0120] The page understanding service is used to parse the data type of all input fields in the first electronic device. For ease of explanation, the data type of the data to be entered in the input fields will be referred to as the data type corresponding to the input field in the following text.

[0121] The recommendation building service is used to construct standard questions based on the data types corresponding to all input boxes, and to obtain intelligent fill recommendation groups.

[0122] It is understandable that the smart fill service is a system service. In this way, users can obtain the same recommendations from the knowledge graph of the image library in different applications, thereby realizing cross-application smart fill and information recommendation and providing users with a better operating experience.

[0123] The large model service is used to intelligently fill in recommendation groups (i.e., recommendations below) based on standard questions, determining the data type corresponding to the input box from the knowledge graph. For example... Figure 3B As shown, a schematic diagram of the structure of a standard problem is presented.

[0124] This standard question may include:

[0125] Please generate populated content for the form attributes of this person based on the provided set of attribute names for the input boxes to be filled in, one or more image groups of personal information for the current person, and the recent usage data of the image groups.

[0126] A collection of attribute names for form input fields: {a collection of attribute names}.

[0127] Several image groups contain attribute information and recent usage data: [{"Personal Information": {<attribute, attribute value> set, "Recent Usage Data": usage}, {Other Image Groups}, ...].

[0128] Based on the above information, please provide one or more sets of recommendation information with a populated form. The attribute names of each set of recommendations should correspond one-to-one with the attributes of the form input boxes. Multiple sets of recommendations should be sorted by the sum of the recent frequency data of the image groups from which the information originates. The recommendation information should be output in JSON format. The JSON data should contain nested lists, each list representing a set of recommendations. Each set of recommendations should be a dictionary structure, where the key is the name of the form input box attribute and the value is the predicted attribute value.

[0129] By leveraging large model services based on standard questions, intelligent fill recommendation groups corresponding to the data types of input boxes can be determined from the graph knowledge graph. This effectively utilizes the graph for intelligent fill, thereby seamlessly improving the scenario coverage of intelligent fill.

[0130] Continue as Figure 2As shown, the framework layer provides application programming interfaces and a programming framework for the application layer. The framework layer includes: the ArkUI Smart Fill framework. The ArkUI Smart Fill framework is a smart fill framework developed based on ArkUI, providing applications with complete infrastructure, including UI functions such as components, layouts, animations, and interactive events, as well as a real-time interface preview tool; a user application framework; and an Ability framework (an Ability is a lightweight application; the Ability framework schedules and manages the operation and lifecycle of Abilities). Different devices may have different operating systems, and the APIs they support may also differ. The ArkUI Smart Fill framework is used to launch the smart fill service after detecting a user's long press on an input field.

[0131] The HarmonyOS API is a series of open capabilities provided to support HarmonyOS application development. The HarmonyOS API can be set at the framework layer or independently of the framework layer. The HarmonyOS API includes AudioAPI (audio service), PushAPI (push service), AccountAPI (account service), and more.

[0132] Applications can include system apps and extended / third-party apps. System apps can include app services / meta services, gallery, desktop, control bar, settings, contacts, phone, camera, etc. Extended / third-party apps can include social apps, travel apps, etc.

[0133] Application services / meta services can include data population functionality. In some implementations, during the development of application services / meta services, developers can set up ArkUI smart input controls, such as input boxes, to provide users with an intuitive and convenient input method.

[0134] Continue as Figure 3A As shown, the image library can include a local image library and a shared image library. The local image library can refer to the image library on the first electronic device. The shared image library can refer to an image library in the cloud, and can include images uploaded by the first electronic device and images uploaded by at least one second electronic device.

[0135] It is understandable that the data population process can include the process of generating a knowledge graph for the library and the process of data population based on the knowledge graph service for the library.

[0136] like Figure 4As shown, the process of generating a knowledge graph for an image library can include several sub-processes, such as identifying image types from images, grouping images, determining key names from each image group, determining the relationship between users and image groups, and updating recent usage data. The specific implementation of each process will be detailed below.

[0137] The data population process based on the knowledge graph of the image library can include detecting the smart fill operation, recalling the user's associated image group, recalling the form's associated image group, building a smart fill recommendation group, and having the user select a recommended item from the recommendation group to fill the data. The specific implementation of each process will be detailed below.

[0138] The following section introduces the process of generating a knowledge graph for a graph library.

[0139] In some methods of generating knowledge graphs for image libraries, the knowledge graph service can read all or part of the images (e.g., images of certificates or forms containing user information) from local and shared image libraries, and identify the user information data, the data type of the user information data, and the image type from each image. Then, based on the image type and user information data of each image, the knowledge graph service can group multiple images of the same type belonging to the same user into multiple image groups. For example, the knowledge graph service can group the front and back images representing the same certificate into one group, the left and right halves of the same certificate into another group, and the top and bottom halves of the same certificate into yet another group. Next, the knowledge graph service can associate the user information data and the data type of the user information data within each image group to generate the knowledge graph. For example, the knowledge graph can include user relationships, relationships between image groups and images, relationships between users and image groups, and relationships between image groups and data groups.

[0140] The following section introduces the update process of the knowledge graph in the graph library.

[0141] It is understandable that whenever any changes occur to the images in the gallery, such as storing new images, accessing images, modifying images, or deleting images, the gallery knowledge graph service will update the gallery knowledge graph.

[0142] In some ways of updating the knowledge graph of a graph library, such as Figure 5AIn the scenario depicted where a first user populates information on a first electronic device, the first electronic device can update the images in its local image library in response to the first user's first operation. For example, if the first operation is a shooting operation (taking a picture and automatically saving it), the first electronic device can store the image in its local image library and determine the time of taking the picture or the time of storing the picture as the access time of that image. If the first operation is a modification operation (editing an image), the first electronic device can delete the original image from its local image library and store the modified image, determining the time of completing the modification or the time of storing the modified image as the access time of the modified image. If the first operation is a deletion operation (deleting an image), the first electronic device can delete the image from its local image library.

[0143] Upon detecting the second user's second operation, the second electronic device can respond to the second operation and send an operation instruction corresponding to the second operation to the cloud. This operation instruction instructs the images in the shared image library to be updated. It is understood that the method for updating images in the shared image library is the same as the method for updating images in the local image library; to avoid repetition, it will not be elaborated here.

[0144] In some implementations, during the scenario where a first user populates information on a first electronic device, after updating images in both the local and shared image libraries, the image library knowledge graph service can read all images from both libraries and identify user information data, the data type of that information, and the image type from each image. Then, the service can group images based on their image type and user information data; for example, grouping multiple images of the same image type from the same user into multiple image groups. Next, the service can associate and store the user information data and data type of each image within each group to achieve a comprehensive update of the image library knowledge graph. This avoids missing any user information data and improves the completeness of the knowledge graph.

[0145] In other implementations, in scenarios where a first user populates information on a first electronic device, after updating images in the local image library and the shared image library, the image library knowledge graph service can read the changed images from both the local and shared image libraries (e.g., changed ID photos containing user information, changed form photos containing user information, etc.), and identify the user information data, its data type, and the image type from each image. Then, the image library knowledge graph service can group images based on their image type and user information data; for example, grouping multiple images of the same image type from the same user into multiple image groups. Next, the service can associate and store the user information data and its data type within each image group to update the changed data in the image library knowledge graph. This reduces the processing resources used by the first electronic device and the update time for the image library knowledge graph, improving update efficiency.

[0146] The following is combined Figure 2 and Figure 5B This section provides further details on updating the knowledge graph of the graph library. For example... Figure 5B The diagram illustrates an interactive schematic for updating a knowledge graph in a graph database. In a scenario where a user populates information on a first electronic device, this method of updating the knowledge graph can be executed by the first electronic device, continuing as follows... Figure 2 As shown, the first electronic device may include a gallery and a gallery knowledge graph service. In some implementations, the method for updating the gallery knowledge graph may include:

[0147] S501: The gallery detected a first user's deletion operation on the deleted image set.

[0148] It's understandable that deleting an image set can include multiple images. For example, the image set to be deleted might include image 1, image 2, and image 3. In some implementations, if the gallery detects that a first user has deleted multiple images from the gallery, the gallery can determine these images as the image set to be deleted. For example, if the gallery detects that a first user has deleted image 1, image 2, and image 3 from the gallery, the gallery can determine image 1, image 2, and image 3 as the image set to be deleted.

[0149] S502: The gallery detected a new operation by the first user on the new image collection.

[0150] It's understandable that a new image set can include multiple images. For example, the new image set might include images 7, 8, and 9. In some implementations, if the gallery detects that the first user took a photo and automatically saves multiple images, the gallery can identify these images as part of the new image set. For example, if the gallery detects that the first user took a photo and automatically saves images 7, 8, and 9, the gallery can identify images 7, 8, and 9 as part of the new image set.

[0151] S503: The library sends a command to the library knowledge graph service to start a timer.

[0152] S504: The knowledge graph service is waiting for N seconds or the page understanding service is detected to be started. Stop waiting.

[0153] S505: Timer to close the knowledge graph service.

[0154] S506: The image library knowledge graph service sends a command to the image library to read the newly added image set.

[0155] S507: The image library returns a newly added image set to the image library knowledge graph service.

[0156] S508: The image library knowledge graph service updates the image library knowledge graph based on newly added image sets.

[0157] In some implementations, the image library knowledge graph service can create subgraphs based on multiple images in a newly added image set. For example, the image library knowledge graph service can identify user information data, the data type of user information data, and the image type from images 7, 8, and 9, respectively. Then, the image library knowledge graph service can classify the multiple images based on the image type and user information data of each image. For example, the image library knowledge graph service can group images 7, 8, and 9 based on their image types and user information data. Next, the image library knowledge graph service can associate the user information data and the data type of user information data for each image group to obtain the subgraph corresponding to the newly added image set, and update or add the subgraph corresponding to the newly added image set in the image library knowledge graph.

[0158] S509: The image library knowledge graph service sends an instruction to the image library to read and delete the image set.

[0159] S510: The image library returns the set of deleted images to the image library knowledge graph service.

[0160] S511: The image library knowledge graph service updates the image library knowledge graph based on the deleted image set.

[0161] In some implementations, the image library knowledge graph service can create subgraphs based on multiple images in a set of deleted images. For example, the image library knowledge graph service can identify user information data, the data type of the user information data, and the image type from images 1, 2, and 3, respectively. Then, the image library knowledge graph service can group the multiple images based on the image type and user information data of each image. For example, the image library knowledge graph service can group images 1, 2, and 3 based on their image types and user information data. Next, the image library knowledge graph service can associate the user information data and the data type of the user information data for each image group to obtain the subgraph corresponding to the set of deleted images, and then delete the subgraph corresponding to the set of deleted images in the image library knowledge graph.

[0162] It is understood that the above-described methods for updating the knowledge graph are merely one example of embodiments of this application, and these methods can also be executed in other orders. For example, the execution order of S501 and S502 can be either S501 first and then S502, or S502 first and then S501; the execution order of S506 and S509 can be either S506 first and then S509, or S509 first and then S506; the execution order of S508 and S511 can be either S508 first and then S511, or S511 first and then S508, or S508 and S511 can be executed in parallel.

[0163] In some implementations, the knowledge graph service can send standard questions to the large model service to construct subgraphs using the large model service. The following section will combine... Figure 5C The standard questions and Figure 5B The method of updating the knowledge graph of the image library shown introduces the standard questions of updating the subgraph corresponding to the image set and deleting the subgraph corresponding to the image set.

[0164] This standard question may include:

[0165] This program processes a collection of images including ID card images, personal information images, and other images. It identifies the ID card or personal information form images, determines the ID card or personal information form type, groups images belonging to the same ID card or personal information form, identifies the main person in the ID card or personal information form, extracts personal information, and outputs the data in JSON format. Note that the input images may contain multiple ID card images or multiple personal information form images, or may not contain any ID card images or images of any individual. The output consists only of JSON data. Output example: [

[0167] {

[0168] "Image group type": "Certificate type 1"

[0169] "Main Character": "A"

[0170] "personal information":{

[0171] Name: "A"

[0172] "Date of Birth": "xxxx-xx-xx",

[0173] "Identity / Permit Number": "xxxxxxx"

[0174] "Expiration Date": "xxxx-xx-xx"

[0175] },

[0176] "List of Source Image Numbers": [1,2,3]

[0177] }

[0178] {

[0179] "Image Group Type": "xxx Registration Form"

[0180] "Main Character": "B"

[0181] "personal information":{

[0182] Name: "B"

[0183] Grade: "xx"

[0184] Class: "xx"

[0185] Enrollment Date: "xxxx-xx-xx"

[0186] },

[0187] "List of Source Image Numbers": [5,6,7]

[0188] } ]

[0190] Image collection: {actual image collection}

[0191] The following section describes the image type recognition implementation method in the image library knowledge graph generation process and the image library knowledge graph update process.

[0192] Among the methods for identifying image types from images, image knowledge graph services can utilize optical character recognition (OCR) algorithms to extract all text blocks within an image. These text blocks can be semantically related segments of characters, such as... Figure 6A The text block in image A is an example of a "family group identifier." A text block can also be a contiguous string of semantically related characters with clearly defined boundaries, such as region 610 in image A containing the "family group identifier." Furthermore, the image library knowledge graph service can determine the image type from the text block based on regular expression matching, text classification neural network models, or object detection algorithms. For example, the image library knowledge graph service can determine that the image type of image A is "Resident XX Registration Card."

[0193] Understandably, usually, the central area at the top of the image (such as...) Figure 6B The area shown is 620) or the center area on the left side of the image (e.g. Figure 6C The area shown (630) contains the image title, and the image title can indicate the image type. Therefore, in methods of identifying the image type from an image, the image library knowledge graph service can use optical character recognition algorithms to extract the text block corresponding to the image title and determine the image type from the text block. In some implementations, the image library knowledge graph service can use the image title as the image type. For example, in some exemplary implementations, the image library knowledge graph service can determine... Figure 6D The image type of image B shown is "Resident XX Registration Card". For example, the image library knowledge graph service can determine... Figure 6E The image type in image C shown is "XX New Student Registration Form".

[0194] The following section introduces the methods for grouping images.

[0195] In some image grouping methods, image library knowledge graph services can identify multiple related images from local and shared image libraries based on the first user's historical behavior. For example, multiple images whose access times fall within a preset time difference range can be identified as related images.

[0196] For example, if image A is stored in the local image library at the first moment of a certain day, month, and year, and image B is stored at the second moment of the same day, with the time difference between the first and second moments falling within a preset shooting time difference interval, then the image semantic association model can determine that images A and B are related and classify them as image group AB. If image C is stored in the local image library at the third moment of the same day, and the time difference between the first and third moments falls outside the preset shooting time difference interval, and the time difference between the second and third moments also falls outside the preset shooting time difference interval, then the image semantic association model can determine that images A and C are not related. Furthermore, the image semantic association model can determine that images B and C are also not related. Thus, the image library knowledge graph service can group image C separately, for example, image group C.

[0197] It is understandable that in some implementations, each image group has an image group identifier (identity, ID). This image group identifier can include the image identifier of each image within that group.

[0198] In other image grouping methods, image library knowledge graph services can use image semantic association models to identify multiple images with related relationships from local and shared image libraries.

[0199] For example, such as Figure 7 As shown, the image semantic association module can be a neural network model, which may include an image encoder, an attention encoder, and a classifier. In some implementations, the image encoder can perform text detection on image A to determine text blocks from image A. Next, the image encoder can encode the text blocks or consecutive characters within the text blocks to obtain a character encoding sequence. Then, the image encoder can extract text features from the character encoding sequence to extract the semantic features of the text blocks or consecutive characters within the text blocks. Furthermore, the image encoder can perform image feature extraction on image A to obtain image features of image A, such as the font size and font color of the text blocks or consecutive characters within the text blocks. After determining the semantic features of the text blocks or consecutive characters within the text blocks in image A, and the image features of image A, the image encoder can perform image-text feature fusion to obtain the fused features of image A.

[0200] Similarly, the image encoder may perform the same operation on image B as that performed on image A to obtain a fused feature of image B. Further, the neural network model may concatenate the fused feature of image A and the fused feature of image B, and input the concatenated features into an attention encoder. The attention encoder may determine an associated feature of image A and image B according to the fused feature of image A and the fused feature of image B, and input the associated feature into a classifier. Further, the classifier may determine whether image A and image B are images having an association relationship according to the associated feature of image A and image B.

[0201] It can be understood that after a plurality of images having an association relationship are grouped into one group, the gallery knowledge graph service may send each image group to the large model service. Further, the large model service may determine a key name from each image group, so as to determine the relationship between the user and the image group. The key name may be a name that uniquely appears in the plurality of images of the image group, the key name may also be a name that appears multiple times in the plurality of images of the image group, the key name may also be the first name that appears from top to bottom in the image with the latest access time among the plurality of images of the image group, and the key name may also be a name that appears in a region corresponding to the image type of the image in the image group. For example, for the image group AB enumerated above, the key name of the image group AB is "Jia", and the key name of the image group C is "Yi".

[0202] Further, the gallery knowledge graph service may generate a gallery knowledge graph or update the gallery knowledge graph according to the relationship between the first user and the second user stored in the first electronic device and the determined key name in each image group. For example, the gallery knowledge graph may include the relationship between the user and the image group. For example, as Figure 8 shown, the first user may correspond to image group 1, image group 2 ... image group M; the second user a may correspond to image group 11, image group 12 ... image group N, and the second user a may have a first relationship with the first user; the second user b may correspond to image group 21, image group 22 ... image group P, and the second user b may have a second relationship with the first user.

[0203] Moreover, each image group may correspond to one data group. In some implementations, the data group may be stored in the form of an attribute table, where the attribute table may include a plurality of data and a data type corresponding to each of the plurality of data. For example, as Figure 9As shown, a data group corresponding to image group 1 is illustrated. Image group 1 may include front and back images of the same document. The data group corresponding to image group 1 may include name data corresponding to data type "Name", date of birth data corresponding to data type "Date of Birth", address data corresponding to data type "Address", unique identity data corresponding to data type "Unique Identifier", validity period data corresponding to data type "Validity Period", and recent usage data corresponding to data type "Recent Frequency Data".

[0204] It is understandable that recent usage data can represent the display priority of a data group. The later the access time of the image group corresponding to the data group, the larger the recent usage data, and the higher the display priority of the data group. In this embodiment, by prioritizing the display of data with large recent usage data, the use of expired and obsolete data can be avoided, thereby perceptibly improving the recommendation accuracy of intelligent data filling.

[0205] For example, if a first user uploads an image of personal information form A or certificate B to the shared image library, and this image contains data c1 of data type c, then when the system detects that the first user has long-pressed an input box in the web form interface 120, the corresponding data type of that input box is c. Thus, the input box in the web form interface 120 can display data c1, and it can also display all data in the data group to which data c1 belongs.

[0206] For example, if a first user uploads an image of personal information form A or certificate B to the shared image library, where the image of personal information form A contains data d1 of data type d, and the image of certificate B contains data d2 of data type d, and the recent usage data of the data group to which data d2 belongs is higher than the recent usage data of the data group to which data d1 belongs, then the input boxes in the web form interface 120 can display data d1, and the input boxes in the web form interface 120 can also display all the data of the data group to which data d2 belongs.

[0207] For example, if the first user deletes an image from Personal Information Form A or Certificate B in the shared gallery, and that image contains data c1 of data type c, then when the system detects that the first user has long-pressed an input box in the web form interface 120, the input box will not display data c1, nor will it display all the data in the data group to which data c1 belongs.

[0208] It's understandable that the recent usage data of the data group corresponding to the image group can be continuously updated. In determining the recent usage data of the data group corresponding to the image group, the image library knowledge graph service can count the number of times images within the image group have been accessed within a preset statistical period, such as the number of times images within the image group have been modified or opened. Based on the number of times images within the image group have been accessed within the preset statistical period, the recent usage data of the data group corresponding to the image group can be determined.

[0209] Understandably, data in a data group can also have recent usage data. For example, the later the access time of the image from which the data originates, the greater the recent usage data of the data, and the higher the display priority of the data.

[0210] In other implementations, data groups can be stored in the form of associations. For example, a data group can be: <first user, user relationship, user>, <user, image type, image group identifier>, <image group identifier, containing, image identifier>, <image identifier, association, data group>, <data group, data type, data>.

[0211] In this embodiment, by identifying image types and grouping images, determining key names from each image group, establishing the relationship between users and image groups, and updating recent usage data, a knowledge graph of multiple associated users is constructed. This solves the problem of user information data being scattered across images in the image library and thus unusable. Furthermore, by grouping images, images without names can be associated with images with names, improving the data integrity of the knowledge graph.

[0212] The following is combined Figure 3A and Figure 10 This paper introduces the process of information population based on knowledge graph.

[0213] It's understandable that during the development of application services / meta-services, developers can configure the prompt information fields and information type fields for ArkUI smart fill input controls (such as input boxes). The information type field can include "visible content" and "invisible content." For example, a developer can set the prompt information fields for a name input box to include "Please enter your name" and "name," where the information type field for "Please enter your name" is "visible content," and the information type field for "name" is "invisible content."

[0214] Thus, when the ArkUI Smart Fill framework detects a smart fill operation by the first user on the first electronic device, such as detecting the first user clicking an input box, the ArkUI Smart Fill framework can invoke the page understanding service within the smart fill service. Furthermore, in some implementations, the page understanding service can read the prompt information field and information type field of the input box to determine the data type corresponding to the input box. For example, the page understanding service can read the prompt information field "Please enter your name" where the information type field of the input box is "screen visible content," and determine that the data type corresponding to the input box is "name input box." In other implementations, the page understanding service can call the screenshot service to take a screenshot of the first electronic device, generating a screen snapshot (corresponding to...). Figure 10 ① Generate a screen snapshot. Furthermore, the page understanding service can parse the input box's prompt information field and information type field from the screen snapshot (e.g., the left or upper left side of the input box). Figure 10 ② Parsing out the prompt information field and information type field of the input box, we can obtain the data type corresponding to the input box (corresponding to...). Figure 10 (③ in the text) Determine the data type of the data to be filled in the input box. For example, the page understanding service can read the information type field of the input box and the prompt field "name" which is "screen invisible content", and determine that the data type of the input box is "name input box".

[0215] For example, continue as Figure 10 As shown, the first electronic device can display a form to be filled, which may include a first input box 1010, a second input box 1020, a third input box 1030, and a fourth input box 1040. The prompt information field of the first input box 1010 is "Field 1," and the information type field is "Screen visible content." This prompt information field is used to guide the first user to fill in the data to be filled. The prompt information field of the second input box 1020 is "Field 2," and the information field type is also "Screen visible content."

[0216] After determining the data type of the input field, the page understanding service can send the data type to the recommendation item construction service. The recommendation item construction service can then read the knowledge graph (corresponding to...) Figure 10 (④ Reading) Calculate the data type of the data to be filled and the attribute correlation of each picture group for each family member, and recall all picture groups associated with form attributes for each family member; based on all picture groups associated with form attributes for each family member, their recent usage data, and form screenshots, construct a standard question structure for each family member (corresponding to...). Figure 10 (⑤ Constructing the standard problem) yields recommendations for each family member (also known as a data set, corresponding to...) Figure 10(⑥ Recommended item). Therefore, the input box of the first electronic device, which controls the recommendation item construction service, displays the recommended item (corresponding to...). Figure 10 (Seventh option in the text shows recommended items).

[0217] Furthermore, when the system detects that the first user has selected a data group from the recommended items, the ArkUI smart fill framework can fill the data from the selected data group into the corresponding input box, thus enabling one-click automatic form filling.

[0218] It is understandable that in some ways of displaying recommended data groups, such as Figure 11A As shown, the first electronic device can display recommended items 1110 in the input box that match the data type corresponding to the input box. The recommended items 1110 may include data groups that match the data type corresponding to the input box.

[0219] For example, in the scenario where A fills in the address registration form, the recommended item 1110 may include A's name data "A", A's unique identification data "123", A's family identification data "234", B's name data "A", B's unique identification data "345", and B's family identification data "567".

[0220] In other ways of displaying recommended data groups, such as Figure 11B As shown, the first electronic device can display recommended items 1110 that match the data type corresponding to the input box in the input box. The recommended items 1110 may include a data group matching the data type corresponding to the input box and a group of images corresponding to the data group.

[0221] For example, in the scenario where A fills in the address registration form, the recommended item 1110 may include A's name data "A", A's unique identification data "123", A's family identification data "234", and which image in the image library A's name data, A's unique identification data, and A's family identification data are from. Similarly, B's name data "A", B's unique identification data "345", B's family identification data "567", and which image in the image library B's name data, B's unique identification data, and B's family identification data are from.

[0222] The first electronic device displays as follows Figure 11B When the recommended items are displayed, if a viewing action by the first user is detected, such as detecting the first user clicking on the source to expand component 1111, the first electronic device can respond to the first user's viewing action by displaying multiple images from the image group selected in the recommended items 1110, and the first electronic device can highlight them (e.g.) Figure 11CThe shading (in the form of 1112) displays the data included in each image on each image, and the data type corresponding to the data is displayed in the area surrounding the data. Alternatively, the first electronic device can display the data included in each image on each image, and highlight the data type corresponding to the data in the area surrounding the data. Furthermore, the first user can annotate new data and its corresponding data type on the images, delete old data and its corresponding data type, modify the data type corresponding to the data, etc. In this way, the image library knowledge graph service can update the image library knowledge graph service in response to the first user's modification operations.

[0223] The following is combined Figure 12 The data filling method mentioned in the embodiments of this application will be described.

[0224] like Figure 12 The diagram illustrates a flowchart of a data population method. It can be understood that this data population method can be executed by a first electronic device. In some implementations, the data population method may include:

[0225] S1201: In response to the user's first input operation on the first interface, obtain the first data type corresponding to the first input operation.

[0226] It is understood that the first electronic device can display a first interface, which may include at least one input box. The first input operation may be a long press on one of the at least one input box, or it may be a click on one of the at least one input box; this embodiment does not specifically limit the operation. The first data type may be the data type corresponding to the input box that the user long presses or clicks, such as data types like name, age, and ID number.

[0227] In some implementations, when a user long-presses or clicks an input box on the first interface, the ArkUI smart fill framework in the first electronic device can launch the smart fill service. The page understanding service in the smart fill service can parse the data type corresponding to the input box that the user long-pressed or clicked.

[0228] For example, in a scenario where a user fills out a web form corresponding to their onboarding registration form, the first interface could be the onboarding registration form interface. For instance, such as... Figure 13 As shown, the first electronic device can display an interface 1310 for an employment registration form. This interface 1310 may include a first input box 1301, a second input box 1302, and a third input box 1303. The data type corresponding to the first input box 1301 is a first address, the data type corresponding to the second input box 1302 is an educational qualification, and the data type corresponding to the third input box 1303 is contact information.

[0229] For example, in a scenario where a user fills out a web form corresponding to their child's kindergarten registration form, the first interface could be the kindergarten registration form interface. For instance, such as... Figure 14 As shown, the first electronic device can display an interface 1410 for the kindergarten registration form. This interface 1410 may include a first input box 1401, a second input box 1402, and a third input box 1404. The data type corresponding to the first input box 1401 is the first address, the data type corresponding to the second input box 1402 is the parent's name, and the data type corresponding to the third input box 1403 is contact information.

[0230] It is understandable that each input field in at least one input field has a corresponding prompt information field, and the information type field of the prompt information field can be either "visible content on screen" or "invisible content on screen".

[0231] In some implementations, the first interface includes at least one input box and a corresponding prompt information field for each input box. For example, the first interface includes a first input box and a corresponding first prompt information field, where the first prompt information field for the first input box is "screen visible content". When a user long-presses or clicks the first input box, the first electronic device can determine the first data type corresponding to the first input box based on the first prompt information field. Thus, by displaying the input box and its corresponding prompt information on the first interface, the user can easily verify the accuracy of the entered data.

[0232] In some implementations, the first interface includes only at least one input box. For example, the first interface includes a first input box, where the first prompt message field of the first input box is "Content not visible on screen". When a user long-presses or taps the first input box, the first electronic device can take a screenshot of the first interface to obtain a second image. Then, the first electronic device can determine the first prompt message field corresponding to the first input box from the second image, and determine the first data type corresponding to the first input box based on the first prompt message field. In this way, by displaying only the input box on the first interface, the display resources of the first electronic device can be reduced.

[0233] S1202: Determine the first data corresponding to the first data type from the graph knowledge graph, wherein the graph knowledge graph includes the correspondence between data determined from the image and data types.

[0234] It is understandable that in some implementations, the first electronic device can determine the first data corresponding to the first data type from a knowledge graph, and this first data can then be displayed in the first input box of the first interface. In this way, the first data required for the first input operation can be directly determined and displayed, which can reduce the display resources of the first electronic device.

[0235] Understandably, in some implementations, the first electronic device can determine at least one data group corresponding to the first data type from the knowledge graph and display at least one data group for the user to select. The at least one data group may include the first data group. Upon detecting a user's selection operation on any one of the at least one data group—for example, when the first electronic device displays at least one data group in a pop-up window and detects that the user clicks on a data group (e.g., the first data group) in the pop-up window—the first electronic device can respond to the user's selection operation on that data group by determining the first data corresponding to the first data type from that data group.

[0236] In some implementations, the first electronic device can determine a data group corresponding to the first data type from a knowledge graph, such as a first data group, which may include the first data, based on the interface type corresponding to the first interface and the first data type corresponding to the first input operation. Then, the first electronic device can display this data group. Thus, by combining the interface type corresponding to the first interface and the first data type corresponding to the first input operation to display the data group, the accuracy of the displayed data group can be improved.

[0237] Since a knowledge graph can include multiple data sets and the correspondence between the data types of each data set, and these data sets can include multiple user information data sets of the same user, the first data set can also include at least one fourth data set associated with the first data set. This fourth data set can be of a second data type, which is different from the first data type. In this embodiment, by grouping the user information data of the same user into a data set, user information data from multiple images can be associated, centralizing multiple user information data sets of the same user, thereby enabling one-click filling of multiple data sets subsequently.

[0238] For example, when a user long press is detected... Figure 13 In the case of the first input box 1301 in the interface 1310 shown, the first electronic device can determine that the interface type corresponding to the interface 1310 is an employment registration form, the first data type corresponding to the first input box 1301 is a first address, and the first electronic device can determine, based on the interface type and the first data type, a first data group containing the user's own user information data from the knowledge graph. Furthermore, as shown... Figure 15 as shown, the first electronic device may display the first data set in the form of a pop-up window 1510, where the first data set may include the user's own user information data. For example, the name is "A 1", the residential address is "XXXXXX", and the contact information is "12345678".

[0239] For another example, in response to detecting that the user long-presses, such as Figure 14 the first input box 1401 in the interface 1410 as shown, the first electronic device may determine that the interface type corresponding to the interface 1410 is a park admission registration form, the first data type corresponding to the first input box 1401 is the first address, and the first electronic device may determine, based on the interface type and the first data type, a first data set containing the user information data of the child from the gallery knowledge graph. And as Figure 16 shown, the first electronic device may display the first data set in the form of a pop-up window 1610, where the first data set may include the user information data of the child. For example, the name is "A 2", the residential address is "XXXXXX", and the date of birth is "YY / MM / DD".

[0240] Because the gallery knowledge graph may include a correspondence between a plurality of data and the data type of each of the plurality of data. Moreover, these data may include a plurality of pieces of user information data for each of a plurality of users. Therefore, in some other implementations, the first electronic device may also determine, based on the first data type corresponding to the first input operation, a plurality of data sets corresponding to the first data type from the gallery knowledge graph, for example, a first data set and a second data set, for another example, a first data set and a third data set, and for another example, a first data set, a second data set, and a third data set. The first data set may include first data. The second data set may include second data, the data type of the second data may be the first data type, the user to which the second data belongs is the same as the user to which the first data belongs, and the access time of the picture corresponding to the first data is later than the access time of the picture corresponding to the second data, such that the display priority of the second data is lower than the display priority of the first data. The third data set may include third data, the data type of the third data may be the first data type, and the user to which the third data belongs is different from the user to which the first data belongs. In this way, by displaying a plurality of data sets according to the first data type corresponding to the first input operation, the selectability for the user can be improved, thereby further improving the accuracy of the subsequently displayed data.

[0241] Similarly, the second data set may also include data associated with the second data, and the data type of this data is different from the first data type. The third data set may also include data associated with the third data, and the data type of this data is also different from the first data type. In the embodiments of the present application, by forming a data set from the user information data of the same user, the user information data in multiple pictures can be associated, and multiple pieces of user information data of the same user can be aggregated, so that one-click filling of multiple pieces of data can be realized subsequently.

[0242] For example, when it is detected that the user long-presses as Figure 14 the first input box 1401 in the interface 1410 shown, the first electronic device may determine that the first data type corresponding to the first input box 1401 is the first address, and the first electronic device may determine, according to the first data type, at least one of the first data set containing the child's user information data, the second data set containing the child's user information data, and the third data set containing the user's own user information data from the gallery knowledge graph.

[0243] Further, as Figure 17A shown, the first electronic device may display the first data set and the second data set in the form of a pop-up window 1710 for the user to select. The first data set may be arranged above the second data set to indicate that the display priority of the first data is higher than the display priority of the second data. The first data set may include the child's user information data, for example, the name is "Jia 2", the residential address is "XXXXXX", and the date of birth is "YY year YY month YY day". The second data set may also include the child's user information data, for example, the name is "Jia 2", the residential address is "ZZZZZZ", and the date of birth is "YY year YY month YY day". Wherein, XXXXXX may be the current residential address, and ZZZZZZ may be the former residential address.

[0244] As Figure 17B shown, the first electronic device may also display the first data set and the third data set in the form of a pop-up window 1720 for the user to select. The first data set may include the child's user information data, for example, the name is "Jia 2", the residential address is "XXXXXX", and the date of birth is "YY year YY month YY day". The third data set may include the user's own user information data, for example, the name is "Jia 1", the residential address is "XXXXXX", and the contact information is "12345678".

[0245] In the embodiments of the present application, by displaying the first data set and the third data instead of displaying the second data set containing expired and discarded user information data (such as the former residential address mentioned above), the use of expired and discarded user information data can be avoided, so that the accuracy of data filling can be improved imperceptibly.

[0246] As Figure 17C shown, the first electronic device may also display the first data set, the second data set and the third data set in the form of a popup 1730 for the user to select. The first data set may include the child's user information data, for example, the name is "A2", the residential address is "XXXXXX", and the date of birth is "YY Year YY Month YY Day". The second data set may also include the child's user information data, for example, the name is "A2", the residential address is "ZZZZZZ", and the date of birth is "YY Year YY Month YY Day". The third data set may include the user's own user information data, for example, the name is "A1", the residential address is "XXXXXX", and the contact information is "12345678".

[0247] It can be understood that, in some implementations, after the first electronic device determines at least one data set corresponding to the first data type from the gallery knowledge graph, the first electronic device may not only display the at least one data set, but also display the image set corresponding to each data set in the at least one data set.

[0248] For example, as Figure 18 shown, the first electronic device may display the first data set, the first image set corresponding to the first data set, the third data set, and the third image set corresponding to the third data set. The first data set may include the child's user information data, for example, the name is "A2", the residential address is "XXXXXX", and the date of birth is "YY Year YY Month YY Day". The image set corresponding to the first data set may include an image of the child's unique identity identifier. The third data set may include the user's own user information data, for example, the name is "A1", the residential address is "XXXXXX", and the contact information is "12345678". The image set corresponding to the third data set may include an image of the user's own payment card and an image of the unique identity identifier.

[0249] Moreover, after the first electronic device displays the at least one data set and the image set corresponding to each data set in the at least one data set, in the case of detecting an update operation on the data type of the data on a certain image in the image set corresponding to a certain data set by the user, the first electronic device may update the data type of the data in the gallery knowledge graph in response to the user's update operation on the image.

[0250] For example, the first electronic device may display at least one data group and an image group corresponding to each data group in the at least one data group, where a first data group in the at least one data group includes a first image, and the first image includes fifth data and a data type of the fifth data. Upon detecting a user's update operation on the data type of the fifth data in the first image, the first electronic device may update the data type of the fifth data in the gallery knowledge graph in response to the user's update operation on the data type of the fifth data on the first image. For example, the data type of the fifth data "A1" in the first image is "name", and in the case where it is detected that the user updates the data type "name" of the fifth data "A1" to "parent name", the first electronic device may update the data type of "A1" in the gallery knowledge graph to "parent name", so as to realize the update of the gallery knowledge graph. In this way, by synchronizing the user's update operation on the image to the gallery knowledge graph, the accuracy of the gallery knowledge graph can be updated in a timely manner, so that the accuracy of subsequently displayed data can be improved.

[0251] In other implementations, the first electronic device may update the gallery knowledge graph based on a preset time, for example, automatically update the gallery knowledge graph at zero o'clock every day. In this way, since the load of the first electronic device in the early morning is relatively low, performing complex tasks has a relatively small overall impact on the system, and users have less activities in the early morning, updating the gallery knowledge graph in the early morning will not interfere with the normal use of users. In addition, the images in the gallery change less in the early morning, so updating the gallery knowledge graph in the early morning can avoid frequent updates of the gallery knowledge graph.

[0252] In addition, the first electronic device may store the gallery knowledge graph for periodic scheduled full update, perform periodic real-time incremental update on the gallery knowledge graph, or update the gallery knowledge graph in time when a change occurs in the gallery, which is not specifically limited in the embodiments of the present application.

[0253] S1203: Display first data on a first interface.

[0254] It can be understood that in some implementations, after determining the first data corresponding to the first data type from the gallery knowledge graph, the first electronic device may display the first data on the first interface. In this way, by displaying the first data corresponding to the first data type, that is, realizing the intelligent filling of data required to be filled in a single input box, the accuracy of data filling can be improved, and the display resources of the first electronic device can be reduced.

[0255] For example, as shown in Figure 19A , the first electronic device may display the first address data "XX" with the data type being "first address".

[0256] It is understandable that a knowledge graph can include multiple data sets and the correspondence between the data types of each data set. Furthermore, these data sets can include multiple user information data sets of the same user. Therefore, when the first interface also includes a second input box, and the data type corresponding to the second input box is a second data type, the first electronic device can display first data in the first input box and fourth data in the second input box. The fourth data is of the second data type and is associated with the first data. That is, after the first electronic device displays at least one data group and detects a user's selection operation on one of the at least one data group, the first electronic device can display the first data in that data group on the first interface, while also displaying other data of the user to which the first data belongs on the first interface. Thus, by displaying the first data corresponding to the first data type and the fourth data associated with the first data, intelligent filling of data for the same user that needs to be filled in multiple input boxes can be achieved, improving the completeness and convenience of data filling.

[0257] For example, such as Figure 19B As shown, the first electronic device can display the first address data "XX" in the input box with the data type "first address", display the second address data "XX" in the input box with the data type "second address", and display the third address data "XX" in the input box with the data type "third address".

[0258] It is understandable that a knowledge graph can include multiple data sets and the correspondence between the data types of each data set. Furthermore, these data sets can include multiple user information data for each user among multiple users. Therefore, when the first interface also includes a third input box, and the data type corresponding to the third input box is a first data type, the first electronic device can display the first data in the first input box and the third data in the third input box. The data type of this third data is the first data type. That is, after the first electronic device displays at least one data group and detects a user's selection operation on one of the at least one data group, the first electronic device can display the first data in that data group on the first interface while simultaneously displaying the data of the user to whom the third data belongs on the first interface. In this way, by displaying the first data belonging to one user and the third data belonging to another user corresponding to the first data type, intelligent filling of data for multiple users that needs to be filled in multiple input boxes can be achieved, thereby improving the comprehensiveness and intelligence of data filling.

[0259] For example, such as Figure 19C As shown, the first electronic device can display the first address data "XX" belonging to the child in one input box with the data type "first address", and display the first address data "XX" belonging to the father in another input box with the data type "first address".

[0260] It is understandable that the first electronic device can also display first data in the first input box, third data in the third input box, and related data of the first data and related data of the third data in other data boxes. For example, such as Figure 19D As shown, the first electronic device can display the second address data "XX" in the input box with the data type "second address", the third address data "XX" in the input box with the data type "third address", the parent name data "A1" in the input box with the data type "parent name", the family relationship data "father" in the input box with the data type "family relationship", and the contact information data "12345678" in the input box with the data type "contact information".

[0261] Compared to some solutions that rely on user input or scanning images of certificates held by the user or selected by the user using a camera, this application embodiment uses data from the image for intelligent filling, which can reduce the complexity of information entry.

[0262] Furthermore, compared to methods that use data previously entered by the user in a specific application for data population, this application's embodiments enable cross-application use by intelligently filling data from images. Moreover, this application's embodiments can provide data for applications populating data for the first time, and can also provide data that the user has not previously entered, thus achieving both efficiency and completeness in data population.

[0263] It is understood that the data filling method mentioned in the embodiments of this application can be applied to the first electronic device. The hardware structure of the first electronic device will be described below.

[0264] Figure 20 A schematic diagram of the hardware structure of a first electronic device is shown. It is understood that the structure illustrated in this embodiment does not constitute a specific limitation on the first electronic device. In other embodiments of this application, the first electronic device may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0265] like Figure 20As shown, the first electronic device may include a processor 2010, an external memory interface 2020, an internal memory 2021, a universal serial bus (USB) interface 2030, a charging management module 2040, a power management module 200, a battery 2042, an antenna 1, an antenna 2, a mobile communication module 2050, a wireless communication module 2060, an audio module 2070, a speaker 2070A, a receiver 2070B, a microphone 2070C, a headphone jack 2070D, a sensor module 2080, buttons 2090, a motor 2091, an indicator 2092, a camera 2093, a display screen 2094, and a subscriber identification module (SIM) card interface 2095, etc. The sensor module 2080 may include a pressure sensor 2080A, a gyroscope sensor 2080B, a barometric pressure sensor 2080C, a magnetic sensor 2080D, an accelerometer sensor 2080E, a distance sensor 2080F, a proximity sensor 2080G, a fingerprint sensor 2080H, a temperature sensor 2080J, a touch sensor 2080K, an ambient light sensor 2080L, a bone conduction sensor 2080M, etc.

[0266] The processor 2010 may include one or more processing units, such as an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural network processing unit (NPU). These different processing units may be independent devices or integrated into one or more processors.

[0267] The controller can generate operation control signals based on the instruction opcode and timing signals to complete the control of instruction fetching and execution.

[0268] The processor 2010 may also include memory for storing instructions and data. In some embodiments, the memory in the processor 2010 is a cache memory. This memory can store instructions or data that the processor 2010 has just used or that are used repeatedly. If the processor 2010 needs to use the instruction or data again, it can retrieve it directly from the memory. This avoids repeated accesses, reduces the processor 2010's waiting time, and thus improves system efficiency.

[0269] In some embodiments, the processor 2010 may include one or more interfaces. Interfaces may include integrated circuit interfaces, integrated circuit built-in audio interfaces, pulse code modulation (PCM) interfaces, universal asynchronous receiver / transmitter (UART) interfaces, mobile industry processor interfaces (MIPI), general-purpose input / output (GPIO) interfaces, subscriber identity module (SIM) interfaces, and / or universal serial bus (USB) interfaces, etc.

[0270] The I2C interface is a bidirectional synchronous serial bus, including a serial data line (SDA) and a serial clock line (SCL). In some embodiments, the processor 2010 may include multiple I2C buses. The processor 2010 can couple to the touch sensor 2080K, charger, flash, camera 2093, etc., through different I2C bus interfaces. For example, the processor 2010 can couple to the touch sensor 2080K through the I2C interface, enabling the processor 2010 and the touch sensor 2080K to communicate through the I2C bus interface, thereby realizing the touch function of the first electronic device.

[0271] The USB 2030 interface is a USB standard compliant interface, specifically a Mini USB interface, Micro USB interface, or USB Type-C interface. The USB 2030 interface can be used to connect a charger to charge a primary electronic device, or to transfer data between the primary electronic device and peripheral devices. It can also be used to connect headphones for audio playback. This interface can also be used to connect other primary electronic devices, such as AR devices.

[0272] It is understood that the interface connection relationships between the modules illustrated in the embodiments of the present invention are merely illustrative and do not constitute a structural limitation on the first electronic device. In other embodiments of this application, the first electronic device may also employ different interface connection methods or combinations of multiple interface connection methods as described in the above embodiments.

[0273] The charging management module 2040 receives charging input from a charger. The charger can be a wireless charger or a wired charger. In some wired charging embodiments, the charging management module 2040 receives charging input from the wired charger via a USB interface 2030. In some wireless charging embodiments, the charging management module 2040 receives wireless charging input via the wireless charging coil of the first electronic device. While charging the battery 2042, the charging management module 2040 can also supply power to the first electronic device via the power management module 200.

[0274] The power management module 200 connects the battery 2042, the charging management module 2040, and the processor 2010. The power management module 200 receives input from the battery 2042 and / or the charging management module 2040, providing power to the processor 2010, internal memory 2021, display screen 2094, camera 2093, and wireless communication module 2060, etc. The power management module 200 can also monitor parameters such as battery capacity, battery cycle count, and battery health status (leakage current, impedance). In some other embodiments, the power management module 200 may also be located within the processor 2010. In other embodiments, the power management module 200 and the charging management module 2040 may be housed in the same device.

[0275] In some implementations, the power management module 200 can supply power to the audio module 2070 so that the audio module 2070 can switch from a power-off stopped state to a power-on working state.

[0276] The first electronic device implements display functions through a GPU, a display screen 2094, and an application processor. The GPU is a microprocessor for image processing, connecting the display screen 2094 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 2010 may include one or more GPUs, which execute program instructions to generate or modify display information.

[0277] The display screen 2094 is used to display images, videos, etc. The display screen 2094 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a Mini LED, a MicroLED, a Micro-OLED, a quantum dot light-emitting diode (QLED), etc. In some embodiments, the first electronic device may include one or N displays 2094, where N is a positive integer greater than 1.

[0278] The first electronic device can achieve the shooting function through an ISP, camera 2093, video codec, GPU, display 2094, and application processor.

[0279] The ISP (Image Signal Processor) is used to process data fed back from the camera 2093. For example, when taking a picture, the shutter is opened, and light is transmitted through the lens to the camera's photosensitive element. The light signal is converted into an electrical signal, and the camera's photosensitive element transmits the electrical signal to the ISP for processing, transforming it into an image visible to the naked eye. The ISP can also perform algorithmic optimization on image noise and brightness. The ISP can also optimize parameters such as exposure and color temperature of the shooting scene. In some embodiments, the ISP can be set in the camera 2093.

[0280] Camera 2093 is used to capture still images or videos. An object is projected onto a photosensitive element by generating an optical image through the lens. The photosensitive element can be a charge-coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the light signal into an electrical signal, which is then passed to an ISP for conversion into a digital image signal. The ISP outputs the digital image signal to a DSP for processing. The DSP converts the digital image signal into image signals in standard RGB, YUV, or other formats. In some embodiments, the first electronic device may include one or N cameras 2093, where N is a positive integer greater than 1.

[0281] Digital signal processors (DSPs) are used to process digital signals. Besides digital image signals, they can also process other digital signals. For example, when a first electronic device selects a frequency, a DSP can perform Fourier transforms on the frequency energy.

[0282] Video codecs are used to compress or decompress digital video. A first electronic device may support one or more video codecs. Thus, the first electronic device can play or record video in various encoded formats, such as Moving Picture Experts Group (MPEG) 1, MPEG 2, MPEG 3, MPEG 4, etc.

[0283] An NPU (Neural Processing Unit) is a neural network (NN) computing processor that, by borrowing the structure of biological neural networks, such as the transmission patterns between neurons in the human brain, rapidly processes input information and can continuously learn on its own. NPUs enable intelligent cognitive applications in electronic devices, such as image recognition, facial recognition, speech recognition, and text understanding.

[0284] The external storage interface 2020 can be used to connect external storage cards, such as Micro SD cards, to expand the storage capacity of the first electronic device. The external storage card communicates with the processor 2010 through the external storage interface 2020 to perform data storage functions. For example, music, video, and other files can be saved on the external storage card.

[0285] Internal memory 2021 can be used to store computer executable program code, which includes instructions. Internal memory 2021 may include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function (such as sound playback, image playback, etc.), etc. The data storage area may store data created during the use of the first electronic device (such as audio data, phonebook, etc.). Furthermore, internal memory 2021 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, universal flash storage (UFS), etc. Processor 2010 executes various functional applications and data processing of the first electronic device by running instructions stored in internal memory 2021 and / or instructions stored in memory disposed in the processor.

[0286] The first electronic device can implement audio functions, such as music playback and recording, through an audio module 2070, a speaker 2070A, a receiver 2070B, a microphone 2070C, a headphone jack 2070D, and an application processor.

[0287] The audio module 2070 is used to convert digital audio information into analog audio signals for output, and also to convert analog audio input into digital audio signals. The audio module 2070 can also be used for encoding and decoding audio signals. In some embodiments, the audio module 2070 may be located in the processor 2010, or some functional modules of the audio module 2070 may be located in the processor 2010.

[0288] The speaker 2070A, also known as a "loudspeaker," is used to convert audio electrical signals into sound signals. The primary electronic device can use the speaker 2070A to listen to music or make hands-free calls.

[0289] This application provides a computer program product that, when run on a device, causes the device to execute the technical solutions described in the above embodiments. Its implementation principle and technical effects are similar to those of the related embodiments described above, and will not be repeated here.

[0290] This application provides a readable storage medium containing instructions that, when executed on a device, cause the device to perform the technical solutions described in the above embodiments. The implementation principle and technical effects are similar and will not be repeated here.

[0291] This application provides a chip for executing instructions. When the chip is running, it executes the technical solutions described in the above embodiments. Its implementation principle and technical effects are similar and will not be repeated here.

[0292] It is understood that the various embodiments of the mechanisms disclosed in this application can be implemented in hardware, software, firmware, or a combination of these implementation methods. Embodiments of this application can be implemented as computer programs or program code executable on a programmable system, the programmable system including at least one processor, a storage system (including volatile and non-volatile memory and / or storage elements), at least one input device, and at least one output device.

[0293] Program code can be applied to input instructions to execute the functions described in this application and generate output information. The output information can be applied to one or more output devices in a known manner. For the purposes of this application, the processing system includes any system having a processor such as, for example, a digital signal processor (DSP), a microcontroller, an application-specific integrated circuit (ASIC), or a microprocessor.

[0294] The program code can be implemented using a high-level procedural language or an object-oriented programming language to communicate with the processing system. Assembly language or machine language can also be used when needed. In fact, the mechanisms described in this application are not limited to any particular programming language. In either case, the language can be a compiled language or an interpreted language.

[0295] It should be noted that in the examples and description of this patent, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one" does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes the aforementioned element.

[0296] Although this application has been illustrated and described with reference to certain embodiments thereof, those skilled in the art will understand that various changes in form and detail may be made thereto without departing from the scope of this application.

Claims

1. A data filling method, characterized in that, Applied to a first electronic device, including: In response to the user's first input operation on the first interface, obtain the first data type corresponding to the first input operation; The first data corresponding to the first data type is determined from the image knowledge graph, wherein the image knowledge graph includes the correspondence between data determined from the image and data types; The first data is displayed on the first interface.

2. The method according to claim 1, characterized in that, The step of determining the first data corresponding to the first data type from the knowledge graph includes: Identify at least one data group corresponding to the first data type from the knowledge graph of the graph library; Display the at least one data group; In response to a user's selection operation of a first data group in the at least one data group, the first data corresponding to the first data type is determined from the first data group.

3. The method according to claim 2, characterized in that, The at least one data group further includes at least one of the following: The second data group includes second data, the data type of the second data is the first data type, the user to which the second data belongs is the same as the user to which the first data belongs, and the display priority of the second data is lower than the display priority of the first data; or The third data group includes third data, the data type of the third data is the first data type, and the user to which the third data belongs is different from the user to which the first data belongs.

4. The method according to claim 3, characterized in that, The first data group also includes at least one fourth data associated with the first data, the fourth data being of a second data type, which is different from the first data type.

5. The method according to claim 4, characterized in that, Displaying the at least one data group includes: Display the at least one data group and the image group corresponding to each data group in the at least one data group, wherein the first data group includes a first image, and the first image includes fifth data and the data type of the fifth data; The method further includes: In response to the user's update operation on the data type of the fifth data in the first image, the data type of the fifth data in the knowledge graph of the image library is updated.

6. The method according to claim 5, characterized in that, The first interface includes a first input box. The response to the user's first input operation on the first interface includes: In response to the user's first input operation on the first input box in the first interface.

7. The method according to claim 6, characterized in that, The first interface also includes a second input box, the data type of which is the second data type. Displaying the first data on the first interface includes: The first data is displayed in the first input box, and the fourth data is displayed in the second input box.

8. The method according to claim 6 or 7, characterized in that, The first interface also includes a third input box, the data type of which is the first data type. Displaying the first data on the first interface includes: The first data is displayed in the first input box, and the third data is displayed in the third input box.

9. The method according to claim 6, characterized in that, The first interface also includes a first prompt information field corresponding to the first input box. The step of obtaining the first data type corresponding to the first input operation includes: Based on the first prompt information field corresponding to the first input box, the first data type corresponding to the first input box is determined.

10. The method according to claim 6, characterized in that, The step of obtaining the first data type corresponding to the first input operation includes: A screenshot of the first interface is taken to obtain a second image; The first prompt information field corresponding to the first input box is determined from the second image; Based on the first prompt information field corresponding to the first input box, the first data type corresponding to the first input box is determined.

11. The method according to any one of claims 1 to 10, characterized in that, The images include images from the first electronic device and / or images from at least one second electronic device, wherein the first electronic device and the second electronic device are associated electronic devices.

12. An electronic device, characterized in that, Includes: a memory for storing instructions executed by one or more processors of the electronic device. And a processor, which is one or more processors of the electronic device, for performing the data filling method according to any one of claims 1 to 11.

13. A readable storage medium, characterized in that, The readable storage medium stores instructions that, when executed on an electronic device, cause the electronic device to perform the data filling method according to any one of claims 1 to 11.

14. A computer program product, characterized in that, The computer program product includes computer instructions that, when executed by an electronic device, enable the electronic device to perform the data filling method as described in any one of claims 1 to 11.

15. A chip, characterized in that, The chip includes a processor coupled to a memory for executing a computer program or instructions stored in the memory, such that the chip implements the data filling method according to any one of claims 1 to 11.

Images