Information processing method

Abstract

The invention discloses an information processing method, and the method comprises the steps: obtaining a description node tree corresponding to a current display interface in response to an instruction for indicating to obtain description information of the display interface, the description node tree comprising at least one node used for representing interface elements on the display interface and a hierarchical relationship between different nodes, different nodes correspond to different interface elements; taking each node in the description node tree as a directory entry, taking a node description text of the node as the content of the directory entry, and generating interface directory information for expressing the display interface based on the hierarchical relationship among the nodes in the description node tree; and feeding back the interface directory information.

Description

Technical Field

[0001] This application relates to the field of information processing technologies, and in particular, to an information processing method. Background Art

[0002] With the continuous development of artificial intelligence technologies, it is often necessary to perform task processing based on the content of the display interface of an electronic device. For example, generating an abstract for describing the display interface content based on the content of the display interface of the electronic device; or, in response to a user's task instruction, combining the display interface content in the electronic device and planning the operation steps required to complete the task indicated by the task instruction with the help of an artificial intelligence service, etc. Based on this, how to more intuitively express the content of the display interface is the key to reliably completing the relevant task processing. Summary of the Invention

[0003] On the one hand, this application provides an information processing method, including:

[0004] In response to an instruction indicating to obtain display interface description information, obtaining a description node tree corresponding to the current display interface, where the description node tree includes: at least one node for representing an interface element on the display interface and a hierarchical relationship between different nodes, and different nodes correspond to different interface elements;

[0005] Taking each node in the description node tree as a table of contents entry and taking the node description text of the node as the content of the table of contents entry, generating interface table of contents information for expressing the display interface based on the hierarchical relationship between the nodes in the description node tree;

[0006] Feeding back the interface table of contents information.

[0007] In a possible implementation manner, before generating the interface table of contents information, it further includes:

[0008] If a node in the description node tree meets a target condition, determining a target interface area associated with the interface element represented by the node in the display interface, and generating a node description text for the node based on the display content in the target interface area to update the description node tree;

[0009] Wherein, the node meeting the target condition includes: there is no node description text in the node for describing the interface element represented by the node.

[0010] In another possible implementation manner, the node meeting the target condition further includes: there is no area overlap between the target interface area corresponding to the node and the interface areas corresponding to other nodes having node description texts, and the interface areas corresponding to the other nodes are the areas of the interface elements represented by the other nodes in the display interface.

[0011] In yet another possible implementation, it further includes:

[0012] Removing the nodes in the updated description node tree that meet the deletion conditions;

[0013] Among them, the nodes that meet the deletion conditions include at least one of the following:

[0014] Leaf nodes without node description text;

[0015] The child nodes of the parent node of the node only include the node, the node has only one child node, and the node does not have node description text.

[0016] On the other hand, the present application also provides yet another information processing method, including:

[0017] In response to an instruction to execute a target manipulation operation, based on the manipulation node information of the target manipulation operation and the display interface description information of the display interface, determining a target node in the display interface that matches the manipulation node information;

[0018] Based on the manipulation information of the target manipulation operation, simulating a manipulation operation on the target interface element corresponding to the target node in the display interface;

[0019] Among them, the display interface description information is interface directory information, and each directory entry in the interface directory information corresponds to a node in the display interface, and the content of the directory entry is the node description text of a node corresponding to the directory entry.

[0020] In a possible implementation manner, the simulating a manipulation operation on the target interface element corresponding to the target node in the display interface based on the manipulation information of the target manipulation operation includes:

[0021] Determining a target interface area in the display interface that is associated with the target interface element represented by the target node;

[0022] Based on the relative manipulation coordinates recorded in the manipulation information of the target manipulation operation, determining the relative manipulation position of the target manipulation operation relative to the target interface area;

[0023] Simulating a manipulation operation at the relative manipulation position in the display interface.

[0024] In yet another possible implementation manner, the responding to an instruction to execute a target manipulation operation includes:

[0025] In response to an instruction to execute an operation sequence, determine a target manipulation operation to be simulated currently in the operation sequence, where the operation sequence includes multiple manipulation operations, and the target manipulation operation is one of the multiple manipulation operations.

[0026] In another possible implementation manner, the determining, based on the manipulation node information of the target manipulation operation and the display interface description information of the display interface, a target node in the display interface that matches the manipulation node information includes:

[0027] If, based on the display interface description information of the display interface, it is determined that there is a unique node in the display interface whose node description text of the corresponding directory entry matches the manipulation node information of the target manipulation operation, determine the unique node as the target node;

[0028] If, based on the display interface description information of the display interface, it is determined that there are multiple candidate nodes in the display interface whose node description text of the corresponding directory entry matches the manipulation node information of the target manipulation operation, based on the operation interface description information of the target manipulation operation, determine a target node with the highest matching degree with the manipulation node information from the multiple candidate nodes.

[0029] In another possible implementation manner, the responding to an instruction to execute an operation sequence includes:

[0030] In response to an operation instruction input by a user, determine an operation sequence that matches the operation instruction from a recorded operation sequence set.

[0031] In another possible implementation manner, it further includes:

[0032] If there is no operation sequence that matches the operation instruction in the operation sequence set, construct an operation sequence corresponding to the operation instruction based on a knowledge graph, where the knowledge graph includes relationships between multiple entity nodes, and one entity node represents an operation interface or a manipulable function element in the operation interface.

[0033] In another aspect, the present application further provides an information processing device, including:

[0034] A tree obtaining unit, configured to obtain a description node tree corresponding to the current display interface in response to an instruction indicating to obtain display interface description information, where the description node tree includes: at least one node for representing interface elements on the display interface and a hierarchical relationship between different nodes, and different nodes correspond to different interface elements;

[0035] A directory generation unit, configured to generate interface directory information for expressing the display interface based on the hierarchical relationship between nodes in the description node tree, with each node in the description node tree as a directory entry and the node description text of the node as the content of the directory entry.

[0036] A directory feedback unit, configured to feedback the interface directory information.

[0037] In another aspect, the present application further provides another information processing device, including:

[0038] A node matching unit, configured to, in response to an instruction to execute a target manipulation operation, determine a target node in the display interface that matches the manipulation node information based on the manipulation node information of the target manipulation operation and the display interface description information of the display interface; wherein, the display interface description information is interface directory information, each directory entry in the interface directory information corresponds to a node in the display interface, and the content of the directory entry is the node description text of a node corresponding to the directory entry.

[0039] An operation simulation unit, configured to simulate a manipulation operation on a target interface element corresponding to the target node in the display interface based on the manipulation information of the target manipulation operation. Description of the Drawings

[0040] In combination with the accompanying drawings and with reference to the following specific embodiments, the above and other features, advantages and aspects of the various embodiments of the present disclosure will become more apparent. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic and the original components and elements are not necessarily drawn to scale.

[0041] Figure 1 It is a schematic flowchart of an information processing method provided by the present application;

[0042] Figure 2 It is another schematic flowchart of an information processing method provided by the present application;

[0043] Figure 3 It is an example diagram of a display interface provided by the present application;

[0044] Figure 4 It is a schematic flowchart of another information processing method provided by the present application;

[0045] Figure 5 It is another schematic flowchart of another information processing method provided by the present application;

[0046] Figure 6 It is a schematic flowchart of a method for generating operation record information of a manipulation operation provided by the present application;

[0047] Figure 7 Another flowchart of another information processing method provided by this application;

[0048] Figure 8 A flowchart of processing operation instructions based on a knowledge graph in this application;

[0049] Figure 9 A schematic diagram of the composition structure of an information processing device provided by this application;

[0050] Figure 10 A schematic diagram of the composition structure of another information processing device provided by this application;

[0051] Figure 11 A schematic diagram of the composition architecture of an electronic device provided by this application. Detailed implementation manners

[0052] The embodiments of this application will be described below with reference to the accompanying drawings in the embodiments of this application. The terms used in the implementation manners part of this application are only used to explain the specific embodiments of this application, rather than intended to limit this application. Those of ordinary skill in the art know that with the development of technology and the emergence of new scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.

[0053] The terms "first", "second", etc. in the specification, claims and above-mentioned drawings of this application are used to distinguish similar objects, and do not have to be used to describe a specific order or sequence. It should be understood that these terms can be interchanged under appropriate circumstances, which is only a way of distinction adopted when describing objects with the same attributes in the embodiments of this application. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, so that a process, method, system, product or device including a series of units does not have to be limited to those units, but may include other units not clearly listed or inherent to these process, method, product or device.

[0054] As Figure 1 , a flowchart of an information processing method provided by this application is shown. The method of this embodiment can be applied to an electronic device, and the electronic device can be a mobile phone, a tablet computer, a laptop computer, etc., without limitation. The method of this embodiment may include:

[0055] S101, in response to an instruction indicating to obtain display interface description information, obtain a description node tree corresponding to the current display interface.

[0056] Among them, the description node tree is a node tree used to describe the composition of the current display interface of the electronic device.

[0057] Among them, the description node tree includes: at least one node for representing interface elements on the display interface and the hierarchical relationship between different nodes. Among them, different nodes in the description node tree correspond to different interface elements in the display interface.

[0058] Among them, the interface elements on the display interface are also called user interface (UI) elements. The interface elements can be various visible elements such as buttons, input boxes, and pictures on the display interface; they can also be interface elements that are invisible on the display interface but support the interaction logic, such as listeners, elements for authentication or permission verification, etc.

[0059] Among them, the hierarchical relationship between different nodes represents the association relationship between different interface elements on the display interface.

[0060] It can be understood that each node in the description node tree has attribute information. The attribute information of the node includes information such as the name of the interface element represented by the node and the position area (the area coordinates corresponding to the position area) of the interface element represented by the node in the display interface.

[0061] In particular, the attribute information of the node can also include the node description text of the node. The node description text is used to describe the characteristics of the interface element represented by the node. Through the node description text of the node, the presentation characteristics of the interface element represented by the node on the display interface can be accurately reflected. For example, the node description text can describe information such as the function, presentation form, and whether it is clickable of the interface element represented by the node. Of course, not every node has a node description text, and the node description text of some nodes can be empty.

[0062] As can be seen from the above, the description node tree is a logical mapping of the current display interface of the electronic device. For example, the operating system can generate a corresponding description node tree in real time based on the dynamic changes of the display interface. Therefore, when the content of the display interface of the electronic device is updated, the description node tree will also be synchronously updated so that the description node tree can accurately represent the composition and content of the current display interface.

[0063] For example, in operating systems such as Android, the description node tree can be an accessibility service node tree (or an accessibility node tree). The accessibility service node tree is used to describe the composition of the display interface, and the node description text in the nodes of the accessibility service node tree can provide additional text descriptions for the blind. This application can use the nodes and the node description text of the nodes in the accessibility service node tree to generate description information for describing the content of the display interface. Of course, in other operating systems or with the continuous development of technology, the description node tree may also have other possibilities or other naming forms.

[0064] It can be understood that, compared with the layout file of the display interface or the HTML file converted from the layout file, the description node tree can more accurately and intuitively represent the interface elements of the display interface and the relationships between the interface elements, and has less redundant information.

[0065] S102, taking each node in the description node tree as a table of contents entry and the node description text of the node as the content of the table of contents entry, based on the hierarchical relationship between the nodes in the description node tree, generate interface table of contents information for expressing the display interface.

[0066] From the implementation of generating the interface table of contents information, it can be known that the interface table of contents information may include: at least one table of contents entry with a hierarchical relationship. Among them, each table of contents entry represents a node corresponding to the display interface. In this embodiment, the node corresponding to the display interface is a node in the description node tree. And the content of each table of contents entry is the node description text of the node corresponding to the table of contents entry. Correspondingly, the hierarchical relationship between different table of contents entries represents the hierarchical relationship between the nodes represented by the table of contents entries.

[0067] It can be understood that since each node in the description node tree represents an interface element in the display interface, each table of contents entry in the interface table of contents information actually corresponds to an interface element of the display interface, and the content of the table of contents entry is the feature description text for describing the interface element of the table of contents entry.

[0068] Among them, in order to distinguish each table of contents entry in the interface table of contents, different table of contents entries in the interface table of contents file correspond to different table of contents entry numbers. Based on the table of contents entry numbers of the table of contents entries, it is also possible to use the table of contents entry numbers as indexes in the subsequent task processing process to trace back to the nodes in the description node tree.

[0069] Since the interface table of contents information uses a table of contents form such as a book table of contents or a file table of contents to describe the content of the display interface, does not contain special syntax such as pictures and HTML, can more intuitively and accurately express the content composition of the display interface, has higher readability, and the book table of contents or file table of contents, etc. are the most basic data for artificial intelligence applications and large language model training. Therefore, using a table of contents form to describe the content of the display interface reduces the difficulty for artificial intelligence applications or large language models, etc. to understand the interface table of contents information.

[0070] S103, feedback the interface table of contents information.

[0071] For example, based on an instruction indicating to obtain the display interface description information, output the interface table of contents information, or feedback the interface table of contents information to an application or a large language model, etc. that needs to obtain the display interface content, without specific limitations.

[0072] As can be seen from the above, when the present application needs to generate the description information of the display interface, it will obtain the description node tree of the display interface. On this basis, taking each node in the description node tree as a table of contents entry, and taking the node description text of the node corresponding to the table of contents entry as the content of this table of contents entry, combined with the hierarchical relationship between the nodes in the description node tree, the interface table of contents information for describing the display interface is generated. Since the interface table of contents information describes the display interface in the form of a table of contents, it can more intuitively and clearly express the content of the display interface, and has stronger readability; moreover, since the description node tree can describe the composition of the display interface and has relatively less redundant information, therefore, the table of contents interface information generated based on the description node tree can more comprehensively express the content of the display interface, which is beneficial for artificial intelligence services or artificial intelligence models, etc. to accurately determine the content of the display interface based on the interface table of contents information, and naturally can improve the accuracy of task processing based on the content of the display interface.

[0073] It can be understood that not every node in the description node tree has a node description text. For example, if the interface element represented by the node is a picture displayed in the display interface, then there may be no corresponding node description text in the node. However, the visible content of the display interface is the key information for task processing based on the content of the display interface. Therefore, in order to accurately express the content of the display interface, before generating the interface table of contents information, the present application also needs to supplement the node description text of some nodes in the description node tree, so that the node description texts of each node in the description node tree can fully express the content presented by the display interface.

[0074] Specifically, if the node in the description node tree meets the target condition, determine the target interface area in the display interface associated with the interface element represented by the node, and generate a node description text for this node based on the display content in the target interface area to update the description node tree.

[0075] Among them, the node meeting the target condition at least includes: there is no node description text in the node for describing the interface element represented by the node.

[0076] Among them, the target interface area associated with the interface element represented by the node is the area of the interface element represented by the node in the display interface. For example, the display area of the interface element in the display interface or the interface area that can sense input operations. For example, if the interface element is a visible interface element, the target interface area associated with the interface element can be the area covered by the interface element in the display interface; if the interface element is an invisible interface element, the target interface area associated with the interface element can be the area in the display interface that belongs to the interface element that can respond to operations or can implement display logic control, etc.

[0077] Among them, the display content within the target interface area corresponding to the interface element based on the node representation is used to generate a node description text for the node, which is a description text for describing the display content within the target interface area. Specifically, the specific implementation of generating the node description text for the node can be unrestricted.

[0078] For example, in a possible implementation manner, the present application can perform semantic recognition on the display content within the target interface area, determine the target semantic information expressed by the display content within the target interface area, and use the target semantic information as the node description text corresponding to the node. For example, obtain a regional screenshot of the target interface area in the display interface, and perform semantic recognition on the regional screenshot to determine the target semantic information expressed by the display content within the target interface area.

[0079] It can be understood that generating a node description text for the node includes adding the node description text to the node in the node description tree. For example, adding the node description text generated for the node to the attribute information of the node.

[0080] It can be understood that the present application can traverse each node in the description node tree, sequentially determine whether each node meets the target condition, and generate a node description text for the node when it is determined that the currently traversed node meets the target condition, until all nodes in the description node tree have been traversed to obtain an updated description node tree.

[0081] It can be understood that in the case of updating the description node tree, the present application will generate interface directory information based on the updated description node tree, as described above, and will not be elaborated here.

[0082] It can be understood that the number of nodes in the description node tree is relatively large, and the interface areas of multiple interface elements with an associated relationship in the display interface may overlap, and the display content of the overlapping interface areas is the same. Therefore, for multiple interface elements with overlapping interface areas, only the node description text of the node corresponding to one of the multiple interface elements needs to be used to describe the content of the overlapping area. It can be seen from this that if node description texts are generated for each node without a node description text in the description node tree, not only is the data processing volume large, but there are also some meaningless data processes, resulting in redundancy between the node description texts of different nodes.

[0083] Based on this, before generating a text description node for a node in the present application, it is also necessary to determine whether there is an overlap between the target interface area corresponding to this node and the interface areas corresponding to other nodes with node description texts. Only when there is no overlap between the target interface area corresponding to this node and the interface areas corresponding to other nodes with node description texts, is it necessary to generate a node description text for this node. Correspondingly, a node meeting the target condition may also include: there is no overlap between the target interface area corresponding to the node and the interface areas corresponding to other nodes with node description texts. Among them, the interface area corresponding to other nodes is the area of the interface element represented by other nodes in the display interface. The area of the interface element in the display interface is as described above and will not be elaborated here.

[0084] On this basis, considering that there may be some meaningless or less significant interface elements in the display interface, which may result in nodes in the description node tree that are useless for describing the display interface. Moreover, the more nodes there are in the description node tree, the more directory entries there will be in the interface directory file. Based on this, before generating interface directory information based on the description node tree, the present application can also remove the nodes in the updated description node tree that meet the deletion conditions to optimize the description node tree.

[0085] Among them, the nodes that meet the deletion conditions include at least one of the following:

[0086] Leaf nodes without node description texts;

[0087] The child nodes of the parent node of the node only include this node. This node has only one child node and this node does not have a node description text.

[0088] Correspondingly, after obtaining the optimized description node tree, interface directory information can be generated based on the optimized description node tree.

[0089] To facilitate the understanding of the above process of updating and optimizing the description node tree, it will be introduced in combination with a possible implementation manner. As Figure 2 , a schematic flowchart of another information processing method provided by the present application is shown. The method of this embodiment may include:

[0090] S201, in response to an instruction indicating to obtain display interface description information, obtain the description node tree corresponding to the current display interface.

[0091] Among them, the description node tree includes: at least one node for representing interface elements on the display interface and the hierarchical relationship between different nodes, and different nodes correspond to different interface elements.

[0092] S202. Traverse each node in the description node tree. If the currently traversed node has no node description text and there is no area overlap between the target interface area corresponding to the interface element represented by this node and the interface areas corresponding to other nodes with node description texts, determine the target interface area associated with the interface element represented by this node in the display interface, and generate a node description text for this node based on the display content within this target interface area to update this description node tree.

[0093] As described above, if there is an overlap between the interface areas corresponding to different interface elements, then there is also duplication in the description information corresponding to the interface areas corresponding to these interface elements. Based on this, although this node itself has no node description text, but there are other nodes corresponding to other interface elements with which the interface area corresponding to the interface element represented by this node overlaps and have node description texts. Then, in order to reduce the redundancy between the node description texts of different nodes, also reduce the data processing volume, and not generate a node description text for this node, the next node in the description node tree will be continued to be traversed.

[0094] Of course, if the currently traversed node has a node description text, it is also possible not to process this node and continue to traverse the next node in this description node tree.

[0095] S203. Remove each leaf node in the updated description node tree that has no node description text to obtain the initially optimized description node tree.

[0096] After the processing of the above step S202, if a certain leaf node in the updated description node tree still has no node description text, it means that the interface element corresponding to this node is meaningless for expressing the content of this display interface. Therefore, in order to reduce the number of nodes in the description node tree, the leaf nodes without node description text can be removed.

[0097] S204. Traverse each node in the initially optimized description node tree. For the currently traversed node, if the child nodes of the parent node of this node only include this node, this node has only one child node, and this node has no node description text, then delete this node from the initially optimized description node tree to obtain the secondarily optimized description node tree.

[0098] Through this step, meaningless nodes in the description node tree can be reduced, thereby reducing the depth and the number of nodes of the description node tree.

[0099] For example, if a node A has only one child node B, and this child node B also has only one child node C, and this child node B does not have node description text, then the existence of child node B has little meaning. Therefore, child node B can be deleted from the description node tree, and child node C can be used as the child node directly connected to node A.

[0100] It can be understood that by traversing each node in the description node tree after the first optimization in sequence, when the traversal of the description node tree after the first optimization is completed, the nodes that need to be deleted in the description node tree after the first optimization have been all deleted, thus obtaining the description node tree after the second optimization.

[0101] S205: Using each node in the description node tree after the second optimization as a directory entry and using the node description text of this node as the content of the directory entry, based on the hierarchical relationship between the nodes in the description node tree after the second optimization, generate interface directory information for expressing the display interface.

[0102] S206: Feedback the interface directory information.

[0103] The above steps S205 and S206 can refer to the relevant introduction in the previous embodiments and will not be elaborated here.

[0104] To facilitate the understanding of the composition of the interface directory information, the following will be described with an example:

[0105] For example Figure 3 shows an example diagram of the display interface.

[0106] In Figure 3 the shown display interface includes an input field, function options or icons, etc.

[0107] In this application, after obtaining the description node tree of the display interface, the description node tree can be updated and optimized in the manner of the Figure 2 embodiment, and then the interface directory information can be generated based on the description node tree. For example, the interface directory information generated for the Figure 3 shown display interface can be as follows:

[0108] (1.1.1) Settings;

[0109] (1.1.2.1) Here is an icon, which may represent "Search, magnifying glass, Find";

[0110] (1.1.2.2) Search settings;

[0111] (1.2.1.1.1) Here is an icon, which may represent "Settings, options, configuration";

[0112] (1.2.1.1.2.1)Complete device setup;

[0113] (1.2.1.1.2.2)Just a few more steps;

[0114] (1.2.1.1.3)Turn off;

[0115] (1.2.2.1.1)Here is an icon that may be used to represent "WiFi, signal, connection";

[0116] (1.2.2.1.2.1)Network & Internet;

[0117] (1.2.2.1.2.2)Mobile, WiFi, hotspot;

[0118] (1.2.2.2.1)Here is an icon that may represent "smartwatch, phone, technology products";

[0119] (1.2.2.2.2.1)Connect devices;

[0120] (1.2.2.2.2.2)Bluetooth, pair;

[0121] (1.2.2.3.1)Here is an icon that may represent "photo album, take pictures, camera";

[0122] (1.2.2.3.2.1)Personalize;

[0123] (1.2.2.3.2.2)Wallpaper, color, icon, font, display size;

[0124] (1.2.2.4.1)Here is an icon that may represent "music player, media control, audio function";

[0125] (1.2.2.4.2.1)External display;

[0126] (1.2.2.4.2.2)Clock style, panel, applications;

[0127] (1.2.2.5.1)Here is an icon that may represent "brightness, adjustment, sunlight";

[0128] (1.2.2.5.2.1)Display;

[0129] (1.2.2.5.2.2)Dark theme, font size, brightness;

[0130] (1.2.2.6.1)Here is an icon that may represent "phone, lock, security";

[0131] (1.2.2.6.2.1) Main Screen & Lock Screen

[0132] (1.2.2.6.2.2) Customize the content displayed on the main screen and lock screen.

[0133] Among them, the numbers in parentheses corresponding to the above directory entries are the serial numbers of the directory entries, and the relationships between different directory entries can be reflected by the serial numbers of the directory entries.

[0134] For example, (1.1.2.1) and (1.1.2.2) are the serial numbers of two directory entries, and it can be known from the serial numbers of these two directory entries that the two nodes corresponding to these two directory entries belong to the same parent node.

[0135] Another example is that the node corresponding to the directory entry (1.2.2.1.1) and the nodes corresponding to the two directory entries (1.2.1.1.2.1) and (1.2.1.1.2.2) belong to nodes at the same level of the parent node. However, the above interface directory information does not include the directory entry (1.2.1.1.2) because the node corresponding to the directory entry (1.2.1.1.2) does not include node description text. Therefore, the corresponding node was deleted when optimizing the description node tree, and thus the interface entry information may not include the directory entry (1.2.1.1.2) corresponding to this node.

[0136] Combined with Figure 3 the shown display interface, it can be seen that each of the above directory entries describes the various interface elements that can be felt by the user in the display interface and the text information presented by the interface elements, etc., so as to be able to fully describe and intuitively display the content of the interface, and the readability is relatively high.

[0137] Next, a scenario for performing related task processing on the interface directory information combined with the display interface will be introduced.

[0138] As Figure 4 shows a flowchart of another information processing method provided by this application. This embodiment may include:

[0139] S401, in response to an instruction to execute a target manipulation operation, based on the manipulation node information of the target manipulation operation and the display interface description information of the display interface, determine the target node in the display interface that matches the manipulation node information.

[0140] Among them, the target manipulation operation is a manipulation operation input in history. For example, the target manipulation operation is a manipulation operation input by the user and recorded before the current moment. Based on this, the target manipulation operation can belong to the recorded historical manipulation operations. For example, the target manipulation operation is a recorded and replayable (or simulatable) manipulation operation.

[0141] The manipulation node information of the target manipulation operation is used to characterize the node features of the historical node corresponding to the target manipulation operation. Among them, the historical node corresponding to the target manipulation operation is the node targeted by the target manipulation operation in the historical display interface where the target manipulation operation is located. The historical display interface where the target manipulation operation is located is also subsequently referred to as the historical display interface corresponding to the target manipulation operation, which refers to the display interface that generates the target manipulation operation. For the convenience of distinguishing from the current display interface and the nodes corresponding to the display interface, the display interface corresponding to the target historical operation is called the historical operation interface.

[0142] In this application, for any one of the historical display interface and the current display interface, the nodes of the display interface are used to represent an interface element of the display interface. For example, the node can be a node in the description node tree of the display interface, or a node determined based on the layout file of the display interface for representing the interface element, without specific limitations.

[0143] Among them, since the historical node corresponds to an interface element in the historical display interface, the node features of the historical node are the features possessed by the interface element represented by the historical node. For example, the node features of the historical node can include the node description text of the historical node, and the node description text of the historical node is used to describe the features of the interface element represented by the historical node. Generally, the node description text of the historical node at least includes the user-perceivable features possessed by the interface element represented by the historical node, such as the presentation form, function, and operable state of the interface element.

[0144] Among them, the display interface description information of the display interface is the interface directory information of the display interface. Among them, the interface directory information includes multiple directory entries, and each directory entry corresponds to a node of the display interface, and the content of the directory entry is the node description text of the node corresponding to the directory entry.

[0145] In a possible case, the interface directory information can be generated based on the nodes and the hierarchical relationship of the nodes in the description node tree of the display interface, as specifically introduced in the previous embodiments, and will not be elaborated here.

[0146] In another possible case, the interface directory information can also be generated based on the hierarchical relationship between the interface elements in the display interface characterized in the layout file of the display interface or the HTML file converted from the layout file. In this case, the node corresponding to the directory entry also represents an interface element, and the content of the directory entry is the node description text corresponding to the node representing the interface element, and the node description text is used to describe the features possessed by the interface element represented by the node.

[0147] It can be understood that the interface directory information of the display interface describes the content of the display interface in the form of a directory, so as to be able to more intuitively and clearly express the content of the display interface, with stronger readability, and is also conducive to achieving more efficient information matching and analysis processing.

[0148] Among them, the target node is the node in the display interface that matches the manipulation node information of the target manipulation operation. Since the display interface description information of the display interface is the interface directory information of the display interface, and each directory entry in the interface directory information corresponds to a node of the display interface, and the content of the directory entry is the node description text of the corresponding node, therefore, the target node is the node whose corresponding node description text matches the manipulation node information of the target manipulation operation.

[0149] For example, taking the manipulation node information of the target manipulation operation including the node description text of the historical node corresponding to the target manipulation operation as an example, the target node is the node in the display interface whose node description text matches the node description text corresponding to the target manipulation operation.

[0150] S402, based on the manipulation information of the target manipulation operation, simulate a manipulation operation on the target interface element corresponding to the target node in the display interface.

[0151] Among them, the manipulation information of the target manipulation operation is used to characterize the manipulation behavior information of the target manipulation operation. For example, the manipulation information at least includes manipulation position information, and may also include manipulation pressure and information about the manipulating user, etc.

[0152] It can be understood that since the target node is actually the node in the current display interface that corresponds to the historical node operated by the target manipulation operation, therefore, based on the operation information of the target manipulation operation, simulating the manipulation operation on the target interface element represented by the target node can simulate the target manipulation operation in the current display interface, achieving the purpose of replaying the target manipulation operation.

[0153] As can be seen from the above content of this embodiment, in response to an instruction to execute a target manipulation operation, the present application determines a target node in the display interface that matches the manipulation node information of the target manipulation operation based on the display interface description information of the display interface. Since the display interface description information of the display interface is interface catalog information, and each catalog entry in the interface catalog information corresponds to a node of the display interface, and the content of the catalog entry is the node description text of a node corresponding to the catalog entry, based on the display catalog information of the display interface, the content of the display interface can be described intuitively and clearly, so that based on the display interface description information of the display interface, a target node that matches the manipulation node information of the target manipulation operation can be matched more accurately and efficiently. Furthermore, based on the manipulation information of the target manipulation operation, a simulated manipulation operation can be accurately performed on the target interface element represented by the target node, realizing a more accurate simulated manipulation operation.

[0154] It can be understood that since there may be correlations or similarities among some interface elements in the display interface, the node description texts of the nodes corresponding to some interface elements are relatively similar. On this basis, based on the manipulation node information of the target manipulation operation, more than one target node may be matched, but not all of these multiple target nodes are the nodes actually operated by the target manipulation operation.

[0155] Based on this, in the present application, the information of the target manipulation operation further includes the operation interface description information of the target manipulation operation, and the operation interface description information is the display interface description information of the historical display interface corresponding to the target manipulation operation. Based on this, the operation interface description information can be the interface catalog information of the historical display interface corresponding to the target manipulation operation. The interface catalog information of the historical display interface includes multiple historical catalog entries, each historical catalog entry corresponds to a node of the historical display interface, and the content of the historical catalog entry is the node description text of the node corresponding to the historical catalog entry. For the convenience of distinction, the nodes of the historical display interface can be called historical nodes, and each historical node corresponds to an interface element in the historical display interface.

[0156] In this embodiment, for the convenience of distinction, the catalog entries in the interface catalog information of the historical display interface are called historical catalog entries.

[0157] On this basis, if based on the display interface description information of the display interface, it is determined that there is a unique node in the display interface whose node description text of the corresponding catalog entry matches the manipulation node information of the target manipulation operation, the unique node is determined as the target node.

[0158] Correspondingly, if, based on the display interface description information of the display interface, it is determined that there are multiple candidate nodes in the display interface whose node description texts of the corresponding directory entries match the manipulation node information of the target manipulation operation, the target node with the highest matching degree with the manipulation node information can also be determined from the multiple candidate nodes based on the operation interface description information of the target manipulation operation.

[0159] Among them, determining the target node from the candidate nodes based on the operation interface description information can be: determining the target historical directory entry that matches the manipulation node information of the target manipulation operation based on the node description texts of each historical directory entry in the operation interface description information. Combining the node description texts of at least one associated directory entry associated with the target historical directory entry in the operation interface description information, and the node description texts of at least one associated node associated with the candidate node, the target node that matches the target historical directory entry is determined from the candidate nodes.

[0160] For example, the target historical directory entry that matches the manipulation node information of the target manipulation operation is the historical directory entry whose corresponding node description text matches the node description text in the manipulation node information.

[0161] Among them, the associated directory entries associated with the target historical directory entry can include at least one of the following: the upper-level directory entry corresponding to the target historical directory entry (i.e., the parent directory entry of the target historical directory entry), the lower-level directory entry of the target historical directory entry, the sibling directory entry that belongs to the upper-level directory entry of the target historical directory entry and is at the same level as the target historical directory entry, and the sibling directory entry that has the same parent directory entry as the upper-level directory entry of the target historical entry and is at the same level as the upper-level directory entry of the target historical entry.

[0162] Similarly, the associated nodes associated with the candidate node can include at least one of the following: the parent node of the candidate node, the child node of the candidate node, the sibling node of the candidate node, and the sibling node of the parent node of the candidate node.

[0163] Among them, when determining the target node, the candidate node whose node description text of the associated associated node has the highest matching degree with the node description text in the associated directory entry can be selected as the target node.

[0164] For example, the target associated directory entry to be currently matched can be determined from the associated directory entries of the target historical directory entry according to the priority order of the associated directory entries and associated nodes, and for each candidate node, the target associated node to be matched in the associated nodes of the candidate node can be determined. If there is a target associated node whose node description text has the highest and unique matching degree with the node description text of the target associated directory entry, the candidate node corresponding to the unique target associated node is determined as the target node. If there are multiple target associated nodes whose node description texts have the highest matching degree with the node description text of the target associated directory entry, then according to the priority of the associated directory entries and associated nodes, the next target associated directory entry to be matched and the next target associated nodes of each candidate node are determined, and a comparison is made until the target node is determined.

[0165] Among them, the priority order of the associated directory entries from high to low can be: the next-level directory entry corresponding to the target historical directory entry, the upper-level directory entry corresponding to the target historical directory entry, the sibling directory entry of the target historical directory entry, and the sibling directory entry of the upper-level directory entry corresponding to the target historical directory entry. Similarly, the priority order of the associated nodes of the candidate node from high to low can be: the child node of the candidate node, the parent node of the candidate node, the sibling node of the candidate node, and the sibling node of the parent node of the candidate node.

[0166] Of course, the priority order can also be set according to actual needs, and there is no restriction on this.

[0167] In this application, based on the manipulation information of the target manipulation operation, there can be various possibilities for simulating the manipulation operation on the target interface element corresponding to the target node. Here, an implementation method is taken as an example for illustration.

[0168] Such as Figure 5 , which shows a flowchart of another information processing method provided by this application. The method of this embodiment may include:

[0169] S501, in response to an instruction to execute a target manipulation operation, based on the manipulation node information of the target manipulation operation and the display interface description information of the display interface, determine the target node in the display interface that matches the manipulation node information.

[0170] Among them, the display interface description information is interface directory information. Each directory entry in this interface directory information corresponds to a node of the display interface, and the content of the directory entry is the node description text of a node corresponding to the directory entry.

[0171] This step can refer to the relevant introduction in the previous embodiment and will not be elaborated here.

[0172] In a possible implementation, the manipulation node information of the target manipulation operation may include: the node description text of the historical node operated in the historical display interface for the target manipulation operation. On this basis, based on the interface directory information of the display interface, a target directory entry whose corresponding node description text matches the node description text in the operation node information of the target manipulation operation can be determined from the interface directory information of the display interface, and the node corresponding to the target directory entry in the display interface is determined as the target node.

[0173] It can be understood that there may be code refactoring in the main interface of the electronic device and the application interfaces of the applications or specific services implemented in the electronic device, resulting in changes in the hierarchical relationship of the interface elements in the display interface, or parameters such as the display size or resolution of the display interface. Among them, the code refactoring related to the display interface is invisible to the user. However, in order to ensure the user experience, the change in the visual effect of the display interface will not be too large. Based on this, if relevant parameters such as the name of the interface element operated by the target manipulation operation are recorded when the target manipulation operation occurs, then once a situation such as code refactoring occurs, it may not be possible to accurately determine the interface element originally operated by the target manipulation operation from the updated display interface, thus affecting the simulation operation of the target manipulation operation.

[0174] In this application, considering that the probability of the visible part in the display interface is relatively small, based on this, this application will record the node description text of the historical node corresponding to the target manipulation operation when the target manipulation operation occurs. Since the node description text can describe the visual features such as the presentation form of the interface element represented by the node and the visible functional role, etc., even if the code of the subsequent display interface changes, this application can still determine from the updated display interface the visual features of the interface element represented by the node that the target manipulation operation actually hopes to operate based on the node description text recorded in the operation node information of the target manipulation operation, so as to accurately locate the target node corresponding to the interface element to be manipulated in the display interface.

[0175] S502. Determine the target interface area in the display interface that is associated with the target interface element represented by the target node.

[0176] For the sake of easy distinction, the interface area associated with the target interface element represented by the target node is called the target interface area. For the introduction of the interface area associated with the target interface element, reference can be made to the relevant introduction above, and details will not be elaborated here.

[0177] S503. Based on the relative manipulation coordinates recorded in the manipulation information of the target manipulation operation, determine the relative manipulation position of the target manipulation operation relative to the target interface area.

[0178] Among them, the relative manipulation coordinates in the manipulation information are the relative coordinate positions of the target manipulation operation on the historical display interface with respect to the interface area associated with the historical node corresponding to the target manipulation operation. Among them, the historical display interface is the display interface where the target manipulation operation is located, and the interface area associated with the historical node corresponding to the target manipulation operation is, in the historical display interface, the interface area corresponding to the interface element represented by the historical node.

[0179] Since the target node is the node in the current display interface that matches the manipulation node information of the target manipulation operation, therefore, the relative manipulation coordinates of the manipulation information are actually the relative coordinate positions with respect to the target interface area associated with the target interface element represented by the target node. Based on this, in this application, based on the relative manipulation coordinates, the relative manipulation position of the target manipulation operation on the display interface with respect to the target interface area can be determined.

[0180] It can be understood that since the display interfaces of applications or services may be updated, etc., therefore, the interface areas corresponding to the interface elements in the display interface at different times will also change. Based on this, if only the operation position points of the touch operation in the display interface are recorded, then after the display interface is updated, the functions or actions that can be triggered based on the operation position points of the touch operation will be different from the functions or actions triggered by the touch operation history, or may cause the inability to trigger the execution of actions, etc., and naturally it will be impossible to simulate (or replay) the touch operation on the updated display interface.

[0181] In this application, however, the manipulation information of the target manipulation operation records the relative manipulation coordinates of the target manipulation operation with respect to the interface area associated with the historical node corresponding to the target manipulation operation. Therefore, even if the display interface changes, causing the position range of the interface area associated with the historical node corresponding to the target manipulation operation to change, after determining the node in the display interface corresponding to the target manipulation operation, based on the relative manipulation coordinates, the relative manipulation position of the target manipulation operation with respect to the target interface area can still be accurately determined, thereby accurately determining the manipulation position point where the target manipulation operation actually needs to be performed in the display interface, and naturally being able to more accurately simulate the corresponding manipulation operation.

[0182] S504, simulate the manipulation operation at the relative manipulation position in the display interface.

[0183] It can be understood that, based on the different operation types of the target manipulation operation, the number of manipulation node information and manipulation information corresponding to the target manipulation operation will also be different. However, the processing of each group of manipulation node information and manipulation information is the same. Correspondingly, a relative manipulation position can be obtained for each group of manipulation node information and manipulation information. On this basis, simulating a touch operation at each relative manipulation position can achieve the playback of the target manipulation operation.

[0184] For example, if the target manipulation operation is a click operation, then there is one position point for the click operation. Correspondingly, there is a node description text of a historical node and a relative manipulation coordinate corresponding to the click operation, so as to determine a relative manipulation position.

[0185] Another example is that if the target manipulation operation is a slide operation, then during the slide operation, the electronic device needs to continuously collect the operation position points of the slide operation. On this basis, for each manipulation position point during the slide of the slide operation, a node description text of a historical node and a relative manipulation coordinate are recorded. Correspondingly, when it is necessary to simulate the target manipulation operation, based on the solution of the present application, a continuous plurality of relative manipulation positions will be determined, and then the manipulation operation can be simulated sequentially at these plurality of relative manipulation positions, so as to simulate the slide operation.

[0186] It can be understood that before obtaining the instruction to execute the target manipulation operation, the present application will record the operation record information of the target manipulation operation when detecting the target manipulation operation. The operation record information includes the manipulation node information, manipulation information, and operation interface description information corresponding to the target manipulation operation. The following takes an implementation manner as an example for illustration. For example Figure 6 shows a schematic flowchart of an implementation process for generating the operation record information of the manipulation operation in the present application. This process may include:

[0187] S601, detecting a manipulation operation on the historical display interface, generating interface directory information for describing the historical display interface, and determining the interface directory information of the historical display interface as the operation interface description information of the manipulation operation.

[0188] For the convenience of distinction, the display interface before obtaining the instruction to execute the target manipulation operation is called the historical display interface.

[0189] Among them, the specific implementation of generating the interface directory information of the historical display interface can adopt any one of the implementation manners for generating the display interface description information of the display interface before, which will not be elaborated here.

[0190] In a possible implementation manner, the present application can adopt the previous Figures 1 to 2Generate the interface directory information of the historical display interface in the way of generating the interface directory information in the embodiments. For example, obtain the description node tree for describing the historical display interface. Generate the interface directory information for expressing the historical display interface based on the hierarchical relationship between the nodes in the description node tree of the historical display interface.

[0191] Of course, before generating the interface directory information, the description node tree of the historical display interface can also be updated and optimized according to the implementation method of optimizing the description tree before, as specifically introduced before, and will not be elaborated here.

[0192] S602, based on the manipulation position of the manipulation operation in the historical display interface, determine the node description text corresponding to the historical node related to the manipulation operation in the historical display interface, and record the node description text of the historical node as the manipulation node information corresponding to the manipulation operation.

[0193] In this application, in order to accurately determine the display object actually operated by the target manipulation operation even when the code and layout of the historical display interface are updated and changed subsequently, this application will record the node description text of the historical node related to the manipulation operation, so as to locate the display object actually operated by the manipulation operation through the node description text of the historical node subsequently. The display object is a user-perceivable object displayed on the display interface, such as text, image, button, or the visible area of a certain interface element on the display interface.

[0194] Among them, the historical node corresponding to the manipulation operation is the node corresponding to the interface element whose corresponding interface area covers the manipulation position of the manipulation operation. The node description text of the historical node is at least used to describe the visual characteristics of the interface element corresponding to the historical node, and can also be used to describe other attribute information of the interface element, and there is no limitation on this.

[0195] In particular, in the case of generating the interface directory information of the historical display interface based on the description node tree of the historical display interface, the historical node is the node in the description node tree of the historical display interface that is related to the interface element corresponding to the manipulation position of the manipulation operation and has a node description text.

[0196] Among them, the historical nodes related to the interface element corresponding to the manipulation operation can belong to the candidate historical nodes for describing the node elements corresponding to the manipulation operation in the description node tree of the historical display interface, as well as the parent nodes, child nodes, and sibling nodes related to the candidate historical nodes.

[0197] If the candidate historical node corresponding to the interface element that matches the manipulation position of the manipulation operation in the description node tree of the historical display interface has a node description text, the candidate historical node will be determined as the historical node of the manipulation operation (or the historical node corresponding to the manipulation operation). If the candidate historical node does not have a node description text, the parent node, child nodes, and sibling nodes of the candidate historical node will be determined from the description node tree of the historical display interface, and in the priority order of child nodes, parent node, and sibling nodes, the node with the highest priority and having a node description text will be determined as the historical node operated by the manipulation operation.

[0198] S603. Determine the historical interface area of the interface element represented by the historical node in the historical display interface, determine the relative manipulation coordinates of the manipulation operation relative to the historical interface area based on the manipulation position of the manipulation operation, and record the relative manipulation coordinates as the manipulation information of the manipulation operation to obtain the operation record information of the manipulation operation.

[0199] It can be understood that if the manipulation operation is a continuous operation such as a double-click or a swipe operation, then at a set sampling interval, multiple manipulation positions of the manipulation operation in the historical display interface can be obtained. For each manipulation position, steps S602 and S603 need to be executed, so that the interface directory information of the historical display interface corresponding to the manipulation operation, the node description texts of at least one historical node, and at least one set of relative manipulation coordinates can be obtained.

[0200] It can be understood that the operation record information generated by the solution of the present application records information such as the interface directory entries of the historical display interface and the node description texts of the nodes operated by the manipulation operation in text form, which is more convenient for artificial intelligence services or large language models to understand. Therefore, it is beneficial for artificial intelligence services and large language models to derive other manipulation operation information similar to the recorded manipulation operations through learning the operation record information of the manipulation operation.

[0201] To facilitate understanding of the specific implementation of simulating the target manipulation operation, the following takes the interface directory information generated based on the description node tree of the display interface as an example of the display interface description information of the display interface, and is described in combination with an implementation manner. As Figure 7 , which is another schematic flowchart of another information processing method provided by the present application. The method of this embodiment may include:

[0202] S701. In response to an instruction to execute a target manipulation operation, obtain the manipulation node information, manipulation information, and operation interface description information of the target manipulation operation, and determine the display interface description information of the display interface.

[0203] Among them, the manipulation node information includes the node description text of the historical nodes of the target operation, and the manipulation information includes the relative manipulation coordinates of the target manipulation operation, where the relative manipulation coordinates are the relative coordinates of the target manipulation operation with respect to the interface area corresponding to the interface element represented by the historical node.

[0204] The operation interface description information is the interface directory information of the historical operation interface corresponding to the target manipulation operation.

[0205] Among them, the display interface description information of the display interface is the interface directory information corresponding to the display interface. The interface directory information of the display interface includes: multiple directory entries, each directory entry corresponds to a node of the display interface, and the content of the directory entry is the node description text of a node corresponding to the directory entry. For the specific implementation of determining the interface directory information of the display interface, reference can be made to the relevant descriptions in the foregoing Figure 1 and Figure 2 and other embodiments, which will not be elaborated here.

[0206] S702, if a unique directory entry whose content matches the node description text in the manipulation node information of the target manipulation operation is determined from the interface directory information of the display interface, determine the target node in the description node tree of the display interface corresponding to the unique directory entry.

[0207] S703, if multiple candidate directory entries whose content matches the node description text in the manipulation node information of the target manipulation operation are determined from the interface directory information of the display interface, determine the target historical directory entry that matches the node description text corresponding to the target manipulation operation from the interface directory information of the historical display interface corresponding to the target manipulation operation.

[0208] S704, determine at least one historical associated directory entry associated with the target historical directory entry in the interface directory information of the historical display interface, and determine at least one associated directory entry associated with each candidate directory entry from the interface directory entries of the display interface.

[0209] S705, based on at least one historical associated directory entry and the associated directory entries of each candidate directory, determine the target directory entry with the highest matching degree with the target historical directory entry from the multiple candidate directory entries, and determine the node corresponding to the target directory entry in the description node tree of the display interface as the target node.

[0210] Among them, for the specific implementation of determining the target directory entry from the multiple candidate directory entries, reference can be made to the relevant introduction in the foregoing embodiments, which will not be elaborated here.

[0211] S706, determine the target interface area associated with the target interface element corresponding to the target directory entry in the display interface.

[0212] Among them, based on the target node corresponding to the target directory entry, the target interface element represented by the target node can be determined, so that the target interface area corresponding to the target interface element in the display interface can be determined.

[0213] S707, based on the relative manipulation coordinates recorded in the manipulation information of the target manipulation operation, determine the relative manipulation position of the target manipulation operation relative to the target interface area.

[0214] S708, simulate a manipulation operation at the relative manipulation position relative to the target interface area on the display interface.

[0215] It can be understood that in order for the artificial intelligence service to operate an application on the electronic device or execute at least one operation related to implementing a certain function according to the user's instruction, etc., generally, the user needs to input multiple manipulation operations to demonstrate the operation steps for implementing the relevant operation requirements. Based on this, the target manipulation operation in this application can belong to the manipulation operations in the historically input operation sequence.

[0216] For example, in a possible implementation manner, this application can respond to an instruction to execute an operation sequence, and determine the target manipulation operation to be currently simulated in the operation sequence. On this basis, based on the manipulation node information of the target manipulation operation and the display interface description information of the display interface, determine the target node in the display interface that matches the manipulation node information. Among them, the operation sequence includes multiple manipulation operations, and the target manipulation operation is one of the multiple manipulation operations.

[0217] Furthermore, this application can also store the operation sequences that can be simulated into an operation sequence set. On this basis, in response to an operation instruction input by the user, an operation sequence that matches the operation instruction can be determined from the recorded operation sequence set. Then, determine the target manipulation operation to be currently simulated from the operation sequence.

[0218] Among them, the operation instruction input by the user is used to indicate the operation behavior to be performed. For example, the operation instruction can be used to indicate sending a message to a specific user using an instant messaging application, opening a food ordering application to order food, sending an email, or scheduling a meeting, etc. The input methods of the user include but are not limited to: touch input on the display screen, voice input, gesture input, etc.

[0219] For example, this application can perform intent recognition on the operation instruction input by the user, determine the operation intent of the operation instruction, and detect whether there is an operation sequence in the operation sequence that matches the operation intent.

[0220] It can be understood that the operation sequences in the operation sequence set are those that the user has input before. If the operation behavior required by the operation instruction indicated by the user does not belong to the operation behaviors that can be realized by the current operation sequences, naturally, no operation sequence can be matched from the operation sequence set. On this basis, in order to respond to the operation instruction input by the user and complete the corresponding operation, the present application can also pre-construct a knowledge graph.

[0221] Among them, the knowledge graph includes the relationships between multiple entity nodes. An entity node represents an operation interface or a manipulable functional element in the operation interface. Among them, the functional element can be a button, an icon, a function option, etc.

[0222] Among them, for any entity node in the knowledge graph used to represent an operation interface, starting from this entity node in the knowledge graph, each entity node link formed by sequentially connecting this entity node with other entity nodes is an operation path link. Each operation path link is used to indicate the operation path required to implement a service function, and one operation path corresponds to one operation sequence. Different operation path links correspond to different service functions.

[0223] In practical applications, since different applications can implement different types of services, in the present application, different knowledge graphs can be configured for different applications. Based on the knowledge graph corresponding to the application, the operation path link required to execute the service function through this application can be determined.

[0224] For example, taking an instant messaging application as an example, the instant messaging application can implement different functions, and the operation steps required to implement the same function on different interfaces of the instant messaging application will also be different. Based on this, the knowledge graph can include entity nodes corresponding to different operation interfaces in the instant messaging application. Moreover, the functional elements such as the icons or buttons that need to be clicked to implement a certain function under each operation interface are also different. Therefore, the knowledge graph can also include entity nodes of different functional elements. Moreover, in order to be able to implement different functions under each operation interface, the relationship between the entity node corresponding to the operation node and the entity nodes corresponding to other operation interfaces or functional elements can also be constructed in the knowledge graph.

[0225] For example, taking the case of sending a message using instant messaging as an example, the specific operations for implementing this service function on different interfaces of the instant messaging application are also different. For sending a message based on the main interface of the instant messaging application, one needs to first enter the "main interface", then click on "Contacts" to enter the contacts interface. Then, enter the target contact in the "search box" of the contacts interface, click on the avatar icon of the "target contact". After that, after entering the conversation interface of the target contact, one can enter the message to be sent in the conversation message input field of the conversation interface. Finally, click on the "Send" option in the conversation interface to complete the message sending. Based on this, the knowledge graph includes multiple entity nodes such as the main interface, the "Contacts" icon, the search box in the contacts interface, etc., and these multiple entity nodes form an entity node link, that is: main interface - "Contacts" icon - search box in the contacts interface - avatar of the "target contact" - message input field in the conversation interface - send. This entity node link actually indicates the operation path for sending a message to the "target contact".

[0226] Based on this, if there is no operation sequence in the operation sequence set that matches the operation instruction, the present application can also construct an operation sequence corresponding to the operation instruction based on the knowledge graph. Of course, after constructing the operation sequence, the operation sequence can be executed.

[0227] For ease of understanding, the following takes a possible implementation manner of processing an operation instruction based on a knowledge graph as an example for illustration. As Figure 8 , shows a schematic diagram of an implementation process for processing an operation instruction based on a knowledge graph in an embodiment of the present application. This embodiment may include:

[0228] S801, in response to an operation instruction input by a user, if there is no operation sequence in the operation sequence set that matches the operation instruction, determine the operation intention of the operation instruction and the operation parameters related to the operation intention in the operation instruction.

[0229] For example, the operation parameter is what is included in the operation instruction and is required for the operation corresponding to the operation intention. For example, taking the operation instruction of sending a message to Xiaoming to tell him to meet at the school gate at 2 pm, then the operation intention is to send an instant messaging message. The operation parameters include: the contact person "Xiaoming" of the message, and the message content "Meet at the school gate at 2 pm".

[0230] S802, based on the operation intention, determine the target application to be called, jump to the target application, and determine the target operation interface where the target application is currently located.

[0231] Among them, determining which specific operation interface the target application is currently in can be achieved through this large language model or other models. In order to more efficiently determine which operation interface the target operation interface that the target application is currently in is, this application can adopt the method mentioned in the previous embodiment to obtain the interface directory entry of the target operation interface that the target application is currently in, so as to determine the category of the target operation interface based on the interface directory entry. For example, the target operation interface is the main interface, or the conversation interface, etc.

[0232] S803. Based on the target operation interface and the operation intention, determine an entity node link from the knowledge graph of the target application.

[0233] For example, based on the target operation interface, match a target entity node from the knowledge graph of the target application, and determine an entity node link that starts from the target entity node and can achieve the operation intention. Among them, the starting node of the entity node link is the target entity node, and the last entity node in the entity node link is the node that matches the operation intention.

[0234] Still taking the example of sending a WeChat message to Xiaoming above: Assume that the target operation interface is the main interface of the target application. It is necessary to find the entity node of "main interface" from the knowledge graph, and then search for the entity node of "send" (or send a message, etc.) from each entity node relationship chain connected to the entity node of the main interface, and determine the shortest entity node link between the entity node of "main interface" and the entity node of "send".

[0235] S804. Based on the operation parameter and the entity node link, generate an operation sequence that matches the operation intention.

[0236] For example, in the shortest entity node link from "main interface" to "send", each entity node after the main interface in each entity node link can be sequentially determined as an operation to be executed in the operation sequence. And if parameters need to be filled in the entity node, the relevant operation parameters can be filled into the operation corresponding to the entity node. For example, the operation sequence includes: "Contacts" icon - "Search box" on the contacts interface - "Target contact" avatar - Message input field on the conversation interface - Send. Then, in the operation of the search box on the contacts interface, the contact to be searched, that is, "Xiaoming", needs to be filled in to obtain the operation of searching for "Xiaoming" in the search box. And in the operation of the message input field on the conversation interface, the parameter of "See you at the school gate at 2 pm" needs to be filled in to achieve the operation of inputting "See you at the school gate at 2 pm" in the message input field.

[0237] S805. Execute the operation sequence that matches the operation intention.

[0238] For example, the operation sequence can be executed successively with the help of a large language model, etc., and there is no specific limitation.

[0239] On the other hand, corresponding to an information processing method provided by the present application, the present application also provides an information processing device. For example Figure 9 , a schematic diagram of a composition structure of the information processing device provided by the present application is shown. The device in this embodiment may include:

[0240] A tree acquisition unit 901, configured to obtain a description node tree corresponding to the current display interface in response to an instruction indicating to acquire display interface description information, where the description node tree includes: at least one node for representing interface elements on the display interface and a hierarchical relationship between different nodes, and different nodes correspond to different interface elements;

[0241] A directory generation unit 902, configured to generate interface directory information for expressing the display interface based on the hierarchical relationship between the nodes in the description node tree, using each node in the description node tree as a directory entry and the node description text of the node as the content of the directory entry;

[0242] A directory feedback unit 903, configured to feedback the interface directory information.

[0243] In a possible implementation manner, the device may further include: a tree update unit, configured to, before the target generation unit generates the interface directory information, if a node in the description node tree meets a target condition, determine a target interface area associated with the interface element represented by the node in the display interface, and generate a node description text for the node based on the display content in the target interface area, so as to update the description node tree;

[0244] Wherein, the node meeting the target condition includes: there is no node description text for describing the interface element represented by the node in the node.

[0245] In another possible implementation manner, the node meeting the target condition in the tree update unit further includes: there is no area overlap between the target interface area corresponding to the node and the interface areas corresponding to other nodes having node description texts, and the interface areas corresponding to the other nodes are the areas of the interface elements represented by the other nodes in the display interface.

[0246] In another possible implementation manner, the information processing device further includes:

[0247] A node deletion unit, configured to remove nodes meeting the deletion condition in the updated description node tree before the target generation unit generates the interface directory information; where the nodes meeting the deletion condition include at least one of the following:

[0248] A leaf node without node description text;

[0249] The child nodes of the parent node of the node only include the node, the node has only one child node, and the node does not have node description text.

[0250] In another aspect, corresponding to another information processing method provided by this application, this application also provides another information processing device. As Figure 10 , a schematic diagram of a composition structure of another information processing device provided by this application is shown. The device in this embodiment may include:

[0251] A node matching unit 1001, configured to, in response to an instruction to perform a target manipulation operation, determine a target node in the display interface that matches the manipulation node information of the target manipulation operation based on the manipulation node information of the target manipulation operation and the display interface description information of the display interface; wherein, the display interface description information is interface directory information, each directory entry in the interface directory information corresponds to a node of the display interface, and the content of the directory entry is the node description text of a node corresponding to the directory entry;

[0252] An operation simulation unit 1002, configured to simulate a manipulation operation on a target interface element corresponding to the target node in the display interface based on the manipulation information of the target manipulation operation.

[0253] In a possible implementation manner, the operation simulation unit includes:

[0254] A region determination subunit, configured to determine a target interface region in the display interface that is associated with the target interface element represented by the target node;

[0255] A position determination subunit, configured to determine a relative manipulation position of the target manipulation operation relative to the target interface region based on the relative manipulation coordinates recorded in the manipulation information of the target manipulation operation;

[0256] A manipulation simulation subunit, configured to simulate a manipulation operation at the relative manipulation position in the display interface.

[0257] In another possible implementation manner, the node matching unit includes:

[0258] A first matching subunit, configured to, in response to an instruction to perform a target manipulation operation, if it is determined based on the display interface description information of the display interface that there is a unique node in the display interface whose node description text of the corresponding directory entry matches the manipulation node information of the target manipulation operation, determine the unique node as the target node;

[0259] A second matching subunit, configured to, if based on the display interface description information of a display interface, it is determined that there are multiple candidate nodes in the display interface where the node description text of the corresponding directory entry matches the manipulation node information of the target manipulation operation, determine, based on the operation interface description information of the target manipulation operation, a target node with the highest matching degree with the manipulation node information from the multiple candidate nodes.

[0260] In another possible implementation manner, when the node matching unit responds to an instruction to execute a target manipulation operation, it is specifically configured to respond to an instruction to execute an operation sequence, determine the target manipulation operation to be simulated currently in the operation sequence, where the operation sequence includes multiple manipulation operations, and the target manipulation operation is one of the multiple manipulation operations.

[0261] In another possible implementation manner, when the node matching unit responds to an instruction to execute an operation sequence, it is specifically configured to respond to an operation instruction input by a user, and determine an operation sequence matching the operation instruction from a recorded operation sequence set.

[0262] In another possible implementation manner, the apparatus further includes:

[0263] A sequence construction unit, configured to, if there is no operation sequence matching the operation instruction in the operation sequence set, construct an operation sequence corresponding to the operation instruction based on a knowledge graph, where the knowledge graph includes relationships between multiple entity nodes, and an entity node represents an operation interface or a manipulable functional element in the operation interface.

[0264] An embodiment of the present application further provides an electronic device. As Figure 11 shown, it shows a schematic structural diagram of a composition of the electronic device. The electronic device includes at least a processor 1101 and a memory 1102;

[0265] The processor 1101 is configured to execute the information processing method described in any one of the above embodiments;

[0266] The memory 1102 is configured to store a program required for the processor to execute operations.

[0267] It can be understood that the electronic device may further include a display unit 1103 and an input unit 1104.

[0268] Of course, the electronic device may also have Figure 11 more or fewer components, and this is not limited.

[0269] In an embodiment of the present application, a computer program product is further provided, including computer-readable instructions. When the computer-readable instructions run on an electronic device, the electronic device is enabled to implement any one of the information processing methods provided in the embodiments of the present application.

[0270] In an embodiment of the present application, a computer-readable storage medium is further provided. The storage medium carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device is enabled to implement any one of the information processing methods provided in the embodiments of the present application.

[0271] In addition, it should be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place, or may be distributed to multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment. In addition, in the drawings of the device embodiments provided in the present application, the connection relationships between modules indicate that they have communication connections, which can be specifically implemented as one or more communication buses or signal lines.

[0272] Through the description of the above embodiments, those skilled in the art can clearly understand that the present application can be implemented by means of software plus necessary general hardware, and of course, can also be implemented by dedicated hardware including application-specific integrated circuits, dedicated CPUs, dedicated memories, dedicated components, etc. Generally, functions completed by computer programs can be easily implemented by corresponding hardware, and the specific hardware structures for implementing the same function can also be various, such as analog circuits, digital circuits or dedicated circuits. However, for the present application, in more cases, software program implementation is a better implementation manner. Based on such an understanding, the technical solution of the present application, in essence, or the part that makes a contribution to the prior art, can be embodied in the form of a software product. The computer software product is stored in a readable storage medium, such as a floppy disk, a USB flash drive, a mobile hard disk, a ROM, a RAM, a magnetic disk or an optical disc of a computer, and includes several instructions for enabling a computer device (which may be a personal computer, a training device, or a network device, etc.) to execute the methods described in the various embodiments of the present application.

[0273] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product.

[0274] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the processes or functions described in the embodiments of the present application are generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, training device, or data center to another website, computer, training device, or data center by wire (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or wirelessly (such as infrared, wireless, microwave, etc.). The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a training device or data center that includes one or more integrated available media. The available medium may be a magnetic medium (such as a floppy disk, hard disk, magnetic tape), an optical medium (such as a DVD), or a semiconductor medium (such as a solid state disk (SSD)).

Claims

1. An information processing method, comprising: In response to an instruction to obtain display interface description information, a description node tree corresponding to the current display interface is obtained, wherein the description node tree includes: at least one node for representing an interface element on the display interface and a hierarchical relationship between different nodes, and different nodes correspond to different interface elements; Taking each node in the description node tree as a directory entry and the node description text of the node as the content of the directory entry, based on the hierarchical relationship between the nodes in the description node tree, generate interface directory information for describing the display interface; Feedback the interface directory information.

2. The information processing method according to claim 1, before generating the interface directory information, further comprising: If a node in the description node tree meets a target condition, determining a target interface area in the display interface associated with the interface element represented by the node, and generating a node description text for the node based on display content in the target interface area to update the description node tree; The node meeting the target condition includes: the node does not contain a node description text for describing the interface element represented by the node.

3. The information processing method according to claim 2, wherein the node satisfies the target condition further comprises: There is no regional overlap between the target interface area corresponding to the node and the interface areas corresponding to other nodes having node description texts, and the interface areas corresponding to the other nodes are the areas of the interface elements represented by the other nodes in the display interface.

4. The information processing method according to claim 3, further comprising: Remove the nodes that meet the deletion condition in the updated description node tree; The nodes that meet the deletion conditions include at least one of the following: Leaf nodes without node description text; The parent node of the node has child nodes including only the node, the node has only one child node, and the node has no node description text.

5. An information processing method, comprising: In response to an instruction to perform a target manipulation operation, based on the manipulation node information of the target manipulation operation and the display interface description information of the display interface, determining a target node in the display interface that matches the manipulation node information; Based on the manipulation information of the target manipulation operation, simulating a manipulation operation on a target interface element corresponding to the target node in the display interface; The display interface description information is interface directory information, each directory entry in the interface directory information corresponds to a node of the display interface, and the content of the directory entry is a node description text of a node corresponding to the directory entry.

6. The information processing method according to claim 5, wherein the step of simulating a manipulation operation on a target interface element corresponding to the target node in the display interface based on the manipulation information of the target manipulation operation comprises: Determine a target interface area in the display interface that is associated with a target interface element represented by the target node; Determining a relative manipulation position of the target manipulation operation relative to the target interface area based on the relative manipulation coordinates recorded in the manipulation information of the target manipulation operation; A manipulation operation is simulated at the relative manipulation position in the display interface.

7. The information processing method according to claim 5, wherein the step of responding to the instruction to perform the target manipulation operation comprises: In response to an instruction to execute an operation sequence, a target manipulation operation to be currently simulated in the operation sequence is determined, the operation sequence includes a plurality of manipulation operations, and the target manipulation operation is one of the plurality of manipulation operations.

8. The information processing method according to claim 5, wherein the determining, based on the manipulation node information of the target manipulation operation and the display interface description information of the display interface, a target node in the display interface that matches the manipulation node information comprises: If, based on the display interface description information of the display interface, it is determined that there is a unique node in the display interface whose node description text of the corresponding directory entry matches the manipulation node information of the target manipulation operation, the unique node is determined as the target node; If, based on the display interface description information of the display interface, it is determined that there are multiple candidate nodes in the display interface whose node description text of the corresponding directory entry matches the control node information of the target control operation, then based on the operation interface description information of the target control operation, the target node with the highest degree of match with the control node information is determined from the multiple candidate nodes.

9. The information processing method according to claim 7, wherein the response to the instruction to execute the sequence of operations comprises: In response to an operation instruction input by a user, an operation sequence matching the operation instruction is determined from a set of recorded operation sequences.

10. The information processing method according to claim 9, further comprising: If there is no operation sequence matching the operation instruction in the operation sequence set, an operation sequence corresponding to the operation instruction is constructed based on a knowledge graph, wherein the knowledge graph includes relationships between multiple entity nodes, and an entity node represents an operation interface or a controllable functional element in the operation interface.

Images