Information processing method, device and electronic equipment

Abstract

The present application discloses an information processing method, device and electronic device, the method comprising: determining a second control in response to obtaining a target operation that satisfies a target condition; the target condition characterizes that the target operation is directed to a first control, the second control has a target display relationship with the first control, and the second control is used to indicate a touch event corresponding to the first control; generating target information based on the first control and the second control; the target information is used to locate the first control to perform the target operation.

Description

Technical Field

[0001] The present application relates to the field of information processing technology, and more specifically to an information processing method, device and electronic equipment. Background Art

[0002] In scenarios such as data analysis, it is necessary to record user operation information in the user operation interface in order to respond or analyze user behavior based on this operation information. However, due to problems such as inaccurate recording of relevant information, it is unable to meet the application requirements of actual data analysis scenarios, reducing the accuracy of information processing. Summary of the invention

[0003] In view of this, this application provides the following technical solutions:

[0004] An information processing method, comprising:

[0005] In response to obtaining a target operation that meets a target condition, determining a second control; the target condition indicates that the target operation is directed to the first control, the second control has a target display relationship with the first control, and the second control is used to indicate a touch event corresponding to the first control;

[0006] Based on the first control and the second control, target information is generated; the target information is used to locate the first control to perform the target operation.

[0007] Optionally, in response to obtaining a target operation that satisfies a target condition, determining a second control includes:

[0008] In response to obtaining a target operation that meets the target condition, determining a target display interface corresponding to the first control targeted by the target operation;

[0009] Based on the interface layout information of the target display interface, a second control is determined, and the first control and the second control have a target position relationship in the target display interface.

[0010] Optionally, the interface layout information includes at least control arrangement information of each control in the target display interface and display feature information of each control, wherein determining the second control based on the interface layout information of the target display interface includes:

[0011] Determining the type of each control based on the display characteristic information of each control in the target display interface;

[0012] Based on the first type of the first control, determining a second type corresponding to the first type;

[0013] Based on the control arrangement information, determine relative position information between each control corresponding to the second type and the first control;

[0014] Based on the relative position information, a second control having a target position relationship with the first control is determined.

[0015] Optionally, in response to obtaining a target operation that satisfies a target condition, determining a second control includes:

[0016] In response to obtaining a target operation that meets the target condition, determining a target page corresponding to the first control targeted by the target operation;

[0017] Parsing the target page to obtain a document object model target tree structure corresponding to the target page, the target tree structure including at least one node and a hierarchical relationship between the nodes, the node representing a control in the target page, and the hierarchical relationship between the nodes representing an association relationship between the controls in a display layer in the target page;

[0018] Based on the target tree structure, a second control corresponding to the first control is determined.

[0019] Optionally, determining the second control corresponding to the first control based on the target tree structure includes:

[0020] Determine a candidate control corresponding to the first control based on the hierarchical relationship of each node in the target tree structure;

[0021] Based on the description information of the nodes corresponding to the candidate controls in the target structure tree, a second control is determined from the candidate controls.

[0022] Optionally, generating target information based on the first control and the second control includes:

[0023] Respectively obtaining operation information corresponding to the first control and control feature information of the second control;

[0024] The operation information and the control feature information are input into a generative model to obtain output information of the generative model; the target information includes the output information.

[0025] Optionally, the target information includes an executable file, and the method further includes:

[0026] The executable file is executed to locate the first control.

[0027] Optionally, positioning the first control includes:

[0028] Determine, based on the control type included in the target information, a plurality of candidate first controls corresponding to the target page;

[0029] Determine the degree of matching between the target description text corresponding to each of the candidate first controls and the text included in the target information; the target description text represents the description text corresponding to the candidate first controls that meets the priority condition;

[0030] Based on the degree of matching, a first control is determined from among the candidate first controls.

[0031] An information processing device, comprising:

[0032] a determination unit, configured to determine a second control in response to obtaining a target operation that satisfies a target condition; the target condition indicates that the target operation is directed to the first control, the second control has a target display relationship with the first control, and the second control is used to indicate a touch event corresponding to the first control;

[0033] A generating unit is used to generate target information based on the first control and the second control; the target information is used to locate the first control to perform the target operation.

[0034] An electronic device, comprising:

[0035] A memory, used to store applications and data generated by the operation of the applications;

[0036] A processor, configured to execute the application program to implement:

[0037] In response to obtaining a target operation that meets a target condition, determining a second control; the target condition indicates that the target operation is directed to the first control, the second control has a target display relationship with the first control, and the second control is used to indicate a touch event corresponding to the first control;

[0038] Based on the first control and the second control, target information is generated; the target information is used to locate the first control to perform the target operation. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are merely embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0040] Figure 1 A flowchart of an information processing method provided in an embodiment of the present application;

[0041] Figure 2A schematic diagram of a user interface for volume configuration provided by an embodiment of the present application;

[0042] Figure 3 A schematic diagram of a processing flow for determining a second control based on a document object model provided in an embodiment of the present application;

[0043] Figure 4 A schematic diagram of a DOM target structure tree provided in an embodiment of the present application;

[0044] Figure 5 A schematic diagram of the structure of an information processing device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0045] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0046] The terms "first" and "second" in this application are used to distinguish different objects, rather than to describe a specific order. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but may include steps or units that are not listed.

[0047] The information processing method provided in the embodiment of the present application is used to locate the relevant controls in the display interface corresponding to the electronic device so that the control accurately performs the target operation. Among them, the electronic device can be a device with a display function, specifically an electronic device including a display component, or an electronic device capable of connecting a display component. For example, the electronic device can be a portable device such as a mobile phone or a computer, or a vehicle-mounted terminal or a wearable device. The specific information processing method can be applied to data analysis scenarios, including scenarios for analyzing and predicting user behavior. For example, the user's current operation data can be inferred through a large model to predict the user's actual operation purpose. It can also be based on the information processing method to achieve the reproduction of user operations to determine the state of the control targeted by the user operation, such as whether a fault occurs, so as to ensure the normal operation of the device.

[0048] See also Figure 1 , is a flow chart of an information processing method provided in an embodiment of the present application, the method may include the following steps:

[0049] S101. In response to obtaining a target operation that meets a target condition, determine a second control.

[0050] S102: Generate target information based on the first control and the second control.

[0051] In step S101, the target condition characterizes that the target operation is directed to the first control. The second control has a target display relationship with the first control, and the second control is used to indicate the touch event corresponding to the first control. In an embodiment of the present application, a control is a basic element of a user interface (UI) that allows a user to interact with the software. Different types of controls have different functions. For example, controls may include dynamic controls such as buttons, text boxes, check boxes, drop-down lists, sliders, progress bars, and may also include static controls such as labels, images, and tables. Taking a slider as an example, a user is allowed to select a corresponding value within a range by sliding; taking a label as an example, a label displays read-only text in the user interface to identify a control or provide information.

[0052] The control can also be distinguished according to the difference of its display area, such as the control can be a control in the display interface of the electronic device, the display interface can be the front interface of the electronic device, and the control can be an operable object on the display interface. For example, when the display interface is a web page, the control on the web page can include a button control, a text box control, a selection control, etc. When the display interface is a user interface, the control can be divided into a selected control, a sliding control, a drop-down selection control, an input control, etc. according to its operation type. When the electronic device detects the target operation for the current display interface, it will parse the target operation. If the target operation is an operation for the first control, it is determined that the target operation meets the target condition. Specifically, the first control can be determined according to the display area in the display interface corresponding to the target operation and / or the operation type of the target operation. If there is a unique control in the display area corresponding to the target operation, the control is determined as the first control. If there are at least two controls in the display area corresponding to the target operation, the first control can also be determined according to the type of the target operation and the type of operation that each control can affect. For example, the current display area includes a control that can respond to a sliding operation and a control that can respond to a click operation. If the target operation represents a sliding operation, the first control is the control that can respond to the sliding operation. For example, the current target operation is an operation for adjusting the brightness in the display interface, and the first control is a brightness adjustment control.

[0053] In order to accurately control the first control to perform the target operation, the target operation that meets the target condition is obtained in the embodiment of the present application, and the second control is determined. The second control has a target display relationship with the first control, and the second control is used to indicate the touch event corresponding to the first control. Thus, the touch event corresponding to the first control can be indicated according to the second control. After the first control is determined, the second control can be determined according to the position information of the first control in the display interface. Correspondingly, the second control can be the control closest to the first control. The second control can also be determined according to the visualization information of the first control in the display interface, such as the visualization information includes the displayable response information corresponding to the control, and then the control matching the visualization information of the first control is determined as the second control. Specifically, the first control and the second control can be controls in the same display area and have the same type of operation response information, such as the first control and the second control can both be controls that can respond to click operations. Correspondingly, the control with a target display relationship can also be determined according to the control configuration description information of the first control, such as the control element corresponding to the control can be determined through the control configuration description information, and then the second control element corresponding to the first control element in the display area is determined according to the first control element corresponding to the first control in the current display area, so that the control corresponding to the second control element is determined as the second control. For example, the control element includes an audio control module of an electronic device, and the first control and the second control may be controls for the audio function module that are closest to each other in the same display area. Thus, the first control can be assisted in positioning according to the information of the second control, or the relevant information of the second control can be used as supplementary information of the first control, so that the target operation for the first control can be recorded and executed more accurately.

[0054] After obtaining the second control corresponding to the first control, the touch event for the first control can be determined according to the control feature information of the second control. Specifically, the touch event may include information such as the touch object, touch configuration, and touch execution order. For example, the first control is a sliding control, and the second control may be a text identification control, so that the touch object of the sliding operation corresponding to the first control can be determined according to the second control. In step S102, target information is generated based on the first control and the second control. The target information is used to locate the first control to perform the target operation. That is, the specific first control and the operation purpose of the target operation for the first control can be clarified through the target information, so as to control the first control to accurately perform the target operation. Among them, the target information can be based on the relationship between the first control and the second control, and the information of the first control is located in the operation configuration information. Specifically, according to the control feature information corresponding to the second control, the control feature of the first control is determined, so that the information of the first control is determined in each control information included in the operation configuration information, and the target operation for the first control is automatically performed based on the operation configuration information.

[0055] Specifically, when a user clicks on a first control in a user interface, the system will record relevant information about the first control, such as the control ID, text or style, class information, etc. of the first control. If the control itself is not configured with relevant detailed information, it will be impossible to accurately record or execute the corresponding operation based on the existing recorded information. For example, when the control is a button, if the user clicks on the button, it can only record that a button representing a switch is clicked, but it cannot record the actual on or off object targeted by the button, and thus it is impossible to accurately record the operation information or accurately execute the operation for the button.

[0056] In the embodiment of the present application, even when the recorded information for the first control is incomplete, it is possible to generate target information for accurately locating the first control based on the second control to perform the target operation, thereby accurately recording the operational meaning of the target operation for the first control and further accurately performing the target operation.

[0057] For an example, see Figure 2, is a schematic diagram of a user interface for volume configuration provided by an embodiment of the present application, in which a button control 201, a label control 202 representing a "microphone" identifier, a slider control 203, and a label control 204 representing a "microphone volume" are included. If the current target operation is for the button control 201, the processing system determines the second control according to the characteristic information of the button control 201 in the user interface of the current volume configuration, such as the position and display characteristics of the button control 201 in the application interface, that is, determines the second control having a target display relationship with the button control 201. For example, the label control 202 that is close to the button control 201 in the user interface and arranged in the same row can be determined as the second control. The label control 202 can indicate a touch event of the button control 201, such as indicating an operation object for the button control. When the user clicks the button control 201, the processing system will record that a button is clicked, that is, record the ID information of the control, etc., but cannot record whether the actual meaning of this button is to start or turn off the microphone. Therefore, based on the information processing method provided in the embodiment of the present application, the label control 202 corresponding to the button control 201 can be determined, so that based on the label control 202, it can be determined that the button control 201 is for turning the microphone on or off, so that complete target information can be generated, so that the information of the button control in the corresponding execution file can be accurately located to perform the target operation, that is, to control the microphone to turn on or off. In this way, when recording the operation information of the target operation for the button control 201, not only the ID information for the button control 201 is recorded, but also the information or instructions with the associated operation are recorded, so that the target operation can be accurately performed, or the complete information of the target operation can be accurately recorded, which is convenient for the subsequent application of the recorded information.

[0058] The following describes the relevant technical features of the information processing method in the embodiment of the present application in conjunction with specific application scenarios.

[0059] In the embodiment of the present application, after obtaining a target operation that meets the target condition, a second control having a target display relationship with the first control targeted by the target operation is determined, so as to determine the touch event corresponding to the first control based on the second control. Since the first control and the second control have a target display relationship, the second control can be determined based on the feature information corresponding to the current display interface.

[0060] In one implementation of the embodiment of the present application, a processing flow for determining a second control is also provided, the processing flow being used to execute Figure 1 In the illustrated embodiment, step S101 determines a second control in response to obtaining a target operation that meets a target condition. Specifically, the following steps may be included:

[0061] S1011. In response to obtaining a target operation that meets a target condition, determine a target display interface corresponding to a first control targeted by the target operation.

[0062] S1012: Determine a second control based on the interface layout information of the target display interface.

[0063] The first control and the second control have a target position relationship in the target display interface. The interface currently displayed by the first control is determined as the target display interface. Then the target display interface is analyzed to obtain the interface layout information of the target display interface. Specifically, the visualization information of the target display interface can be analyzed to obtain the interface layout information; wherein the visualization information can represent the information displayed in the target display interface, such as the position of the control, the display feature information of the control, etc. In this way, the interface layout information can be determined according to the visualization information, and the interface layout information can include the position information, display style information and association information between the controls in the target display interface. Further, according to the interface layout information, the second control corresponding to the first control is determined. Correspondingly, the target position relationship between the first control and the second control can be determined according to the characteristics of the controls in the current target display interface, such as the target position relationship can represent that the position distance between the first control and the second control in the target display interface is the smallest, or that the two are in the same row or the same column in the display interface, so as to determine the second control corresponding to the first control according to the restriction information corresponding to the target position relationship. In this way, the second control having a target position relationship with the first control can be accurately determined based on the interface layout information of the target display interface corresponding to the first control, so that the touch event indicating the first control can be accurately obtained based on the second control, which facilitates the subsequent positioning of the first control to perform the target operation.

[0064] In the process of determining the second control, there may be at least one control that satisfies a target position relationship with the first control. In order to obtain the second control more accurately, the second control may also be determined according to the control type of the first control.

[0065] In one implementation of the embodiment of the present application, the interface layout information includes at least control arrangement information of each control in the target display interface and display feature information of each control, wherein determining the second control based on the interface layout information of the target display interface includes:

[0066] Based on the display feature information of each control in the target display interface, the type of each control is determined; based on the first type of the first control, the second type corresponding to the first type is determined; based on the control arrangement information, the relative position information of each control corresponding to the second type and the first control is determined; based on the relative position information, a second control that has a target position relationship with the first control is determined.

[0067] Specifically, the display feature information of each control in the target display interface includes the display form, display mode, or response display information after being touched, etc. of the control, such as the display form may include information such as the display shape and color of the control; the display mode may include a transparent display mode, a non-transparent display mode, a shadow display mode of the control, or an animation display mode of the control. The response display information after being touched may include click jump, slide, drop-down menu, etc. Therefore, the controls with matching display feature information can be divided into controls of the same type, thereby obtaining the types of each control. For example, the types of each control can be divided according to the color of the control in the display interface. In order to further improve the accuracy of the control type division, the control type division can also be combined with multi-dimensional information, for example, according to the display form of the control and the response display information after being touched, etc., to determine the type of each control. Then, according to the first type of the first control, a second type corresponding to the first type is determined. The second type may be the same as the first type, or it may match the first type (such as being similar to the first type). For example, the first type represents a click control, and the corresponding second type may represent a label control or a click control; for another example, if the first type represents a search box, the second type may represent a control that is a list box that displays search results. If the control corresponding to the second type is unique, the control can be used as the second control. If the control corresponding to the second type is not unique, the relative position information between each control corresponding to the second type and the first control can also be determined based on the control arrangement information. For example, the second type of control that is in the same row as the first control is used as a candidate control, and the second control is determined in the candidate controls. Specifically, the relative position information between each control in the candidate controls and the first control can be calculated, such as calculating the relative distance between each control in the candidate controls and the first control. At this time, the target position relationship includes the condition of the shortest relative distance, that is, the control in the candidate controls that is closest to the first control is used as the second control. Still with Figure 2 For example, when the first control is a click control 201, the corresponding second type of control can be a label type control, that is, it can be control 202 and control 204, and then based on the control arrangement information, the relative position information of control 201 and control 202 and control 204 is determined. If the target position relationship includes a target relationship that the controls belong to the same row or the same column, the relative position information indicates that control 201 and control 202 are in the same row, and control 202 is determined as the second control. In this way, the second control corresponding to the first control can be determined more accurately according to the type of each control in the target display interface, which can meet the needs of application scenarios with complex display interface layouts.

[0068] In the embodiment of the present application, in addition to determining the second control corresponding to the first control according to the layout information of the display interface, the second control can also be determined according to the document object model to meet the processing requirements of different application scenarios.

[0069] In one implementation of the embodiment of the present application, a processing flow for determining a second control based on a document object model is also provided. Figure 3 , the processing flow may include the following steps:

[0070] S301: In response to obtaining a target operation that meets a target condition, determine a target page corresponding to a first control targeted by the target operation.

[0071] S302: Parse the target page to obtain a document object model target tree structure corresponding to the target page.

[0072] S303: Determine a second control corresponding to the first control based on the target tree structure.

[0073] The target page corresponding to the first control can be determined based on the application architecture, display configuration information, etc., and then the target page can be parsed to obtain the target tree structure of the document object model. For example, the HTML (Hyper Text Markup Language, standard markup language) document of the target page can be parsed to construct a target tree structure of the document object model (DOM). The target tree structure includes at least one node and a hierarchical relationship between each node. The node represents the control in the target page, and the hierarchical relationship between each node represents the association relationship between the display layers of the controls in the target page. According to the hierarchical relationship and / or association relationship of each node in the target tree structure, the associated nodes of each first node representing the first control are determined, and then the second node is determined in the associated node, and the control represented by the second node is determined as the second control.

[0074] For details, see Figure 4, which shows a schematic diagram of a DOM target structure tree provided by an embodiment of the present application. The DOM target structure tree includes several nodes, which can be arranged according to a tree structure. The entire target page in the embodiment of the present application (such as, the target page can be a web page) corresponds to a document node, and each control in the target page corresponds to an element node, such as the first element node represents the first control in the target page, the second element node represents the second control in the target page, and the third element node represents the third control in the target page. If the target page is a web page, it can be the root node of all nodes. The hierarchical relationship of each element node can characterize the association relationship between the display layers of the element nodes in the target page, so that the element nodes with corresponding relationships and their corresponding controls can be determined. In some application scenarios, the DOM target tree structure can also include attribute nodes representing element attributes; text nodes containing text content between elements; annotation nodes containing annotation content in the document, etc. Specific nodes or attributes can be searched through the document object model target tree structure, which is convenient for complex queries and operations on the document structure of the target page, thereby improving the efficiency of obtaining node corresponding controls, which helps to improve the performance and response speed of information processing.

[0075] Furthermore, in an embodiment of the present application, based on the target tree structure, the second control corresponding to the first control is determined, including: based on the hierarchical relationship of each node in the target tree structure, determining the candidate control corresponding to the first control; based on the description information of the nodes corresponding to each candidate control in the target tree structure, determining the second control in the candidate controls.

[0076] Specifically, the candidate controls corresponding to the first control represent controls that have a hierarchical correspondence with the first control, and then the description information of the nodes corresponding to these candidate controls is obtained, and the description information represents the text interpretation content corresponding to the nodes, and the second control is determined based on the description information. Specifically, the target description information can be determined based on the position priority of the description information and the node corresponding to the first control, thereby determining the second control corresponding to the target description information.

[0077] For example, after obtaining the DOM target tree structure of the target page where the first control is located, in order to obtain the description information in the target tree structure, that is, to obtain the text information contained therein, the label controls (such as icons, pictures) in the target page can be converted into text information. Then, the level where these text information are located in the target tree structure, and the level where the first control targeted by the target operation is located in the target tree structure are obtained. Then, the second control is determined based on this information. Figure 2As shown, in the target tree structure, the "microphone" icon node corresponding to the control 202 is at the top of the target tree structure, and the corresponding "microphone" icon node is at the bottom layer, as well as the node corresponding to the control 201 (the checkbox button of the microphone is usually used to control the on and off state of the microphone), and there are also "microphone volume" nodes and "volume adjustment slider" nodes at the same level. When a certain control is triggered to click or other action records are recorded, the position of the node corresponding to the control itself in the DOM target tree structure is traced back. For example, taking the click control 201 as an example, the node description information (i.e., text information) closest to the node corresponding to the control 201 can be obtained first as the text interpretation of this control, that is, the node closest to the node corresponding to the control 201 is used as the second control. Then these text information and the control information clicked are merged to obtain the target information, which is used to locate the first control to perform the target operation. In this way, the recorded information when the target operation is directed to the first control can be fully recorded, so that the target operation can be performed more accurately.

[0078] In one implementation of the embodiment of the present application, in order to efficiently and accurately generate target information, the large model technology can also be used to process the relevant information. Specifically, based on the first control and the second control, the target information is generated, including: respectively obtaining the operation information corresponding to the first control and the control feature information of the second control; inputting the operation information and the control feature information into the generative model to obtain the output information of the generative model; the target information includes the output information.

[0079] Among them, the operation information corresponding to the first control may be the response operation information of the first control for the target operation, such as the first control is a control that represents the microphone is turned on or off. When the user clicks the first control, the corresponding operation information may be information about turning the microphone on or off. The control feature information of the second control includes the type of the second control, text description information or display features of the second control, response features and other information. The generative model may be a model created based on large model technology, which has the function of automatically generating corresponding output information based on input information. Specifically, the generative model can be learned based on a large number of data samples to obtain the corresponding relationship between input information and output information in the data samples, so that in the process of actual application, the input information can be input into the generative model to obtain the corresponding output information. For example, the generative model may be GenAI, and GenAI (Generative Artificial Intelligence) refers to general generative artificial intelligence, which refers to an artificial intelligence system that can perform any intelligent task. It can be used to generate natural language instructions, which are used to guide the corresponding control components to perform specific tasks. The generative model learns the characteristics and laws of input information by training a large amount of diverse data. Typically, the model parameters range from hundreds of millions to hundreds of billions (model parameters are variables that control the behavior of the target model) and can capture complex relationships and patterns in the input information.

[0080] For example, in the project development phase, the feasibility of the project is usually verified by collecting corresponding information. For example, in the browser agent project, it is necessary to learn or train the user behavior in the browser-based large model so that the browser agent can accurately predict the operation result corresponding to the operation based on the current user operation, thereby performing the target operation. An information recording tool can be used to record the user's behavior in the current display interface. In the recording process, the user's operations, such as mouse clicks, scrolling, keyboard input, etc., can be recorded, and some information about the specific control clicked will also be recorded, such as the control's ID, text, xpath (a query language for selecting nodes in XML documents) or css (cascading style sheet) class information. For example, in the verification process of this project, the first control is an on or off button. When the user clicks the on or off button of the microphone, the information recording tool will first record the information of clicking a button. If the automatic execution instruction is directly generated with this information, it is impossible to clarify whether the actual meaning of the button is to turn on or off the microphone, which will lead to the failure of the automatic operation. Therefore, in an embodiment of the present application, a second control corresponding to the first control is determined, and then the operation information corresponding to the first control and the control feature information of the second control are input into the generative model, and then the output information of the model is determined as the target information, that is, the target information represents a natural language instruction, so that executing the instruction can turn the microphone on or off, complete the automation task, and improve the accuracy of the automation operation.

[0081] Furthermore, based on the information processing method provided in the embodiment of the present application, the user operation behavior can be reproduced, thereby generating operation record data that matches the user's operation intention. The large model in the above-mentioned browser agent can be trained based on these data that can accurately reproduce the user's operation behavior, so that the subsequent browser agent can accurately locate the control corresponding to the user operation after receiving the user operation, and control the execution of the operation.

[0082] In one implementation of the embodiment of the present application, the target information includes an executable file, and the method further includes: executing the executable file to locate the first control. The executable file may be an automated execution script, and the automated execution script is used to locate the first control. Correspondingly, the automated script may also be used to locate the first control and perform the corresponding target operation. In this way, the information processing method of the embodiment of the present application may be used to generate an executable file for the current target operation, and the first control may be located by executing the executable file, or the corresponding target operation may be automatically performed after the first control is located, thereby meeting the requirements of efficient and automated information processing.

[0083] In an embodiment of the present application, a processing process for locating a first control is also provided, the processing process comprising: determining multiple candidate first controls corresponding to a target page based on the control type included in the target information; determining the degree of match between the target description text corresponding to each candidate first control and the text included in the target information; and determining the first control from the candidate first controls based on the degree of match. The target description text represents the description text corresponding to the candidate first control that meets the priority condition.

[0084] Based on the control type in the target information, firstly, multiple candidate first controls with the control type are indexed from the target tree structure of the document object model corresponding to the target page. Then, the target description text of each candidate first control is searched, such as the priority condition represents the condition with the highest priority, and the target description text represents the text with the highest priority. Then, the target description text is matched with the text included in the target information, and the candidate first control corresponding to the text with the highest matching degree is determined as the first control, that is, the control that needs to be automatically executed, and then the target operation is executed.

[0085] Specifically, in the process of determining the target description text corresponding to each candidate first control, in order to improve the efficiency of determining the target description text, the nodes without text information can be first removed from the target tree structure of the document object model, that is, all nodes with text information and the nodes of the first control targeted by the target operation are retained, and the interference of other nodes without text information is eliminated. Further, a pure content tree including text information can be regenerated, thereby facilitating the determination of the description text. Then, according to the traversal priority, the text of the left side, right side and parent node of the node corresponding to the first control can be selected as the target description text. The priority order can be determined based on the target tree structure of the document object model, or it can be determined based on the current application scenario. For example, the priority order can be that if there is a left node description information, the description information of the left node is selected as the target description text, and if there is no left node, the description information of the right node is selected as the target description text. If there is no description information of the left and right nodes, the description information of the parent node (i.e., the upper layer node) is selected as the target description text. Then, according to the matching degree of the target description text and the text included in the target information, the first control is located. After locating the first control, an execution instruction is generated according to the target information to execute the target operation. Specifically, the execution instruction can be generated based on the target description text and the control information of the first control. For example, the information corresponding to the target description text is "microphone". The control information of the first control can be the meaning of the control itself, such as "switch", "select or cancel", "fold or expand", "activate or deactivate", "open or close", etc., and then the target description text and the control information of the first control are input into the generative model to output the execution instruction. If the control information of the first control is "open or close", the execution instruction can be an execution instruction for controlling the microphone to turn on or off. If it is an execution instruction for controlling the microphone to turn on, the instruction can be automatically executed to control the microphone to turn on. Thereby, it is possible to accurately locate the first control targeted by the current target operation in the execution file and perform automated operations, thereby improving the execution efficiency and accuracy of automated tasks.

[0086] In another embodiment of the present application, an information processing device is also provided. Figure 5 , the device comprises:

[0087] A determination unit 501 is configured to determine a second control in response to obtaining a target operation that satisfies a target condition; the target condition indicates that the target operation is directed to a first control, the second control has a target display relationship with the first control, and the second control is used to indicate a touch event corresponding to the first control;

[0088] The generating unit 502 is used to generate target information based on the first control and the second control; the target information is used to locate the first control to perform the target operation.

[0089] Optionally, the determining unit includes:

[0090] A first determining subunit, configured to respond to obtaining a target operation that satisfies a target condition and determine a target display interface corresponding to a first control targeted by the target operation;

[0091] The second determining subunit is used to determine a second control based on the interface layout information of the target display interface, and the first control and the second control have a target position relationship in the target display interface.

[0092] Optionally, the interface layout information includes at least control arrangement information of each control in the target display interface and display characteristic information of each control, wherein the second determining subunit is configured to:

[0093] Determining the type of each control based on the display characteristic information of each control in the target display interface;

[0094] Based on the first type of the first control, determining a second type corresponding to the first type;

[0095] Based on the control arrangement information, determine relative position information between each control corresponding to the second type and the first control;

[0096] Based on the relative position information, a second control having a target position relationship with the first control is determined.

[0097] Optionally, the determining unit includes:

[0098] A third determining subunit is configured to, in response to obtaining a target operation that satisfies a target condition, determine a target page corresponding to the first control targeted by the target operation;

[0099] A parsing subunit, used to parse the target page to obtain a document object model target tree structure corresponding to the target page, wherein the target tree structure includes at least one node and a hierarchical relationship between each node, wherein the node represents a control in the target page, and the hierarchical relationship between each node represents an association relationship between the controls in the display layer in the target page;

[0100] The fourth determining subunit is used to determine the second control corresponding to the first control based on the target tree structure.

[0101] Optionally, the fourth determining subunit is configured to:

[0102] Determine a candidate control corresponding to the first control based on the hierarchical relationship of each node in the target tree structure;

[0103] Based on the description information of the nodes corresponding to the candidate controls in the target structure tree, a second control is determined from the candidate controls.

[0104] Optionally, the generating unit includes:

[0105] A first acquisition subunit, used to respectively acquire operation information corresponding to the first control and control feature information of the second control;

[0106] The second acquisition subunit is used to input the operation information and the control feature information into a generative model to obtain output information of the generative model; the target information includes the output information.

[0107] Optionally, the target information includes an executable file, and the device further includes:

[0108] A positioning unit is used to execute the executable file and locate the first control.

[0109] Optionally, positioning the first control includes:

[0110] Determine, based on the control type included in the target information, a plurality of candidate first controls corresponding to the target page;

[0111] Determine the degree of matching between the target description text corresponding to each of the candidate first controls and the text included in the target information; the target description text represents the description text corresponding to the candidate first controls that meets the priority condition;

[0112] Based on the degree of matching, a first control is determined from among the candidate first controls.

[0113] It should be noted that the specific implementation of each unit and sub-unit in this embodiment can refer to the corresponding content in the previous text and will not be described in detail here.

[0114] In another embodiment of the present application, a readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the information processing method described above is implemented.

[0115] In another embodiment of the present application, an electronic device is further provided, and the electronic device may include:

[0116] A memory, used to store applications and data generated by the operation of the applications;

[0117] A processor, configured to execute the application program to implement:

[0118] In response to obtaining a target operation that meets a target condition, determining a second control; the target condition indicates that the target operation is directed to the first control, the second control has a target display relationship with the first control, and the second control is used to indicate a touch event corresponding to the first control;

[0119] Based on the first control and the second control, target information is generated; the target information is used to locate the first control to perform the target operation.

[0120] It should be noted that the specific implementation of the processor in this embodiment can refer to the corresponding content in the previous text and will not be described in detail here.

[0121] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0122] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0123] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0124] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. An information processing method, comprising: In response to obtaining a target operation that satisfies the target condition, determining a second control; The target condition indicates that the target operation is directed to a first control, the second control has a target display relationship with the first control, and the second control is used to indicate a touch event corresponding to the first control; Generate target information based on the first control and the second control; The target information is used to locate the first control to perform the target operation.

2. The method according to claim 1, wherein the step of determining the second control in response to obtaining a target operation that satisfies a target condition comprises: In response to obtaining a target operation that meets the target condition, determining a target display interface corresponding to the first control targeted by the target operation; Based on the interface layout information of the target display interface, a second control is determined, and the first control and the second control have a target position relationship in the target display interface.

3. The method according to claim 2, wherein the interface layout information at least includes control arrangement information of each control in the target display interface and display characteristic information of each control, wherein: The determining the second control based on the interface layout information of the target display interface includes: Determining the type of each control based on the display characteristic information of each control in the target display interface; Based on the first type of the first control, determining a second type corresponding to the first type; Based on the control arrangement information, determine relative position information between each control corresponding to the second type and the first control; Based on the relative position information, a second control having a target position relationship with the first control is determined.

4. The method according to claim 1, wherein the step of determining the second control in response to obtaining a target operation that satisfies a target condition comprises: In response to obtaining a target operation that meets the target condition, determining a target page corresponding to the first control targeted by the target operation; Parsing the target page to obtain a document object model target tree structure corresponding to the target page, the target tree structure including at least one node and a hierarchical relationship between the nodes, the node representing a control in the target page, and the hierarchical relationship between the nodes representing an association relationship between the controls in a display layer in the target page; Based on the target tree structure, a second control corresponding to the first control is determined.

5. The method according to claim 4, wherein determining the second control corresponding to the first control based on the target tree structure comprises: Determine a candidate control corresponding to the first control based on the hierarchical relationship of each node in the target tree structure; Based on the description information of the nodes corresponding to the candidate controls in the target structure tree, a second control is determined from the candidate controls.

6. The method according to claim 1, wherein generating target information based on the first control and the second control comprises: Respectively obtaining operation information corresponding to the first control and control feature information of the second control; The operation information and the control feature information are input into a generative model to obtain output information of the generative model; the target information includes the output information.

7. The method according to claim 6, wherein the target information comprises an executable file, and the method further comprises: The executable file is executed to locate the first control.

8. The method according to claim 1, positioning the first control, comprising: Determine, based on the control type included in the target information, a plurality of candidate first controls corresponding to the target page; Determine the degree of matching between the target description text corresponding to each of the candidate first controls and the text included in the target information; The target description text represents the description text that satisfies the priority condition corresponding to the candidate first control; Based on the degree of matching, a first control is determined from among the candidate first controls.

9. An information processing device, comprising: A determining unit, configured to determine a second control in response to obtaining a target operation that satisfies a target condition; The target condition indicates that the target operation is directed to a first control, the second control has a target display relationship with the first control, and the second control is used to indicate a touch event corresponding to the first control; A generating unit, configured to generate target information based on the first control and the second control; The target information is used to locate the first control to perform the target operation.

10. An electronic device comprising: A memory, used to store applications and data generated by the operation of the applications; A processor, configured to execute the application program to implement: In response to obtaining a target operation that meets a target condition, determining a second control; the target condition indicates that the target operation is directed to the first control, the second control has a target display relationship with the first control, and the second control is used to indicate a touch event corresponding to the first control; Generate target information based on the first control and the second control; The target information is used to locate the first control to perform the target operation.

Images