Page element positioning method and device

Abstract

The embodiment of the present application provides a page element positioning method and device, the method comprises the following steps: receiving a user instruction of a target page; the target page comprises at least one page element; generating description text information according to the user instruction; the description text information comprises at least user intention and descriptive target; determining the similarity between at least one text information of the page element and the description text information; the text information is obtained by at least one text recognition scheme; determining the total similarity of the page element according to the similarity of the page element; and positioning a target page element from the page element in the target page according to the total similarity. The embodiment of the present application improves the robustness and accuracy of page element positioning.

Description

TECHNICAL FIELD

[0001] Embodiments of the present application relate to the technical field of computer, and in particular, to a page element positioning method, a page element positioning device, an electronic device and a computer readable storage medium. BACKGROUND

[0002] A DOM (Document Object Model Element) element is a structured representation of a page, which is a tree-shaped object collection generated by a browser after parsing code.

[0003] In a specific implementation, DOM element positioning is a commonly used technology in the field of browser automation testing. However, the accuracy of current DOM element positioning is not high. SUMMARY

[0004] In view of the above problems, a page element positioning method and device are provided to overcome the above problems or at least partially solve the above problems. The specific technical solutions are as follows:

[0005] In a first aspect of the present application, a page element positioning method is provided, which comprises:

[0006] receiving a user instruction of a target page; the target page comprising at least one page element;

[0007] generating description text information according to the user instruction; the description text information comprising at least user intention and descriptive target;

[0008] determining the similarity between at least one text information of the page element and the description text information; the text information being obtained by at least one text recognition scheme;

[0009] determining the total similarity of the page element according to the similarity of the page element;

[0010] positioning a target page element from the page elements in the target page according to the total similarity.

[0011] In an embodiment of the present application, determining the similarity between at least one text information of the page element and the descriptive target comprises:

[0012] screening a candidate page element from the page elements in the target page according to the user intention and / or the descriptive target;

[0013] determining the similarity between at least one text information of the candidate page element and the user intention and / or the descriptive target.

[0014] In an embodiment of the present application, determining the similarity between the at least one text information of the candidate page element and the user intent and / or the descriptive target comprises:

[0015] Obtaining a page element screenshot of a candidate page element in the target page;

[0016] Identifying text information of the page element screenshot through an optical character recognition technology, and determining a first similarity between the text information and the user intent and / or the descriptive target.

[0017] In an embodiment of the present application, determining the similarity between the at least one text information of the candidate page element and the user intent and / or the descriptive target comprises:

[0018] Obtaining page code of a candidate page element in the target page;

[0019] Extracting text information corresponding to the candidate page element from the page code, and determining a second similarity between the text information and the user intent and / or the descriptive target.

[0020] In an embodiment of the present application, determining the similarity between the at least one text information of the candidate page element and the user intent and / or the descriptive target comprises:

[0021] Obtaining a page element screenshot of a candidate page element in the target page;

[0022] Sending the page element screenshot to a server; the server is configured to call a multi-modal visual model to identify text information of the page element screenshot;

[0023] Receiving the text information sent by the server, and determining a third similarity between the text information and the user intent and / or the descriptive target.

[0024] In an embodiment of the present application, sending the page element screenshot to the server comprises:

[0025] When a preset condition is met, sending the page element screenshot to the server;

[0026] The preset condition at least comprises: the descriptive text information comprises a preset text information, a difference between the first similarity and the second similarity is greater than a preset similarity threshold, and / or a network connection state with the server is higher than a preset network connection state.

[0027] In an embodiment of the present application, determining the total similarity of the page element according to the similarity of the page element comprises:

[0028] obtain a text recognition scheme corresponding to the similarity of the page element; the text recognition scheme has a corresponding weight value respectively;

[0029] determine a total similarity of the page element according to the similarity and the weight value of the page element.

[0030] In an embodiment of the present application, the description text information is generated according to the user instruction, comprising:

[0031] obtain an action type corresponding to the user instruction, and take the action type as a user intention;

[0032] generate a descriptive target according to the user instruction, and take the user intention and the descriptive target as the description text information.

[0033] In a second aspect of the embodiment of the present application, a page element positioning device is further provided, and the device comprises:

[0034] a user instruction receiving module, configured to receive a user instruction of a target page; the target page comprises at least one page element;

[0035] a description text information generating module, configured to generate description text information according to the user instruction; the description text information comprises at least a user intention and a descriptive target;

[0036] a similarity determining module, configured to determine a similarity between at least one text information of the page element and the description text information; the text information is obtained by at least one text recognition scheme;

[0037] a total similarity determining module, configured to determine a total similarity of the page element according to the similarity of the page element;

[0038] a target page element positioning module, configured to position a target page element from the page element in the target page according to the total similarity.

[0039] In still another aspect of the embodiment of the present application, a computer readable storage medium is further provided, and the computer readable storage medium stores instructions, when the instructions are run on a computer, the computer executes the page element positioning method of any one of the above.

[0040] In still another aspect of the embodiment of the present application, a computer program product comprising instructions is further provided, when the instructions are run on a computer, the computer executes the page element positioning method of any one of the above.

[0041] Compared with the related art, the embodiment of the present application has at least the following advantages:

[0042] In the embodiment of the present application, a user instruction of a target page is received, the target page can include at least one page element, description text information is generated according to the user instruction, wherein the description text information can include at least user intention and descriptive target, then the similarity between at least one text information of the page element and the description text information is determined, the text information is obtained by at least one text recognition scheme, the total similarity of the page element is determined according to the similarity of the page element, and the target page element is located from the page element in the target page according to the total similarity. The embodiment of the present application can determine the total similarity based on the similarity between at least one text information of the page element and the description text information of the user instruction, and then locate the target page element from the page element in the target page based on the total similarity. Since the page element is not located in the target page based on a single similarity, the robustness and accuracy of the page element positioning are improved. BRIEF DESCRIPTION OF DRAWINGS

[0043] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments or the prior art description will be briefly introduced.

[0044] Figure 1 A step flow chart of a page element positioning method provided in the embodiment of the present application;

[0045] Figure 2 A structural block diagram of a page element positioning device provided in the embodiment of the present application;

[0046] Figure 3 A structural block diagram of an electronic device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0047] The technical solutions in the embodiments of the present application will be described below with reference to the drawings in the embodiments of the present application.

[0048] REFERENCE Figure 1 A step flow chart of a page element positioning method provided in the embodiment of the present application, as shown in Figure 1 The method can specifically include the following steps:

[0049] Step 101, receiving a user instruction of a target page; the target page includes at least one page element.

[0050] Step 102, generating description text information according to the user instruction; the description text information includes at least user intention and descriptive target.

[0051] In a specific implementation, it is necessary to perform automated testing on a browser (client) to simulate user operations of a user on one or more pages of an application, so as to ensure that the pages of the application can normally implement human-computer interaction when displayed in the browser, thereby ensuring the user experience of the user using the pages of the browser. One or more DOM elements, i.e., page elements, can be included in the page, such as a login button, a search button, an input box, or a picture in the page, etc.

[0052] In the embodiment of the application, when performing automated testing on the browser, a specific intention list can be given, the specific intention list can include one or more user instructions for generating a target page in the browser, such as “click the search button on the home page”, “click the login button on the home page”, “click the round button on the detail page”……, and then the corresponding page element is located in the browser according to the user instruction, and it is determined whether the page element correctly implements the corresponding human-computer interaction according to the user instruction, for example, when the user instruction is “click the login button on the home page”, the position of the login button on the home page is located, and it is verified whether the login button changes the state (for example, whether the color changes, becomes bright or dark, etc.), and whether it jumps to the login interface, etc. The target page can be an interface / page of an application, or multiple interfaces of an application, or even all interfaces, and the embodiment of the application does not need to be limited thereto.

[0053] In the embodiment of the application, the browser can receive one or more user instructions for a target page, and then generate description text information according to the user instruction, wherein the description text information can include at least user intention and descriptive target, wherein the user intention can include at least action type, such as “click”, “slide”, “input”, and “select” action types, and the descriptive target refers to a semantic feature used to locate the page element positioning extracted from the user instruction, for example, from the user instruction “click the login button on the home page”, the descriptive targets “login” and “home page” can be extracted.

[0054] Step 103, determining the similarity between at least one text information of the page element and the description text information; the text information is obtained by at least one text recognition scheme.

[0055] In the embodiment of the present application, different text recognition schemes can be used to determine different text information corresponding to each page element of the target page, and then the similarity between at least one text information of each page element and the description text information can be determined. For example, if there are three text recognition schemes, three text information corresponding to the page element can be obtained by using the three text recognition schemes. It can be understood that the accuracy of text information recognition is different due to different implementation principles of different text recognition schemes or different computing capabilities of devices, and the text information of the page element obtained based on different text recognition schemes is also different, and thus the similarity between the text information of the page element and the description text information is also different.

[0056] In step 104, the total similarity of the page element is determined according to the similarity of the page element.

[0057] In step 105, the target page element is located from the page element in the target page according to the total similarity.

[0058] In the embodiment of the present application, after obtaining one or more similarities corresponding to each page element, the total similarity of the page element can be calculated according to the similarity of each page element, and then the corresponding target page element can be located from the target page according to the total similarity of each page element, for example, the page element with the highest total similarity can be located as the target page element, and then it is determined whether the target page element can correctly implement the corresponding human-computer interaction according to the user instruction.

[0059] In the embodiment of the present application, the user instruction of the target page is received, and the target page can include at least one page element. The description text information is generated according to the user instruction, wherein the description text information can include at least the user intention and the description target. Then, the similarity between at least one text information of the page element and the description text information is determined, and the text information is obtained by using at least one text recognition scheme. The total similarity of the page element is determined according to the similarity of the page element, so as to locate the target page element from the page element in the target page according to the total similarity. In the embodiment of the present application, the similarity between at least one text information of the page element and the description text information of the user instruction is determined, and then the total similarity is determined based on the at least one similarity, and the target page element is located from the page element in the target page based on the total similarity. Since the page element is not located in the target page based on a single similarity, the robustness and accuracy of the page element positioning are improved.

[0060] In an embodiment of the present application, determining the similarity between at least one text information of the page element and the description target can include:

[0061] screening a candidate page element from a page element in the target page according to the user intention and / or the descriptive target;

[0062] determining a similarity between at least one text information of the candidate page element and the user intention and / or the descriptive target.

[0063] In a specific implementation, the DOM tree is a structured object model parsed by a browser from an HTML (Hyper Text Markup Language) document of a page, and represents a hierarchical relationship of all elements (such as labels, texts, attributes, etc.) in a web page in a tree structure, and is a core basis for browser automation and front-end development.

[0064] In an embodiment of the present application, the DOM tree of the target page is parsed, and the DOM tree of the target page can be optimized and reduced according to the user intention, the descriptive target, or the user intention and the descriptive target, that is, the DOM tree of the target page can be optimized and reduced according to the user intention and / or the descriptive target, and only a candidate target list conforming to the user intention and / or the descriptive target is reserved, wherein the candidate target list includes one or more candidate page elements. Then, a similarity between one or more text information of the candidate page element and the user intention, the descriptive target, or the user intention and the descriptive target, that is, a similarity between one or more text information of the candidate page element and the user intention and / or the descriptive target can be calculated.

[0065] For example, if the user intention is "clicking" and the descriptive target is "button", the DOM tree of the target page is optimized and reduced, and a page element that is a button and allows clicking in the target page can be screened as a candidate page element. In this way, when there are many page elements in the target page, only the candidate page elements in the target page can be subjected to a similarity calculation and other data processing procedures, thereby greatly reducing the data processing amount and improving the efficiency of browser automation testing.

[0066] In an embodiment of the present application, the step 102 of generating descriptive text information according to the user instruction can include:

[0067] obtaining an action type corresponding to the user instruction, and taking the action type as the user intention;

[0068] generating a descriptive target according to the user instruction, and taking the user intention and the descriptive target as the descriptive text information.

[0069] In a specific implementation, the user instruction has a corresponding action type respectively, and the action type can reflect the user intention. For example, when the user instruction is "click the blue login button in the home page", the action type corresponding to the user instruction can be "click", and "click" can be determined as the user intention. In the embodiment of the present application, the descriptive target refers to the semantic features of the user positioning page element positioning extracted from the user instruction. For example, from the user instruction "click the blue login button in the home page", the descriptive targets "blue", "login" and "home page" can be extracted, and then the user intention and the descriptive target can be used as the description text information for data processing processes such as screening and similarity calculation of the candidate page elements in the target page.

[0070] In an embodiment of the present application, determining the similarity between the at least one text information of the candidate page element and the user intention and / or the descriptive target can include:

[0071] Obtaining a page element screenshot of the candidate page element in the target page;

[0072] Identifying the text information of the page element screenshot through an optical character recognition technology, and determining a first similarity between the text information and the user intention and / or the descriptive target.

[0073] The text recognition scheme can include a scheme based on an OCR (Optical Character Recognition) technology, and the text recognition scheme is implemented through a browser.

[0074] In the embodiment of the present application, the browser can use the OCR technology to identify the page element screenshot of the candidate page element, and identify the text information of the page element screenshot through the COR, and then determine the first similarity between the text information and the user intention, the descriptive target or the user intention and the descriptive target, that is, the first similarity between the text information and the user intention and / or the descriptive target.

[0075] In an embodiment of the present application, determining the similarity between the at least one text information of the candidate page element and the user intention and / or the descriptive target can include:

[0076] Obtaining a page code of the candidate page element in the target page;

[0077] Extracting the text information corresponding to the candidate page element from the page code, and determining a second similarity between the text information and the user intention and / or the descriptive target.

[0078] The text recognition scheme is based on the page code of the target page, and this text recognition scheme is implemented through a browser.

[0079] In this embodiment of the invention, the browser can obtain the page code of the candidate page elements of the target page, extract the text information corresponding to the candidate page elements from the page code, and then determine the second similarity between the text information and the user intent, the descriptive target, or the user intent and the descriptive target, that is, the second similarity between the text information and the user intent and / or the descriptive target.

[0080] In one embodiment of the present invention, determining the similarity between at least one textual information of the candidate page element and the user intent and / or the descriptive target may include:

[0081] Obtain screenshots of the candidate page elements in the target page;

[0082] The screenshot of the page element is sent to the server; the server is used to call a multimodal visual model to identify the text information of the screenshot of the page element.

[0083] Receive the text information sent by the server and determine a third similarity between the text information and the user intent and / or the descriptive target.

[0084] The text recognition scheme is based on a multimodal visual model deployed on the server side.

[0085] In this embodiment of the invention, the browser can take a screenshot of the page element of the candidate page element and send the screenshot to the server. The server can call the modal vision model to extract the text information from the page element screenshot and send the text information to the browser. The browser can then determine the third similarity between the text information and the user intent, the descriptive target, or the user intent and the descriptive target, that is, the third similarity between the text information and the user intent and / or the descriptive target.

[0086] In one embodiment of the present invention, sending the screenshot of the page element to the server includes:

[0087] When the preset conditions are met, a screenshot of the page element is sent to the server;

[0088] The preset conditions include at least: the descriptive text information includes preset text information, the difference between the first similarity and the second similarity is greater than a preset similarity threshold, and / or the network connection status with the server is higher than a preset network connection status.

[0089] In actual application, the computing capability of the server is far higher than that of the browser, and therefore, if the precision requirement is high, the page element screenshot of the page element can be sent to the server to calculate the third similarity by the multimodal visual model of the server, and then the total similarity can be calculated according to the first similarity, the second similarity and the third similarity, so as to improve the positioning accuracy of the target page element of the target page. However, due to the interaction between the browser and the server, the network connection state between the browser and the server, the server cost and other factors, if the third similarity is calculated by the multimodal visual model of the server in all cases, the cost and the overall test time will be increased. Therefore, in the embodiment of the application, the page element screenshot can be sent to the server to calculate the third similarity by the multimodal visual model of the server only when the preset condition is met.

[0090] In an embodiment of the application, the description text information includes preset text information, and the preset text information reflects that the page element screenshot of the page element needs strong computing capability for recognition. For example, the preset text information can be visual text information such as “blue”, and for another example, the preset text information can be shape information such as “circle”, which indicates that the color information or the contour information in the page element screenshot of the page element needs to be recognized. Therefore, it can be determined that the page element screenshot needs strong computing capability for recognition, and then the page element screenshot of the page element can be sent to the server to calculate the third similarity by the multimodal visual model of the server.

[0091] In another embodiment of the application, the difference between the first similarity and the second similarity is greater than the preset similarity threshold, which indicates that the similarity results calculated by the text recognition scheme based on the OCR technology and the text recognition scheme based on the page code of the target page are quite different, and it is indicated that the text information recognized by one of the text recognition schemes is not reliable / mismatched. Therefore, the page element screenshot of the page element can be sent to the server to calculate the third similarity by the multimodal visual model of the server, and then the target page element in the target page can be positioned by combining the similarities of various text recognition schemes including the text recognition scheme implemented based on the server, so as to ensure the accuracy of the positioning of the page element.

[0092] In another embodiment of the application, the page element screenshot of the page element is sent to the server to calculate the third similarity by the multimodal visual model of the server only when the network connection state between the browser and the server is higher than the preset network connection state. In this way, the multimodal visual model of the server can be avoided to be called when the network connection state between the browser and the server is poor, and the problem of the increase of the overall test time can be avoided.

[0093] In an embodiment of the present application, determining the total similarity of the page element according to the similarity of the page element can comprise:

[0094] Obtaining a text recognition scheme corresponding to the similarity of the page element; the text recognition scheme has a corresponding weight value respectively;

[0095] Determining the total similarity of the page element according to the similarity and the weight value of the page element.

[0096] In an embodiment of the present application, different weight values can be given to the similarities obtained by different text recognition schemes, so that the total similarity of the page element can be obtained according to the similarity and the weight value of the page element.

[0097] In an embodiment, assuming that there are three text recognition schemes, the calculation formula of the total similarity is as follows:

[0098] s=a*l+b*m+c*n;

[0099] a+b+c=1

[0100] Where s represents the total similarity; l, m and n represent the similarities calculated based on the text information corresponding to the three text recognition schemes respectively; a, b and c represent the weight values corresponding to the three text recognition schemes respectively.

[0101] In order for those skilled in the art to better understand the embodiments of the present application, the following will be described with a specific example. The method for matching DOM elements in a target page based on action type and descriptive target in an embodiment of the present application can be as follows:

[0102] 1. Analyzing the DOM tree of the target page, optimizing and reducing the DOM tree according to the descriptive text information, and only retaining the candidate target list that meets the requirements; wherein the descriptive text information can include user intent (action type) and descriptive target;

[0103] 2. Using client-side OCR to recognize the text information in the candidate page element in the candidate target list, and calculating the similarity l with the descriptive text information;

[0104] 4. Directly calculating the similarity m of the text information contained in the page code of the candidate target list and the descriptive text information;

[0105] 5. Using the multi-modal visual model of the server to recognize the similarity n between the candidate target list and the descriptive text information;

[0106] 6. Calculating the total similarity s=a*l+b*m+c*n, a+b+c=1.

[0107] In the above embodiments, the context of the target page is compressed by pruning, the final total similarity is calculated using weighting, the robustness of page element recognition in the target page is improved, and in the case of text mismatch, the user instruction is better understood using the multi-modal visual model of the server, and the positioning accuracy of the target page element in the browser based on the user instruction is improved.

[0108] It should be noted that, for the method embodiments, in order to simply describe, they are all described as a series of action combinations, but those skilled in the art should know that the embodiments of the present application are not limited to the order of the described actions, because according to the embodiments of the present application, certain steps can be performed in other order or at the same time. Secondly, those skilled in the art should know that the embodiments described in the specification all belong to preferred embodiments, and the actions involved are not necessarily essential for the embodiments of the present application.

[0109] Reference Figure 2 The structural block diagram of a page element positioning device provided in the embodiments of the present application is shown in Figure 2 The device can specifically include the following modules:

[0110] The user instruction receiving module 201 is configured to receive a user instruction of a target page; the target page includes at least one page element;

[0111] The description text information generating module 202 is configured to generate description text information according to the user instruction; the description text information includes at least a user intent and a descriptive target;

[0112] The similarity determining module 203 is configured to determine the similarity between at least one text information of the page element and the description text information; the text information is obtained by at least one text recognition scheme;

[0113] The total similarity determining module 204 is configured to determine the total similarity of the page element according to the similarity of the page element;

[0114] The target page element positioning module 205 is configured to position a target page element from the page elements in the target page according to the total similarity.

[0115] In an embodiment of the present application, the similarity determining module 203 is configured to:

[0116] Filter a candidate page element from the page elements in the target page according to the user intent and / or the descriptive target;

[0117] Determine the similarity between at least one text information of the candidate page element and the user intent and / or the descriptive target.

[0118] In an embodiment of the present application, the similarity determination module 203 is configured to:

[0119] obtain a page element screenshot of a candidate page element in the target page;

[0120] identify text information of the page element screenshot by using an optical character recognition technology, and determine a first similarity between the text information and the user intent and / or the descriptive target.

[0121] In an embodiment of the present application, the similarity determination module 203 is configured to:

[0122] obtain a page code of a candidate page element in the target page;

[0123] extract text information corresponding to the candidate page element from the page code, and determine a second similarity between the text information and the user intent and / or the descriptive target.

[0124] In an embodiment of the present application, the similarity determination module 203 is configured to:

[0125] obtain a page element screenshot of a candidate page element in the target page;

[0126] send the page element screenshot to a server; the server is configured to call a multi-modal visual model to identify text information of the page element screenshot;

[0127] receive the text information sent by the server, and determine a third similarity between the text information and the user intent and / or the descriptive target.

[0128] In an embodiment of the present application, the similarity determination module 203 is further configured to:

[0129] when a preset condition is met, send the page element screenshot to a server;

[0130] the preset condition at least includes that the descriptive text information includes a preset text information, a difference between the first similarity and the second similarity is greater than a preset similarity threshold, and / or a network connection state with the server is higher than a preset network connection state.

[0131] In an embodiment of the present application, the total similarity determination module 204 is configured to:

[0132] obtain a text recognition scheme corresponding to the similarity of the page element; the text recognition scheme has a corresponding weight value respectively;

[0133] The total similarity of the page elements is determined based on their similarity and weight values.

[0134] In one embodiment of the present invention, generating descriptive text information according to the user instruction includes:

[0135] Obtain the action type corresponding to the user instruction, and use the action type as the user intent;

[0136] A descriptive target is generated based on the user instruction, and the user intent and the descriptive target are used as descriptive text information.

[0137] In this embodiment of the invention, a user instruction for a target page is received. The target page may include at least one page element. Descriptive text information is generated according to the user instruction, wherein the descriptive text information may at least include user intent and descriptive target. Then, the similarity between at least one text information of the page element and the descriptive text information is determined. The text information is obtained through at least one text recognition scheme. The total similarity of the page elements is determined based on the similarity of the page elements, so as to locate the target page element from the page elements in the target page based on the total similarity. This embodiment of the invention can determine the total similarity based on at least one similarity between the text information of a page element and the descriptive text information of the user instruction, and then locate the target page element from the page elements in the target page based on the total similarity. Since the page element location is not based on a single similarity, the robustness and accuracy of page element location are improved.

[0138] The above-described apparatus embodiments are basically similar to the method embodiments, so they are described in a relatively simple manner. For relevant details, please refer to the description of the method embodiments.

[0139] It should be noted that the embodiments of the present invention may involve the use of user data. In practical applications, user-specific personal data may be used in the scheme described herein within the scope permitted by applicable laws and regulations, provided that it complies with the applicable laws and regulations of the country (e.g., with the user's explicit consent, with the user being properly notified, etc.).

[0140] This invention also provides an electronic device, such as... Figure 3 As shown, it includes a processor 301, a communication interface 302, a memory 303, and a communication bus 304, wherein the processor 301, the communication interface 302, and the memory 303 communicate with each other through the communication bus 304.

[0141] Memory 303 is used to store computer programs;

[0142] The processor 301 is configured to implement the page element positioning method according to any one of the above embodiments when executing the program stored in the memory 303.

[0143] receive a user instruction of a target page, wherein the target page comprises at least one page element;

[0144] generate description text information according to the user instruction, wherein the description text information comprises at least user intention and descriptive target;

[0145] determine similarity between at least one text information of the page element and the description text information, wherein the text information is obtained by at least one text recognition scheme;

[0146] determine total similarity of the page element according to the similarity of the page element;

[0147] position a target page element from the page element in the target page according to the total similarity.

[0148] The communication bus mentioned above can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For the convenience of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or only one type of bus.

[0149] The communication interface is configured to communicate between the terminal and other devices.

[0150] The memory can include a Random Access Memory (RAM) and can also include a non-volatile memory, such as at least one disk memory. Optionally, the memory can also be at least one storage device located away from the processor.

[0151] The processor described above can be a general processor, including a central processing unit (CPU), a network processor (NP), etc.; or can be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component.

[0152] In yet another embodiment provided by the present application, a computer readable storage medium is provided, which stores instructions, when executed on a computer, cause the computer to perform the page element positioning method according to any one of the above embodiments.

[0153] In yet another embodiment provided by the present application, a computer program product is provided, which contains instructions, when executed on a computer, cause the computer to perform the page element positioning method according to any one of the above embodiments.

[0154] In the above embodiments, the implementation can be achieved entirely or partially by software, hardware, firmware or any combination thereof. When implemented by software, the implementation can be achieved entirely or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the flow or function described in the embodiments of the present application is entirely or partially generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network or other programmable device. The computer instructions can be stored in a computer readable storage medium or transferred from one computer readable storage medium to another, for example, the computer instructions can be transferred from one website, computer, server or data center to another website, computer, server or data center through wired (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (such as infrared, wireless, microwave, etc.) mode. The computer readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server, data center, etc. integrated with one or more available media. The available media can be a magnetic medium (for example, floppy disk, hard disk, magnetic tape), an optical medium (for example, DVD), or a semiconductor medium (for example, solid state disk (SSD)) and the like.

[0155] It is to be understood that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting; it is not intended to exclude myriad other embodiments of the present application that other inventors can develop based on the same general inventive concepts embodied by the described embodiments. That is, although the present application is described in terms of particular embodiments and implementations, it is to be understood that the terminology used is for the purpose of descriptive clarity and that it should be taken in its broadest possible sense. For example, the terms "process," "method," "procedure," and techniques can include a sequence of steps, which can be wholly or partially performed automatically (e.g., as in an automated system) or performed on a computer or machine (e.g., as in a software application or a firmware application). Similarly, the term "article of manufacture" as used herein is intended to encompass a computer program accessible from a computer-readable medium directing a computer or machine to create the processes, methods, procedures, or techniques described. Alternatively, the terms "process," "method," "procedure," and techniques can include a single step, a collection of steps, or a number of steps performed in any suitable order. Further, the terms "first," "second," and the like can be used to describe various elements in the above-described embodiments, but such elements should not be limited by these terms. Such terms are only used to distinguish one element from another. The terms "comprising," "including," and the like are to be read expansively and without limitation. The terms "process," "method," "procedure," and techniques can include any steps or processes that can be performed automatically or on a computer or machine (e.g., as in a software application or a firmware application). The terms "process," "method," "procedure," and techniques can also include a single step, a collection of steps, or a number of steps performed in any suitable order.

[0156] Each of the embodiments described in the specification are presented in a related manner, and the same or similar parts among the embodiments can be referred to each other. Each of the embodiments focuses on the difference from other embodiments. In particular, the system embodiments are described simply because they are substantially similar to the method embodiments, and the related parts can be referred to the description of the method embodiments.

[0157] The above description is merely illustrative of the application and is not intended to limit the scope of the application. Any modification, equivalent replacement, improvement, and the like made within the spirit and principle of the application are included in the scope of the application.

Claims

1. A method for locating page elements, characterized in that, The method includes: Receive user instructions for a target page; the target page includes at least one page element; Descriptive text information is generated based on the user instructions; the descriptive text information includes at least the user intent and the descriptive objective; Determine the similarity between at least one textual information of the page element and the descriptive textual information; the textual information is obtained through at least one text recognition scheme; Determining the total similarity of the page elements based on their similarity includes: Obtain the text recognition scheme corresponding to the similarity of the page elements; each text recognition scheme has a corresponding weight value; determine the total similarity of the page elements based on the similarity and weight values ​​of the page elements; Based on the total similarity, the target page element is located from the page element in the target page; Determining the similarity between at least one textual piece of information of the page element and the descriptive textual information includes: Candidate page elements are selected from the page elements in the target page based on the user intent and / or the descriptive objective. Determine the similarity between at least one textual piece of information of the candidate page element and the user intent and / or the descriptive target; Wherein, determining the similarity between at least one textual information of the candidate page element and the user intent and / or the descriptive target includes: Obtain screenshots of the candidate page elements in the target page; The text information of the screenshot of the page element is identified by optical character recognition technology, and a first similarity between the text information and the user intent and / or the descriptive target is determined. as well as Obtain the page code of the candidate page elements in the target page; Extract the text information corresponding to the candidate page element from the page code, and determine the second similarity between the text information and the user intent and / or the descriptive target.

2. The method according to claim 1, characterized in that, Determining the similarity between at least one textual piece of information of the candidate page element and the user intent and / or the descriptive target includes: Obtain screenshots of the candidate page elements in the target page; The screenshot of the page element is sent to the server; the server is used to call a multimodal visual model to identify the text information of the screenshot of the page element. Receive the text information sent by the server and determine a third similarity between the text information and the user intent and / or the descriptive target.

3. The method according to claim 2, characterized in that, Sending screenshots of the page elements to the server includes: When the preset conditions are met, a screenshot of the page element is sent to the server; The preset conditions include at least: the descriptive text information includes preset text information, the difference between the first similarity and the second similarity is greater than a preset similarity threshold, and / or the network connection status with the server is higher than a preset network connection status.

4. The method according to claim 1, characterized in that, Generate descriptive text information according to the user instructions, including: Obtain the action type corresponding to the user instruction, and use the action type as the user intent; A descriptive target is generated based on the user instruction, and the user intent and the descriptive target are used as descriptive text information.

5. A page element positioning device, characterized in that, The device includes: The user instruction receiving module is used to receive user instructions for the target page; the target page includes at least one page element. A descriptive text information generation module is used to generate descriptive text information based on the user instructions; the descriptive text information includes at least the user intent and the descriptive objective; A similarity determination module is used to determine the similarity between at least one text information of the page element and the descriptive text information; the text information is obtained through at least one text recognition scheme; The total similarity determination module is used to determine the total similarity of the page elements based on the similarity of the page elements, including: obtaining the text recognition scheme corresponding to the similarity of the page elements; the text recognition scheme has a corresponding weight value; and determining the total similarity of the page elements based on the similarity and weight value of the page elements. The target page element localization module is used to locate the target page element from the page elements in the target page based on the total similarity. The similarity determination module is further configured to filter candidate page elements from page elements in the target page based on the user intent and / or the descriptive target; and determine the similarity between at least one textual information of the candidate page element and the user intent and / or the descriptive target; Wherein, determining the similarity between at least one textual information of the candidate page element and the user intent and / or the descriptive target includes: Obtain screenshots of candidate page elements in the target page; identify the text information of the page element screenshots using optical character recognition technology, and determine the first similarity between the text information and the user intent and / or the descriptive target; as well as Obtain the page code of the candidate page element in the target page; extract the text information corresponding to the candidate page element from the page code, and determine the second similarity between the text information and the user intent and / or the descriptive target.

6. An electronic device, characterized in that, It includes a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; Memory, used to store computer programs; A processor, when executing a program stored in memory, implements the method of any one of claims 1-4.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1-4.

Images