Element positioning method and device based on large model, electronic equipment and storage medium

Abstract

The invention discloses an element positioning method and device based on a large model, electronic equipment and a storage medium, and relates to the technical field of large models. The method comprises the steps of obtaining a target tree structure corresponding to a target element in response to a positioning description instruction of the target element; the target element is a target child node of the target tree structure; traversing the target tree structure step by step from a root node of the target tree structure to the target child node, and extracting node features of each node of the target tree structure; determining a target feature of the target element according to the node feature of each node; and generating a large model based on the target feature and a path selector obtained by pre-training, obtaining a selector statement matched with the target element, and positioning the target element based on the selector statement. According to the scheme provided by the embodiment of the invention, the elements can be accurately and quickly positioned, and a basis is provided for subsequent specific business operation.

Description

Technical Field

[0001] The present invention relates to the field of large model technology, and in particular to an element positioning method, device, electronic equipment and storage medium based on a large model. Background Art

[0002] RPA (Robotic Process Automation) is software that runs on personal computers or servers. Its main function is to simulate human control of various system software and automatically execute business processes. It is suitable for large-scale, highly repetitive business scenarios. By accurately positioning each element in RPA (for example, interface elements), the execution efficiency of RPA can be improved and comprehensive automation can be achieved.

[0003] With more and more complex elements, how to locate them accurately and quickly to provide a basis for subsequent specific business operations is a major research issue in the industry. Summary of the invention

[0004] The present invention provides a large-model-based element positioning method, device, electronic device and storage medium to accurately and quickly position elements and provide a basis for subsequent specific business operations.

[0005] According to one aspect of the present invention, a method for locating an element based on a large model is provided, the method comprising:

[0006] In response to a location description instruction of a target element, a target tree structure corresponding to the target element is acquired; the target element is a target child node of the target tree structure;

[0007] Traversing the target tree structure from the root node of the target tree structure to the target child node level by level, and extracting node features of each node of the target tree structure;

[0008] Determining the target feature of the target element according to the node feature of each of the nodes;

[0009] A large model is generated based on the target features and a pre-trained path selector to obtain a selector statement matching the target element, and the target element is located based on the selector statement.

[0010] According to another aspect of the present invention, there is provided an element positioning device based on a large model, the device comprising:

[0011] A target tree structure acquisition module, configured to acquire a target tree structure corresponding to the target element in response to a location description instruction of the target element; the target element is a target child node of the target tree structure;

[0012] A node feature extraction module, used to traverse the target tree structure from the root node of the target tree structure to the target child node, and extract node features of each node of the target tree structure;

[0013] A target feature determination module, used to determine the target feature of the target element according to the node feature of each node;

[0014] A positioning module is used to generate a large model based on the target features and a pre-trained path selector, obtain a selector statement matching the target element, and locate the target element based on the selector statement.

[0015] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:

[0016] at least one processor; and

[0017] a memory communicatively connected to the at least one processor; wherein,

[0018] The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the element positioning method based on a large model described in any embodiment of the present invention.

[0019] According to another aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the large model-based element positioning method described in any embodiment of the present invention when executed.

[0020] According to another aspect of the present invention, a computer program product is provided, comprising a computer program, wherein when the computer program is executed by a processor, the method for locating an element based on a large model according to any embodiment of the present invention is implemented.

[0021] The technical solution of the embodiment of the present invention obtains a target tree structure corresponding to the target element by responding to a positioning description instruction of the target element; the target element is a target child node of the target tree structure; the target tree structure is traversed level by level from the root node of the target tree structure to the target child node, and the node features of each node of the target tree structure are extracted; the target features of the target element are determined according to the node features of each node; a large model is generated based on the target features and a pre-trained path selector to obtain a selector statement matching the target element, and the target element is positioned based on the selector statement, so that the element can be accurately and quickly positioned, providing a basis for subsequent specific business operations.

[0022] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0024] Figure 1 is a flow chart of an element positioning method based on a large model provided according to the first embodiment of the present invention;

[0025] Figure 2 is a flow chart of an element positioning method based on a large model provided according to the second embodiment of the present invention;

[0026] Figure 3 is a structural schematic diagram of an element positioning device based on a large model provided according to Embodiment 3 of the present invention;

[0027] Figure 4 It is a structural schematic diagram of an electronic device for implementing the large model-based element positioning method of an embodiment of the present invention. DETAILED DESCRIPTION

[0028] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0029] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0030] Embodiment 1

[0031] Figure 1 This is a flowchart of a method for locating an element based on a large model according to the first embodiment of the present invention. This embodiment is applicable to the situation of quickly and accurately locating an element. The method can be executed by an element locating device based on a large model. The element locating device based on a large model can be implemented in the form of hardware and / or software. The element locating device based on a large model can be configured in an electronic device such as a computer, a server, a cloud server or a tablet computer. Specifically, refer to Figure 1 , the method comprising:

[0032] Step 110: In response to a location description instruction of a target element, a target tree structure corresponding to the target element is acquired; the target element is a target child node of the target tree structure.

[0033] Among them, the target element can be an interface element involved in RPA, such as a button, a label, a toolbar or a status bar; it can also be an element in an HTML (HyperText Markup Language) document, such as a block-level element, an inline element or a form element, etc., which is not limited in this embodiment.

[0034] In an optional implementation of this embodiment, when a user clicks or triggers a target element, a positioning description instruction of the target element is automatically generated and sent to a corresponding processor; after receiving the positioning description instruction for the target element, the processor can further obtain a target tree structure corresponding to the target element; wherein the target element can be a child node of the target tree structure, also known as a leaf node; the root node of the target tree structure is fixed and can be determined by the operating system.

[0035] It can be understood that the target tree structure is composed of a root node and multiple leaf nodes, and each leaf node is arranged at a different level of the target tree structure; each level may contain one or more leaf nodes.

[0036] Optionally, in this embodiment, after receiving the location description instruction for the target element, the target tree structure can be further obtained from the system file of the system to which the target element belongs; in this embodiment, the target tree structure can be a DOM (Document Object Model) tree structure.

[0037] Step 120: traverse the target tree structure from the root node of the target tree structure to the target child node, and extract node features of each node of the target tree structure.

[0038] In an optional implementation of the present embodiment, after obtaining the target tree structure corresponding to the target element, the target tree structure can be further traversed level by level starting from the root node of the target tree structure until the target child node of the target tree structure is traversed, that is, the target element is traversed; during the traversal process, the node features of the root node and each child node can be obtained.

[0039] In this embodiment, the node characteristics of each node may include the label, attribute, content or position of each node in the target tree structure, etc., which are not limited in this embodiment.

[0040] Step 130: Determine the target feature of the target element according to the node feature of each of the nodes.

[0041] In an optional implementation of the present embodiment, after extracting the node features of each node in the target tree structure, the target features of the target element can be further determined based on the features of each node; optionally, the extracted node features can be classified according to the feature type, and the node features of the same type can be combined to obtain the target features of the target element.

[0042] In an example of this embodiment, the label features, attribute features and content features of each node can be combined respectively to obtain different types of features; further, different types of features can be determined as target features of the target element.

[0043] Step 140: Generate a large model based on the target features and the pre-trained path selector, obtain a selector statement matching the target element, and locate the target element based on the selector statement.

[0044] In an optional implementation of this embodiment, after determining the target features of the target element, a large model can be further generated based on the target features of the target element and a pre-trained path selector to obtain a selector statement that matches the target element; further, the target element can be located based on the selector statement.

[0045] Optionally, in this embodiment, obtaining a selector statement matching the target element based on the target feature and a pre-trained path selector generation model may include: inputting the target feature into the pre-trained path selector generation model, and outputting a selector statement matching the target element.

[0046] In an optional implementation of the present embodiment, after determining the target feature of the target element, the target element feature can be further input into a pre-trained path selector generation model, and the target feature is processed by the path selector generation model to obtain a selector statement that matches the target element; the selector statement can be used to locate the target element in the operating system to which the target element belongs; this can improve the processing efficiency of subsequent business.

[0047] It should be noted that the selector statement generated in this embodiment can be directly applied in an automated test script to ensure that the test case can accurately locate and operate the target element.

[0048] The technical solution of this embodiment obtains a target tree structure corresponding to the target element by responding to a positioning description instruction of the target element; the target element is a target child node of the target tree structure; the target tree structure is traversed level by level from the root node of the target tree structure to the target child node, and the node features of each node of the target tree structure are extracted; the target features of the target element are determined according to the node features of each node; a large model is generated based on the target features and a pre-trained path selector to obtain a selector statement matching the target element, and the target element is located based on the selector statement, so that the element can be accurately and quickly located, providing a basis for subsequent specific business operations.

[0049] Based on the above technical solution, the path selector generation large model can be trained through the following steps: obtaining a reference element data set; determining the data features of each element in the reference element data set, and the selector statements matching the data features; inputting the data features of each element, and the selector statements matching the data features into the pre-trained large language model for fine-tuning to obtain the path selector generation large model.

[0050] Optionally, in this embodiment, a large number of interface elements may be obtained, for example, ten thousand, one hundred thousand or one million, and these interface elements may be added to a target database or folder to obtain a reference element data set.

[0051] Furthermore, the data features of each element in the reference element data set and the selector statements matching the data features can be determined; in an optional implementation of this embodiment, each element can be annotated by a predetermined annotation tool to obtain the data features of each element and the selector statements matching the data features. In another optional implementation of this embodiment, each element can also be annotated by manual annotation to obtain the data features of each element and the selector statements matching the data features.

[0052] In an optional implementation of this embodiment, determining the data feature of each element in the reference element data set may include: extracting the data feature of each element in the reference element data set by a feature extraction module.

[0053] In an optional implementation of the present embodiment, the feature extraction module can extract the data features of each element through the following steps: starting from the root node of the target tree structure, extracting the node features of each node in the target tree structure step by step; combining each of the node features to obtain at least one path feature, and determining each of the path features as the data features of each of the elements; the node features include at least one of the following: label, attribute, content, and position in the target tree structure; the path features include at least one of the following: label sequence, attribute combination, and text content combination.

[0054] Optionally, in this embodiment, after obtaining the reference element data set, the node features of each node in the target tree structure can be extracted step by step starting from the root node of the target tree structure, for example, the label, attributes, content, and position of each node in the target tree structure can be extracted; further, the acquired node features can be combined to obtain at least one path feature, and each path feature can be determined as the data feature of each element.

[0055] In a specific implementation, the acquired label features can be combined to obtain a label sequence, which is a path feature; the acquired attribute features can be combined to obtain an attribute combination, which is a path feature; the acquired content features can be combined to obtain a text content combination, which is a path feature; further, these path features can be respectively determined as data features of each element in the reference element data set.

[0056] Furthermore, the data features of an element and the selector statement that matches it can be determined as a piece of training data, and these data features are input into the large language model for fine-tuning, so as to obtain a path selector generation large model.

[0057] The advantage of this setting is that a path selector can be generated based on the data features of each element in the reference element data set and the selector statements matching each data feature to generate a large model, providing a basis for the subsequent positioning of each interface element.

[0058] In another optional implementation of this embodiment, the path selector generates a large model that can generate a unique identifier or path for each interface element in the RPA. These identifiers can be used by the RPA to perform corresponding GUI (Graphical User Interface) operations, which can ensure the accuracy and stability of the operations.

[0059] Embodiment 2

[0060] Figure 2 1 is a flowchart of a method for locating an element based on a large model according to Embodiment 2 of the present invention. This embodiment is a further refinement of the above technical solution. The technical solution in this embodiment can be combined with each optional solution in one or more of the above embodiments. Figure 2 As shown, the method includes:

[0061] Step 210: In response to the location description instruction of the target element, obtain a target tree structure corresponding to the target element; the target element is a target child node of the target tree structure.

[0062] In this embodiment, the target tree structure may be a DOM tree structure.

[0063] Step 220 : Starting from the root node of the DOM tree structure, extract node features of each node in the DOM tree structure in sequence until the target child node.

[0064] Each node of the DOM tree structure includes a root node and child nodes; a node feature includes at least one of the following: a tag, an attribute, a content, and a position in the DOM tree structure.

[0065] Optionally, in this embodiment, when the acquired target tree structure corresponding to the target element is a DOM tree structure, node features of each node in the DOM tree structure may be extracted in sequence starting from the root node of the DOM tree structure until the target child node.

[0066] Step 230: Combine the node features to obtain at least one path feature; and determine the path features as target features of the target element.

[0067] The path feature includes at least one of the following: a tag sequence, an attribute combination, and a text content combination.

[0068] Optionally, after traversing and obtaining the features of each node, the features of each node can be further combined. For example, different node features can be combined according to feature type, and the combined feature combination can be determined as a path feature; illustratively, the obtained label features can be combined to obtain a label sequence, which is the one-day path feature of the target element; the obtained attribute features can be combined to obtain an attribute combination, which is a path feature of the target element; the obtained content features can be combined to obtain a text content combination, which is a path feature of the target element; further, these path features can be determined as target features of the target element.

[0069] Step 240: Input the target feature into the pre-trained path selector generation model, and output a selector statement matching the target element.

[0070] In an optional implementation of this embodiment, after obtaining the target feature of the target element, the target feature can be further input into the pre-trained path selector generation model, and the path selector generation model outputs a selector statement that matches the target element.

[0071] Step 250: Locate the target element based on the selector statement that matches the target element.

[0072] In this embodiment, the selector statement matching the target element may record the path information for finding the target element.

[0073] In an optional implementation of this embodiment, after obtaining a selector statement matching the target element, the target element may be located according to the path information of the target element recorded in the selector statement.

[0074] In a specific example of this embodiment, the element positioning method based on the large model may include the following content:

[0075] 1. Initial data collection and annotation:

[0076] In this embodiment, a large number of interface elements can be captured from various application interfaces, and the attributes of these elements and their corresponding selector statements can be manually annotated to build a diverse dataset to cover different types of interfaces and elements.

[0077] 2. Fine-tuning of large models:

[0078] In this embodiment, the collected annotated data can be used to fine-tune the pre-trained large language model. Through supervised learning, the model can generate corresponding selector statements based on the input element features.

[0079] 3. Feature extraction module:

[0080] In this embodiment, the function of the feature extraction module is to extract various features that are helpful for generating selectors from the elements captured by the user interface element capturer; the feature extraction module is used to extract the following features from the HTML elements on the web page or the interface element objects in the desktop application according to the multi-tree path finding algorithm: tag name: the HTML tag name of the element, such as div, span or input; id: the unique identifier of the element; class name: the class attribute of the element, which can contain multiple class names; text content: the text content of the element; element position: the depth and relative position of the element in the desktop hierarchical tree, as well as the position relative to the parent node, etc.; size: the width and height of the element; other attributes of the element, such as name, type or placeholder, etc.

[0081] In this embodiment, after acquiring various attributes and feature values ​​of an element, the feature extraction module extracts information useful for generating a selector by parsing these attribute values, and outputs a dictionary object containing element features.

[0082] Furthermore, the feature extraction module inputs the extracted features into the fine-tuned large language model, and the model generates the corresponding selector statements.

[0083] 4. Multi-tree path finding algorithm:

[0084] 1. DOM tree structure traversal

[0085] In this embodiment, the DOM tree is traversed level by level starting from the root node; for each node, the algorithm will consider features such as its tag name, attributes, text content, and position in the DOM tree.

[0086] 2. Node feature extraction:

[0087] In this embodiment, for each node, the algorithm will extract multiple features: for example, tag name and attributes, text content, such as the text content contained in the node, the text of a button or the text of a link, etc.; node position, such as the hierarchical depth and relative position of the node in the DOM tree, and its index position in the parent node.

[0088] 3. Multi-branch tree path generation:

[0089] In this embodiment, the position of a node in a DOM tree can be represented by combining the features of the node into a path. The path can be a combination of a node's tag sequence, attribute combination, text content, etc. These paths form a multi-branch tree structure, in which each path represents a specific element positioning feature.

[0090] The solution of this embodiment, by capturing a large number of interface elements of different application interfaces and manually annotating them, constructs a diverse data set, ensuring that the model can cover various types of interfaces and elements during the training process, and improving the model's adaptability and generalization capabilities to different scenarios; based on various key features of interface elements, the user interface element capture selector statement is generated through a large language model, which can significantly improve the accuracy and efficiency of element capture in the RPA process; the multi-tree path finding algorithm is used to traverse the DOM tree structure, which can fully consider the node's label name, attributes, text content, and location. The algorithm generates multiple paths by combining these features, representing different element positioning features, so as to more accurately locate and select interface elements, which can effectively solve the complexity and diversity of element positioning in the DOM tree structure.

[0091] In an optional implementation of this embodiment, the element positioning method based on the large model involved in this embodiment can realize the generation of a selector statement for a web page element, which may specifically include:

[0092] 1. Capture the DOM structure of a large number of different pages, and manually annotate the attributes of each interface element and its corresponding XPath or CSS Selector.

[0093] 2. Fine-tune the collected annotated data using the pre-trained language model.

[0094] 3. Extract features such as tag name, ID, class name, text content, etc. from each captured interface element through the feature extraction module, and input these features into the fine-tuned language model.

[0095] 4. The fine-tuned language model generates the XPath or CSS Selector corresponding to each interface element based on the data provided by the feature extraction module; these Selectors can be directly used in automated test scripts to ensure that test cases can accurately locate and operate target elements.

[0096] In another optional implementation of this embodiment, the element positioning method based on the large model involved in this embodiment can realize the generation of a selector statement for a desktop application element, which may specifically include:

[0097] 1. Develop an interface element capturer for desktop applications that can capture element objects of each interface and annotate their properties.

[0098] 2. Use the pre-trained language model to fine-tune the captured interface element data.

[0099] 3. Extract feature information such as class name, ID, location, etc. from the captured interface elements through the feature extraction module, and input these features into the fine-tuned language model.

[0100] 4. The fine-tuned language model generates a unique identifier or path for each interface element based on the data provided by the feature extraction module; these identifiers or paths are used by the automation tool to perform corresponding GUI operations to ensure the accuracy and stability of the operations.

[0101] Embodiment 3

[0102] Figure 3 Schematic diagram of a large model-based element positioning device according to Embodiment 3 of the present invention. Figure 3 As shown, the device includes: a target tree structure acquisition module 310, a node feature extraction module 320, a target feature determination module 330 and a positioning module 340.

[0103] The target tree structure acquisition module 310 is used to acquire a target tree structure corresponding to the target element in response to a location description instruction of the target element; the target element is a target child node of the target tree structure;

[0104] A node feature extraction module 320 is used to traverse the target tree structure from the root node of the target tree structure to the target child node, and extract node features of each node of the target tree structure;

[0105] A target feature determination module 330, configured to determine the target feature of the target element according to the node feature of each of the nodes;

[0106] The positioning module 340 is used to generate a large model based on the target features and the pre-trained path selector, obtain a selector statement matching the target element, and locate the target element based on the selector statement.

[0107] The scheme of this embodiment is to obtain a target tree structure corresponding to the target element in response to a positioning description instruction of the target element through a target tree structure acquisition module; the target element is a target child node of the target tree structure; the target tree structure is traversed level by level from the root node of the target tree structure to the target child node through a node feature extraction module to extract node features of each node of the target tree structure; the target feature of the target element is determined according to the node features of each node through a target feature determination module; a large model is generated based on the target feature and a pre-trained path selector through a positioning module to obtain a selector statement matching the target element, and the target element is positioned based on the selector statement, so that the element can be positioned accurately and quickly, providing a basis for subsequent specific business operations.

[0108] In an optional implementation of this embodiment, the element positioning device based on the large model further includes: a path selector generating a large model training module, which is used to train the path selector to generate a large model through the following steps:

[0109] Obtaining a reference element dataset;

[0110] Determine data features of each element in the reference element data set, and selector statements matching the data features;

[0111] The data features of each of the elements and the selector statements matching the data features are input into the pre-trained large language model for fine-tuning to obtain a path selector generation large model.

[0112] In an optional implementation of this embodiment, the path selector generates a large model training module, including: a feature extraction module, used to extract data features of each element in the reference element data set.

[0113] In an optional implementation of this embodiment, the feature extraction module extracts data features of each element through the following steps:

[0114] Starting from the root node of the target tree structure, node features of each node in the target tree structure are extracted step by step;

[0115] Combining the node features to obtain at least one path feature, and determining the path features as data features of the elements;

[0116] The node characteristics include at least one of the following:

[0117] Tags, attributes, content, and location in the target tree structure;

[0118] The path characteristics include at least one of the following:

[0119] Sequence of tags, combination of attributes, and combination of text content.

[0120] In an optional implementation of this embodiment, the target tree structure is a document object model DOM tree structure;

[0121] Correspondingly, the node feature extraction module 320 is specifically used to extract node features of each node in the DOM tree structure in sequence starting from the root node of the DOM tree structure until the target child node;

[0122] Each node of the DOM tree structure includes a root node and child nodes;

[0123] The node characteristics include at least one of the following:

[0124] Tags, attributes, content, and position in the DOM tree structure.

[0125] In an optional implementation of this embodiment, the target feature determination module 330 is specifically configured to combine the node features to obtain at least one path feature;

[0126] Determine each of the path features as a target feature of the target element;

[0127] The path characteristics include at least one of the following:

[0128] Sequence of tags, combination of attributes, and combination of text content.

[0129] In an optional implementation of this embodiment, the positioning module 340 is specifically used to input the target feature into the pre-trained path selector generation model, and output a selector statement matching the target element.

[0130] The large model-based element positioning device provided in the embodiment of the present invention can execute the large model-based element positioning method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0131] Embodiment 4

[0132] Figure 4A schematic diagram of the structure of an electronic device 10 that can be used to implement an embodiment of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0133] like Figure 4 As shown, the electronic device 10 includes at least one processor 11, and a memory connected to the at least one processor 11, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 to the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other through a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0134] A number of components in the electronic device 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0135] The processor 11 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The processor 11 executes the various methods and processes described above, such as an element positioning method based on a large model, the method comprising: in response to a positioning description instruction of a target element, obtaining a target tree structure corresponding to the target element; the target element is a target child node of the target tree structure; from the root node of the target tree structure to the target child node, the target tree structure is traversed level by level to extract the node features of each node of the target tree structure; the target feature of the target element is determined according to the node features of each node; a large model is generated based on the target feature and a pre-trained path selector to obtain a selector statement matching the target element, and the target element is positioned based on the selector statement.

[0136] In some embodiments, the element positioning method based on the large model can be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 18. In some embodiments, part or all of the computer program can be loaded and / or installed on the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the element positioning method based on the large model described above can be performed. Alternatively, in other embodiments, the processor 11 can be configured to execute the element positioning method based on the large model in any other appropriate manner (for example, by means of firmware).

[0137] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0138] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0139] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, device, or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0140] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0141] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0142] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.

[0143] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.

[0144] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for locating elements based on a large model, characterized in that: include: In response to a location description instruction of a target element, acquiring a target tree structure corresponding to the target element; The target element is a target child node of the target tree structure; Traversing the target tree structure from the root node of the target tree structure to the target child node level by level, and extracting node features of each node of the target tree structure; Determining the target feature of the target element according to the node feature of each of the nodes; A large model is generated based on the target features and a pre-trained path selector to obtain a selector statement matching the target element, and the target element is located based on the selector statement.

2. The element positioning method based on a large model according to claim 1, characterized in that: The path selector generation model is trained by the following steps: Obtaining a reference element dataset; Determine data features of each element in the reference element data set, and selector statements matching the data features; The data features of each of the elements and the selector statements matching the data features are input into the pre-trained large language model for fine-tuning to obtain a path selector generation large model.

3. The element positioning method based on a large model according to claim 2 is characterized in that: Determining the data features of each element in the reference element data set includes: The data features of each element in the reference element data set are extracted by a feature extraction module.

4. The element positioning method based on a large model according to claim 3 is characterized in that: The feature extraction module extracts the data features of each element through the following steps: Starting from the root node of the target tree structure, node features of each node in the target tree structure are extracted step by step; Combining the node features to obtain at least one path feature, and determining the path features as data features of the elements; The node characteristics include at least one of the following: Tags, attributes, content, and location in the target tree structure; The path characteristics include at least one of the following: Sequence of tags, combination of attributes, and combination of text content.

5. The element positioning method based on a large model according to claim 1, characterized in that: The target tree structure is a document object model DOM tree structure; Accordingly, traversing the target tree structure level by level from the root node of the target tree structure to the target child node to extract node features of each node of the target tree structure includes: Starting from the root node of the DOM tree structure, extracting node features of each node in the DOM tree structure in sequence until the target child node; Each node of the DOM tree structure includes a root node and child nodes; The node characteristics include at least one of the following: Tags, attributes, content, and position in the DOM tree structure.

6. The element positioning method based on a large model according to claim 5, characterized in that: The determining the target feature of the target element according to the node feature of each of the nodes includes: Combining the node features to obtain at least one path feature; Determine each of the path features as a target feature of the target element; The path characteristics include at least one of the following: Sequence of tags, combination of attributes, and combination of text content.

7. The element positioning method based on a large model according to claim 1, characterized in that: The generating of a large model based on the target feature and the pre-trained path selector to obtain a selector statement matching the target element includes: The target feature is input into the pre-trained path selector generation model, and a selector statement matching the target element is output.

8. An element positioning device based on a large model, characterized in that: include: A target tree structure acquisition module, configured to acquire a target tree structure corresponding to the target element in response to a location description instruction of the target element; the target element is a target child node of the target tree structure; A node feature extraction module, used to traverse the target tree structure from the root node of the target tree structure to the target child node, and extract node features of each node of the target tree structure; A target feature determination module, used to determine the target feature of the target element according to the node feature of each node; A positioning module is used to generate a large model based on the target features and a pre-trained path selector, obtain a selector statement matching the target element, and locate the target element based on the selector statement.

9. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can perform the large model-based element positioning method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the large model-based element positioning method according to any one of claims 1 to 7 when executed.

Images