Browser element positioning method and system based on natural language processing

Abstract

The invention provides a browser element positioning method and system based on natural language processing, and the method comprises the steps: obtaining a natural language description inputted by a user, carrying out the preprocessing, carrying out the feature extraction and semantic understanding, and obtaining the operation intention of the user; performing analysis and semantic understanding on the combination of natural language description and a context window by utilizing a natural language processing technology; the method comprises the following steps: starting a WebDriver, starting a specified browser, and constructing a JavaScript script; highlighting the elements conforming to the description, and classifying the click events; and performing fusion processing on the natural language description of the user and the webpage screenshot to obtain a natural language expression word vector, and obtaining final target element positioning by adopting a target detection network. According to the method, the defects of traditional hard coding and script modes are overcome, the technical threshold of a user is lowered, the flexibility and the intelligent level of the system are improved, a low-code platform can better meet the personalized requirements of the user, the user experience is improved, and the method has wide application prospects.

Description

Technical Field

[0001] The present invention belongs to the field of natural language processing, and in particular relates to a browser element positioning method and system based on natural language processing. Background Art

[0002] With the development of Robotic Process Automation (RPA) technology, browser automation has become one of its core applications. In practical applications, one of the main challenges of browser automation is how to accurately and efficiently locate interface elements in the browser for corresponding control and operation. Existing solutions usually rely on browser plug-ins to achieve the location of interface elements.

[0003] Specifically, the existing technology is mainly implemented through the following process: install a plug-in in the browser to be controlled, start the plug-in and monitor the positioning request issued by the RPA designer. When the RPA designer initiates an element positioning request, the browser plug-in receives the request and manipulates the browser's DOM (Document Object Model) structure to obtain the element's positioning information, and then sends this information back to the RPA designer, which then displays and saves it.

[0004] However, this plugin-dependent approach has significant drawbacks:

[0005] 1. Large development workload: Different browsers (such as Chrome, IE, Firefox, Edge, etc.) require the development of dedicated plug-ins. Due to the different development mechanisms of browser plug-ins, the development workload is large and the technical difficulty is high.

[0006] 2. High deployment and maintenance costs: When deploying an application, you need to manually add plug-ins to the browser, which not only increases the deployment cost, but also brings additional maintenance costs when the plug-in is updated and upgraded.

[0007] 3. Stability and compatibility issues: The stability and reliability of plug-ins are poor, and compatibility issues are prone to occur, especially during frequent browser version updates.

[0008] 4. Security risks: Plug-ins need to access and control the browser's internal data, which may involve security risks and privacy issues.

[0009] In order to solve the above problems, the industry urgently needs an interface element positioning method that does not rely on specific browser plug-ins. Summary of the invention

[0010] The purpose of the present invention is to propose a browser element positioning method and system based on natural language processing, which realizes efficient and accurate interface element positioning by combining NLP and deep learning models and utilizing unified WebDriver technology, thus solving many problems in the prior art.

[0011] In order to achieve the above object, the present invention provides a browser element positioning method based on natural language processing, the method comprising:

[0012] S1. Obtain and preprocess the natural language description input by the user, perform feature extraction and semantic understanding on the preprocessed natural language description, and perform intent recognition based on the representation vector obtained by the semantic understanding to obtain the user's operation intention;

[0013] S2. Use natural language processing technology to parse and semantically understand the preprocessed natural language description and the context window, convert the natural language description input by the user into a structured information representation, obtain the user's intention and classify it;

[0014] S3, start WebDriver to start the specified browser, and build a JavaScript script for locating web page elements, inject and execute the JavaScript script through the WebDriver interface, obtain the location information of the target element, verify whether the obtained XPath is correct, and output it in the console, and pass the obtained element information to S4 for processing;

[0015] S4, based on the result of S3, highlight the elements that meet the description, capture the user's click event through the browser event listener and classify the click event using the enhanced support vector machine, filter out non-target click events, and enhance the accuracy of the click event through context information and user behavior patterns, and further use computational geometry methods to generate the precise bounding box of the element to enhance the element information, and then let the user confirm whether the selected target element information is correct, and pass the target element information selected and confirmed by the user to S5 for processing;

[0016] S5. The user's natural language description and web page screenshot are fused as multimodal input data to obtain a natural language description word vector, and then the feature vectors of the natural language description and the web page screenshot are extracted based on the natural language description word vector. Based on the attention mechanism, the feature vectors of the natural language description and the web page screenshot are multimodally fused to obtain a fused comprehensive feature vector, and the comprehensive feature vectors are jointly represented to obtain a final feature representation. According to the final feature representation, a target detection network is used to locate the target element of the web page screenshot to obtain the final target element location.

[0017] Furthermore, the preprocessing includes cleaning the text S input by the user, removing special characters, punctuation marks and extra spaces in the user input to obtain a cleaned sentence S', then performing word segmentation on the cleaned sentence S' to obtain a vocabulary sequence W, performing part-of-speech tagging on the vocabulary sequence W to generate a part-of-speech sequence P;

[0018] The feature extraction and semantic understanding of the preprocessed natural language description includes converting the vocabulary sequence W into a word vector sequence V using a word frequency-inverse document frequency method, and then reducing the dimension of the word vector sequence V using a latent semantic analysis technique to obtain a representation vector U in a low-dimensional semantic space;

[0019] The method of performing intention recognition based on the representation vector obtained by semantic understanding to obtain the user's operation intention includes using a naive Bayes classifier to classify the sentence representation vector U to obtain the user's operation intention I.

[0020] Furthermore, the verification of the user's operation intention I is expressed as follows:

[0021]

[0022] Among them, P(C j |U) is the category C under the given input U j The posterior probability, P(U|C j ) is in category C j The likelihood of the input U, P(C j ) is Category C j The prior probability of , P(U) is the marginal probability of input U.

[0023] Further, a context window is obtained according to the vocabulary sequence W, where W = {w1, w2, ..., w i ,w j ,w n}, each word w i The context window C i Contains a fixed number of words before and after k, and then according to the context window C i Construct the vocabulary co-occurrence matrix M, which is expressed as follows:

[0024]

[0025] Wherein, δ(x, y) represents the Kroneckerdelta function, when x=y, δ(x, y)=1, otherwise δ(x, y)=0; C k represents the context window; n represents the number of words; l represents the current context window, and the matrix element M ij Representation word w i and w j The number of occurrences in the same context window;

[0026] Then the co-occurrence matrix M is converted into a word vector using the global vector method, which is expressed as follows:

[0027]

[0028] Among them, v i and vj Representation word w i and w j The word vector of b i and b j represents the bias term, ∈ ij represents the error term;

[0029] Calculate the central word vector v of the specific domain vocabulary according to the preset scenario domain And perform semantic enhancement to obtain the enhanced semantic vector v' i , according to the enhanced semantic vector v' i The word vectors are weighted averaged to generate the sentence vector s, which is expressed as follows:

[0030]

[0031] Among them, tf-idf(w i ) represents word w i TF-IDF value of

[0032] The support vector machine is then used to perform intent recognition and classification on the sentence representation vector s.

[0033] Furthermore, the objective function of the support vector machine is expressed as follows:

[0034]

[0035] And satisfy the constraints, expressed as follows:

[0036] y i (w·s i +b)≥1-ξ i ,ξ i ≥0

[0037] Where w and b represent the weight and bias of the support vector machine respectively, ξ i represents the slack variable, C represents the regularization parameter, and y i represents the classification label, m represents the number of samples, s i Represents the current sentence vector.

[0038] Further, in S4, the objective function of the enhanced support vector machine is expressed as follows:

[0039]

[0040] And satisfy the constraints:

[0041] y i (w·φ(x i )+b)≥1-ξ i ,ξ i ≥0

[0042] Where w and b represent the weight and bias of the enhanced support vector machine, ξ i represents the slack variable, C represents the regularization parameter, and w i represents the importance weight of the click event, λ represents the feature interaction regularization coefficient, γ represents the L1 regularization coefficient, φ(x i ) represents the feature interaction function.

[0043] Furthermore, during the training process, the sensitivity to the target click event is increased by combining the specific behavior patterns and context information in the preset scenario through the weighted enhanced support vector machine method, where w in the objective function is i The calculation formula is as follows:

[0044] w i =1+αf(x i )+βg(t i )

[0045] Among them, α and β are adjustment parameters, f(x i ) is a weighting function based on element features, g(t i ) is a weighting function based on time characteristics;

[0046] According to the high-order interaction between features, the feature interaction function φ(x i ), which is expressed as follows:

[0047]

[0048] in, Represents the outer product operation of features.

[0049] Furthermore, in the computational geometry method, when generating an accurate bounding box of an element to enhance the element information, the bounding box calculation is expressed as follows:

[0050] BoundingBox(x,y,width,height)=(x,y,x+width,y+height)

[0051] Semantic enhancement vector calculation formula:

[0052] v enhanced =v element +λ·v context

[0053] Among them, λ is the semantic enhancement coefficient, v enhanced represents the enhanced semantic vector, v element represents an element vector, v context Represents a sentence vector.

[0054] Furthermore, the comprehensive feature vectors are jointly represented to obtain the final feature representation, including:

[0055] The fused feature vector H is passed through the first fully connected layer and expressed as follows:

[0056] F1=ReLU(W1H+b1)

[0057] Among them, W1 represents the weight matrix of the first fully connected layer, b1 represents the bias term, and ReLU represents the activation function;

[0058] Then pass F1 through the second fully connected layer, expressed as follows:

[0059] F2=ReLU(W2F1+b2)

[0060] Among them, W2 represents the weight matrix of the second fully connected layer, b2 represents the bias term, and ReLU represents the activation function;

[0061] Finally, the final feature representation F is generated, which is expressed as follows:

[0062] F=ReLU(W3F2+b3+λ||F2|| 2 +γ||W3||)

[0063] Among them, W3 represents the weight matrix of the third fully connected layer, b3 represents the bias term, ReLU represents the activation function, and λ||F2|| 2 is the L2 regularization term, γ||W3|| represents the weight attenuation term;

[0064] The target element location is obtained by using the target detection network to locate the target element in the webpage screenshot according to the final feature representation, which is expressed as follows:

[0065] L = Faster R-CNN (S, F)

[0066] Among them, L represents the location of the target element, Faster R-CNN represents the target detection network, and S represents the web page screenshot.

[0067] In another aspect of the present invention, a browser element positioning system based on natural language processing is provided, the system comprising:

[0068] A natural language acquisition unit is used to acquire and preprocess the natural language description input by the user, perform feature extraction and semantic understanding on the preprocessed natural language description, and perform intent recognition based on the representation vector obtained by the semantic understanding to acquire the user's operation intention;

[0069] A natural language analysis unit is used to analyze and semantically understand the preprocessed natural language description and the context window using natural language processing technology, convert the natural language description input by the user into a structured information representation, obtain the user's intention and classify it;

[0070] The browser initialization unit is used to start the WebDriver to start the specified browser, and build a JavaScript script for locating web page elements, inject and execute the JavaScript script through the WebDriver interface, obtain the location information of the target element, verify whether the obtained XPath is correct, and output it in the console, and pass the obtained element information to the target element processing unit for processing;

[0071] The target element processing unit is used to highlight the elements that meet the description according to the result of the browser initialization unit, capture the user click event through the browser event listener and classify the click event by using the enhanced support vector machine, filter out non-target click events, and enhance the accuracy of the click event by using context information and user behavior patterns, and further generate the precise bounding box of the element by using the computational geometry method to enhance the element information, and then let the user confirm whether the selected target element information is correct, and pass the target element information selected and confirmed by the user to the target element positioning unit for processing;

[0072] The target element positioning unit is used to fuse the user's natural language description and web page screenshot as multimodal input data to obtain a natural language description word vector, then extract the feature vectors of the natural language description and the web page screenshot based on the natural language description word vector, and based on the attention mechanism, multimodally fuse the feature vectors of the natural language description and the web page screenshot to obtain a fused comprehensive feature vector, jointly represent the comprehensive feature vector to obtain a final feature representation, and use a target detection network to locate the target element of the web page screenshot based on the final feature representation to obtain the final target element location.

[0073] The beneficial technical effects of the present invention are at least as follows:

[0074] (1) The present invention introduces NLP technology to parse the natural language description input by the user and extract key features and context information. Through pre-trained NLP models (such as BERT or GPT), the system can understand the user's operation intention and accurately locate elements in combination with the web page content. This innovation solves the problem of manually adding plug-ins and complex configuration in traditional methods, and improves the user experience.

[0075] (2) The present invention uses the browser's WebDriver technology to achieve the positioning of interface elements of different browsers by dynamically injecting scripts. This method does not require the development of dedicated plug-ins for each browser, greatly reducing the development workload and technical difficulty, and improving the versatility and flexibility of the system. Users only need to initiate a request through the RPA designer, and the system can complete the element positioning, simplifying the operation process and improving the ease of use and security of the system.

[0076] (3) The present invention combines a deep learning model with multimodal fusion technology to combine web page screenshots with text description information to achieve high-precision element positioning. Using a deep learning model (such as Transformer or ViLT), the system can accurately identify and locate target elements in complex and changeable web page structures, significantly improving the accuracy and reliability of positioning.

[0077] (4) This invention introduces NLP and deep learning technologies, combined with unified WebDriver technology, to propose an innovative intelligent browser element positioning method and system, which comprehensively solves the problems of development, deployment, stability and accuracy in the prior art. This method not only significantly improves the efficiency and accuracy of interface element positioning, but also greatly reduces development and maintenance costs, providing strong support for the widespread application of RPA technology. BRIEF DESCRIPTION OF THE DRAWINGS

[0078] The present invention is further described using the accompanying drawings, but the embodiments in the accompanying drawings do not constitute any limitation to the present invention. A person skilled in the art can obtain other drawings based on the following drawings without creative work.

[0079] Figure 1 This is a flow chart of the browser element positioning method based on natural language processing of the present invention. DETAILED DESCRIPTION

[0080] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and cannot be understood as limiting the present invention.

[0081] like Figure 1 As shown, the browser element positioning based on natural language processing provided by an embodiment of the present invention comprises the following steps S1-S5:

[0082] S1. Obtain the natural language description of the user input and perform preprocessing, perform feature extraction and semantic understanding on the preprocessed natural language description, and perform intent recognition based on the representation vector obtained by the semantic understanding to obtain the user's operation intention.

[0083] Specifically, the preprocessing includes:

[0084] Remove special characters, punctuation marks, and extra spaces from user input to ensure clean input. Assume the user inputs a sentence S, and the processed sentence is S':

[0085] S'=clean(S) (1)

[0086] Among them, the clean() function is used to remove special characters, punctuation marks and extra spaces. The specific implementation is as follows:

[0087] S'=regex_sub("[ α -zA-Z0-9\s]”,””,S) (2)

[0088] The regex_sub() function uses regular expressions to remove non-alphanumeric and space characters.

[0089] The cleaned sentence S' is segmented to obtain the vocabulary sequence W = {w1,w2,…,w n}:

[0090] W=tokenize(S') (3)

[0091] The tokenize() function is used to convert a sentence into a word sequence. The specific implementation is as follows:

[0092] W=S'.split(””) (4)

[0093] Perform part-of-speech tagging on the word sequence W after word segmentation to generate a part-of-speech sequence P = {p1, p2, ..., p n}:

[0094] P = pos_tag(W) (5)

[0095] Among them, the pos_tag() function is used to perform part-of-speech tagging on the vocabulary sequence. The specific implementation can use the pos_tag function in the NLTK library.

[0096] Furthermore, the term frequency-inverse document frequency (TF-IDF) method is used to convert the vocabulary sequence W into a word vector sequence V = {v1, v2, ..., v n}.

[0097] v i =TF-IDF(wi ), i=1,2,…,n (6)

[0098] Among them, v i Represents word w i The TF-IDF vector of . The specific implementation is as follows:

[0099]

[0100] Among them, tf(w i ) is the word w i The frequency of the word in the document, df(w i ) contains the word w i N is the total number of documents.

[0101] Furthermore, the latent semantic analysis (LSA) technique is used to reduce the dimension of the word vector sequence V to obtain the representation vector U={u1,u2,…,u k}, where k is the dimension of the vector after dimensionality reduction:

[0102] U=LSA(V) (8)

[0103] Among them, the LSA() function is used to convert high-dimensional word vectors into representation vectors in low-dimensional semantic space. The specific implementation is as follows:

[0104] U=V·T (9)

[0105] Where T is the transformation matrix obtained from SVD decomposition:

[0106]

[0107] Among them, U k ,Σ k and They are the matrices corresponding to the first k eigenvalues ​​retained after SVD decomposition of the word vector matrix V.

[0108] For a further understanding, we use the Naive Bayes Classifier to classify the sentence representation vector U and determine the user's operation intention I:

[0109] I=NaiveBayes(U) (11)

[0110] Where I represents the user's operation intention. The specific implementation is as follows:

[0111]

[0112] Among them, P(C j |U) is the category C under the given input U jThe posterior probability, P(U|C j ) is in category C j The likelihood of the input U, P(C j ) is Category C j The prior probability of , P(U) is the marginal probability of input U.

[0113] S2. Use natural language processing technology to parse and semantically understand the preprocessed natural language description and the context window, convert the natural language description input by the user into a structured information representation, obtain the user's intention and classify it.

[0114] According to the vocabulary sequence W = {w1, w2, ..., w n}, each word w i The context window C i Contains a fixed number of words before and after (k):

[0115] C i ={w i-k ,…,w i-1 ,w i+1 ,…,w i+k} (13)

[0116] For example, for word w i , context window C i Contains the word w i-1 and w i+1 , if k = 1. The goal of context window extraction is to capture the relevance of each word with its neighboring words to enhance the semantic information of the word.

[0117] Further, construct the word co-occurrence matrix M, the matrix element M ij Representation word w i and w j Number of occurrences in the same context window:

[0118]

[0119] Among them, δ(x, y) is the Kroneckerdelta function, when x = y, δ(x, y) = 1, otherwise δ(x, y) = 0. The construction of the co-occurrence matrix can capture the co-occurrence relationship between words and provide a basis for word vector representation.

[0120] Furthermore, the co-occurrence matrix M is converted into word vectors using the global vector (GloVe) method. The goal is to find the word vector matrix V such that:

[0121]

[0122] Among them, v i and vj It is the word w i and w j The word vector of b i and b j is the bias term, ∈ ij is the error term. The GloVe method can generate high-quality word vectors through the statistical characteristics of the co-occurrence matrix.

[0123] In order to adapt to the specific scenario of patents, we perform a specific transformation on word vectors. Assuming that the patent involves a specific domain vocabulary (such as "login button"), by calculating the central word vector v of the specific domain vocabulary domain And perform semantic enhancement:

[0124]

[0125] Where D is a domain-specific vocabulary set, v d is the word vector of vocabulary d. Semantic vector conversion can improve the semantic understanding ability in patent scenarios.

[0126] Perform semantic enhancement on each word vector:

[0127] v' i =v i +λ·v domain (17)

[0128] Among them, λ is the semantic enhancement coefficient. The semantic enhancement coefficient can control the impact of domain semantics on word vectors.

[0129] The sentence vector s is generated by taking a weighted average of the word vectors, and the weight coefficient is calculated based on the term frequency-inverse document frequency (TF-IDF):

[0130]

[0131] Among them, tf-idf(w i ) represents word w i The TF-IDF value of . The sentence representation is generated by weighted average to ensure that the overall semantics of the sentence is captured.

[0132] It can be understood that the support vector machine (SVM) is used to identify and classify the sentence representation vector s. The objective function of SVM is:

[0133]

[0134] Satisfy the constraints:

[0135] y i (w·s i +b)≥1-ξ i ,ξ i ≥0 (20)

[0136] Where w and b are the weight and bias of SVM, ξ i is the slack variable, C is the regularization parameter, and y i It is a classification label. SVM is used to identify and classify intentions, ensuring that the system can accurately understand the user's operation intentions.

[0137] S3, start WebDriver to start the specified browser, and build a JavaScript script for locating web page elements, inject and execute the JavaScript script through the WebDriver interface, obtain the location information of the target element, verify whether the obtained XPath is correct, and output it in the console, and pass the obtained element information to S4 for processing.

[0138] Specifically, use WebDriver to start the specified browser. Assuming that the browser selected by the user is Chrome, the browser options need to be configured during startup to ensure that the window is maximized, extensions are disabled, etc., to ensure the consistency of the script execution environment.

[0139] The startup command includes setting the browser driver path, startup options, etc.:

[0140] driver=webdriver.Chrome(service=chromeService,options=chromeOptions) (21)

[0141] Among them, chromeService is the path of ChromeDriver, and chromeOptions is the browser startup options.

[0142] Furthermore, a JavaScript script for locating web page elements is constructed. This script needs to be dynamically generated according to the user's natural language input parsing result to achieve accurate positioning of specific elements.

[0143] Assuming the user entered a description of "Login Button", the script should locate the button element containing the "Login" text:

[0144] script="return document.querySelector('button:contains(login)').getXPath();" (22)

[0145] Among them, getXPath() is a custom JavaScript function used to obtain the XPath of an element.

[0146] Inject and execute JavaScript scripts through the WebDriver interface to obtain the location information of the target element.

[0147] Execute the script and get the results:

[0148] xpath=driver.executeScript(script) (23)

[0149] Among them, executeScript() is a WebDriver interface used to execute the injected JavaScript script.

[0150] It is understandable that the obtained XPath is verified to be correct and outputted in the console. The user intent parsed in step S2 is compared with the result obtained in step S3 to ensure that the obtained element information is accurate.

[0151] The verification process involves comparing the user's intent with the actual element attributes (such as tag name, text content, etc.) to ensure a match:

[0152] assert(element tag name == 'button' and element text contains 'login') (24)

[0153] Furthermore, the obtained element information is passed to the next step for processing. If the obtained XPath needs to be used for subsequent operations, it is stored in a specific variable for subsequent steps to call.

[0154] The information delivery format should be kept consistent to ensure the consistency of subsequent processing:

[0155] elementXpath=xpath(25).

[0156] S4. Highlight the elements that meet the description based on the results of S3, capture user click events through the browser event listener and classify the click events using the enhanced support vector machine, filter out non-target click events, and enhance the accuracy of click events through context information and user behavior patterns. Further use computational geometry methods to generate precise bounding boxes of elements to enhance element information, then ask the user to confirm whether the selected target element information is correct, and pass the target element information selected and confirmed by the user to S5 for processing.

[0157] Specifically, natural language processing (NLP) technology is used to parse user input and extract keywords and context information. The user input is converted into a high-dimensional vector representation through a word vector model (such as Word2Vec). Similarity calculation is used to match the elements in the page and highlight the elements that meet the description. Assuming that the user input description is D, the page element is E, and the word vector representation is v, the similarity calculation formula is as follows:

[0158]

[0159] Elements with similarity higher than the threshold are highlighted.

[0160] Then, the user click events are captured through the browser event listener. Combined with the patent scenario, the enhanced support vector machine (SVM) is used to classify click events, filter non-target click events, and enhance the accuracy of click events through contextual information and user behavior patterns. When the page is loaded, JavaScript code is injected to listen to all click events. The detailed information of the click event (such as element location, tag name, click time) is recorded to form a click data set.

[0161] Further, extract the feature vector x from the click dataset i , including the element's XPath, the element's CSS style, the mouse position, timestamp, etc. Combined with the user's historical click behavior and page context information, high-level features such as user click frequency and element relative position are further extracted.

[0162] Furthermore, combined with patent scenarios, special features are designed for SVM classification:

[0163] Consider the relative position of the element on the page (such as in the navigation bar, text area, footer, etc.), the visibility of the element (whether it is blocked);

[0164] Analyze the user's historical click behavior patterns, such as the location where the user usually clicks, click frequency, and continuous click patterns;

[0165] Consider the time characteristics of user clicks, such as the time interval between clicks, timestamps, etc.

[0166] It can be understood that the enhanced support vector machine (SVM) is used to classify the feature vector to determine whether the click event is a target click event. The enhanced SVM objective function is as follows:

[0167]

[0168] Satisfy the constraints:

[0169] y i (w·φ(xi )+b)≥1-ξ i ,ξ i ≥0 (28)

[0170] Where w and b are the weight and bias of SVM, ξ i is the slack variable, C is the regularization parameter, and w i is the importance weight of the click event, λ is the feature interaction regularization coefficient, γ is the L1 regularization coefficient, φ(x i ) is the feature interaction function.

[0171] During the training process, the specific behavior patterns and context information in the patent scenario are combined to increase the sensitivity to the target click event through the weighted SVM method. i The calculation formula is:

[0172] w i =1+αf(x i )+βg(t i ) (29)

[0173] Among them, α and β are adjustment parameters, f(x i ) is a weighting function based on element features, g(t i ) is a weighting function based on time characteristics.

[0174] Furthermore, considering the high-order interaction between features, the feature interaction function φ(x i ) is defined as:

[0175]

[0176] in, Represents the outer product operation of features.

[0177] Through the above method, the present invention can accurately capture the target element selected by the user in the browser, and enhance the accuracy of click events through context information and user behavior patterns, filter out non-target click events, and improve the accuracy of the overall system and user experience.

[0178] Furthermore, computational geometry methods are used to generate the precise bounding box of the element, and the XPath, CSS selector, tag name, and text content of the element are obtained. The word vector model is combined to enhance the semantic understanding of element information. Bounding box calculation formula:

[0179] BoundingBox(x,y,width,height)=(x,y,x+width,y+height) (31)

[0180] Semantic enhancement vector calculation formula:

[0181] v enhanced =v element +λ·v context (32)

[0182] Among them, λ is the semantic enhancement coefficient.

[0183] Generate a detailed description of the selection result, and use a visualization tool to show the location and information of the element on the page. If the selection is incorrect, the user can reselect. Use natural language generation (NLG) technology to generate the description of the selection result:

[0184] Description=NLG(ElementInfo) (33)

[0185] Among them, ElementInfo contains the element's tag name, text content, XPath and other information.

[0186] Furthermore, the element information is stored in a specific data structure, and the message queue system (such as Kafka) is used to ensure the transmission and consistency of information between multiple system modules. Information transmission format:

[0187] selectedElementInfo={XPath:xpath,CSS:css,Tag:tag,Text:text} (34)

[0188] S5. The user's natural language description and web page screenshot are fused as multimodal input data to obtain a natural language description word vector, and then the feature vectors of the natural language description and the web page screenshot are extracted based on the natural language description word vector. Based on the attention mechanism, the feature vectors of the natural language description and the web page screenshot are multimodally fused to obtain a fused comprehensive feature vector, and the comprehensive feature vectors are jointly represented to obtain a final feature representation. According to the final feature representation, a target detection network is used to locate the target element of the web page screenshot to obtain the final target element location.

[0189] It can be understood that the user's natural language description is converted into a word vector representation, and a screenshot of the web page is obtained.

[0190] Then, the natural language description word vector representation T:

[0191]

[0192] Where D is the natural language description entered by the user, N is the number of words in the description, and v(w i ) is the word w iThe word vectors are obtained through pre-trained word vector models (such as Word2Vec). The word vector model is trained on a large-scale corpus to capture the semantic relationship between words.

[0193] Get webpage screenshot S: Get a visual representation of the current webpage through the browser screenshot function, ensuring that the image matches the user's natural language description.

[0194] By converting the user's natural language description into word vectors and obtaining web page screenshots, basic data for multimodal fusion is provided to ensure that the system can process input information in different modalities.

[0195] Furthermore, feature vectors of natural language descriptions and web page screenshots are extracted:

[0196] Extract context features from word vector T to generate text feature vector H t :

[0197] H t =BiLSTM(T) (36)

[0198] Among them, BiLSTM is a bidirectional long short-term memory network, which is used to extract text features. The forward and reverse LSTM units of BiLSTM process the forward and backward information of the sequence respectively, and obtain the text feature vector by combining the hidden states:

[0199]

[0200] Perform convolutional neural network (CNN) feature extraction on the webpage screenshot S to generate the image feature vector H s :

[0201] H s =CNN(S) (38)

[0202] Among them, CNN consists of multiple layers of convolutional layers and pooling layers, which extract the spatial features of the image through local receptive fields and weight sharing mechanisms. The output of the convolutional layer is processed by the ReLU activation function:

[0203] H s =ReLU(conv(S)) (39)

[0204] Text and image features are extracted through BiLSTM and CNN to generate high-quality feature vectors, ensuring that the system can capture the deep semantic information of the input data.

[0205] Furthermore, a multimodal fusion method based on attention mechanism is used to transform the text feature vector H t and image feature vector H s The fusion is performed to generate a joint feature vector H.

[0206] Specifically, calculate the attention weight α of text features to image features:

[0207]

[0208] Among them, W t is a learnable weight matrix and softmax is an activation function used to normalize the weights.

[0209] Calculate the attention weight β of image features to text features:

[0210]

[0211] Among them, W s is the learnable weight matrix.

[0212] Generate fused feature vector H:

[0213] H=αH t +βH s (42)

[0214] Through a multimodal fusion method based on the attention mechanism, text features and image features are weightedly fused to generate a joint feature vector, which improves the expressiveness and robustness of the model.

[0215] Furthermore, the joint feature vector H is transformed into a complex nonlinear form through multiple layers of fully connected layers to generate the final feature representation F, as follows:

[0216] First, the fused feature vector H is passed through the first fully connected layer:

[0217] F1=ReLU(W1H+b1) (43)

[0218] Among them, W1 is the weight matrix of the first fully connected layer, b1 is the bias term, and ReLU is the activation function.

[0219] Secondly, pass F1 through the second fully connected layer:

[0220] F2=ReLU(W2F1+b2) (44)

[0221] Among them, W2 is the weight matrix of the second fully connected layer, b2 is the bias term, and ReLU is the activation function.

[0222] Finally, the final feature representation F is generated:

[0223] F=ReLU(W3F2+b3+λ||F2|| 2 +γ||W3||) (45)

[0224] Among them, W3 is the weight matrix of the third fully connected layer, b3 is the bias term, ReLU is the activation function, λ||F2|| 2 is the L2 regularization term, and γ||W3|| is the weight decay term. In the process of locating web page elements, the size of the data set may be relatively small or the correlation between features is strong, which can easily lead to overfitting of the model. Overfitting will make the model perform well on the training data, but perform poorly in practical applications. The regularization term λ|F2| is introduced. 2 It can effectively prevent overfitting and ensure that the model has better generalization ability on different web pages and elements. In the task of locating web page elements, the model may need to adapt to web pages of different styles, structures and layouts, which places high demands on the stability of the model. Introducing the weight decay term γ|W3| can prevent the model's weight from being too large, improve the model's stability and robustness, and enable it to maintain high accuracy in different web pages and scenarios.

[0225] The joint feature vector is processed through multiple layers of fully connected layers and complex nonlinear transformations to generate the final feature representation, which provides rich feature information for target element positioning.

[0226] Furthermore, based on the joint feature representation F, a target detection network (such as FasterR-CNN) is used to locate the target elements in the web page screenshot.

[0227] Specifically, FasterR-CNN generates candidate regions through the region proposal network (RPN) and uses convolutional features for classification and boundary regression. Locate the position L of the target element:

[0228] L = Faster R-CNN (S, F) (46)

[0229] The target detection network is used to locate the target elements in web page screenshots, and the precise positioning of the target elements is achieved based on the joint feature representation, which improves the accuracy and robustness of positioning.

[0230] Another embodiment of the present invention is provided, a browser element positioning system based on natural language processing, the system comprising:

[0231] The natural language acquisition unit 101 is used to acquire and preprocess the natural language description input by the user, perform feature extraction and semantic understanding on the preprocessed natural language description, and perform intent recognition based on the representation vector obtained by the semantic understanding to acquire the user's operation intention;

[0232] A natural language analysis unit 102 is used to analyze and semantically understand the preprocessed natural language description and the context window using natural language processing technology, convert the natural language description input by the user into a structured information representation, obtain the user's intention and classify it;

[0233] The browser initialization unit 103 is used to start the WebDriver to start the specified browser, and build a JavaScript script for locating the web page element, inject and execute the JavaScript script through the WebDriver interface, obtain the location information of the target element, verify whether the obtained XPath is correct, and output it in the console, and pass the obtained element information to the target element processing unit for processing;

[0234] The target element processing unit 104 is used to highlight the elements that meet the description according to the result of the browser initialization unit, capture the user click event through the browser event listener and classify the click event using the enhanced support vector machine, filter out non-target click events, and enhance the accuracy of the click event through context information and user behavior patterns, and further generate the precise bounding box of the element using the computational geometry method to enhance the element information, and then let the user confirm whether the selected target element information is correct, and pass the target element information selected and confirmed by the user to the target element positioning unit for processing;

[0235] The target element positioning unit 105 is used to fuse the user's natural language description and the web page screenshot as multimodal input data to obtain a natural language description word vector, then extract the feature vectors of the natural language description and the web page screenshot based on the natural language description word vector, and based on the attention mechanism, multimodally fuse the feature vectors of the natural language description and the web page screenshot to obtain a fused comprehensive feature vector, jointly represent the comprehensive feature vector to obtain a final feature representation, and use a target detection network to locate the target element of the web page screenshot based on the final feature representation to obtain the final target element location.

[0236] It should be noted that the workflow described above is merely illustrative and does not limit the scope of protection of the present invention. In practical applications, technicians in this field can select part or all of them according to actual needs to achieve the purpose of the present embodiment, and no limitation is made here.

[0237] In addition, for technical details that are not described in detail in this embodiment, reference can be made to the parameter operation method provided in any embodiment of the present invention, and will not be repeated here.

[0238] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or system including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or system. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or system including the element.

[0239] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

[0240] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as a read-only memory / random access memory, a magnetic disk, or an optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to execute the methods described in each embodiment of the present invention.

[0241] The above are only preferred embodiments of the present invention, and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A browser element positioning method based on natural language processing, characterized in that: The method comprises: S1. Obtain and preprocess the natural language description input by the user, perform feature extraction and semantic understanding on the preprocessed natural language description, and perform intent recognition based on the representation vector obtained by the semantic understanding to obtain the user's operation intention; S2. Use natural language processing technology to parse and semantically understand the preprocessed natural language description and the context window, convert the natural language description input by the user into a structured information representation, obtain the user's intention and classify it; S3, start WebDriver to start the specified browser, and build a JavaScript script for locating web page elements, inject and execute the JavaScript script through the WebDriver interface, obtain the location information of the target element, verify whether the obtained XPath is correct, and output it in the console, and pass the obtained element information to S4 for processing; S4, highlighting the elements that meet the description according to the result of S3, capturing the user click event through the browser event listener and classifying the click event using the enhanced support vector machine, filtering out non-target click events, and enhancing the accuracy of the click event through context information and user behavior patterns, and further using computational geometry methods to generate the precise bounding box of the element to enhance the element information, and then asking the user to confirm whether the selected target element information is correct, and passing the target element information selected and confirmed by the user to S5 for processing; S5. The user's natural language description and web page screenshot are fused as multimodal input data to obtain a natural language description word vector, and then the feature vectors of the natural language description and the web page screenshot are extracted based on the natural language description word vector. Based on the attention mechanism, the feature vectors of the natural language description and the web page screenshot are multimodally fused to obtain a fused comprehensive feature vector, and the comprehensive feature vectors are jointly represented to obtain a final feature representation. According to the final feature representation, a target detection network is used to locate the target element of the web page screenshot to obtain the final target element location.

2. The browser element positioning method based on natural language processing according to claim 1 is characterized in that: The preprocessing includes cleaning the text S input by the user, removing special characters, punctuation marks and extra spaces in the user input to obtain a cleaned sentence S′, then performing word segmentation on the cleaned sentence S′ to obtain a vocabulary sequence W, and performing part-of-speech tagging on the vocabulary sequence W to generate a part-of-speech sequence P; The feature extraction and semantic understanding of the preprocessed natural language description includes converting the vocabulary sequence W into a word vector sequence V using a word frequency-inverse document frequency method, and then reducing the dimension of the word vector sequence V using a latent semantic analysis technique to obtain a representation vector U in a low-dimensional semantic space; The method of performing intention recognition based on the representation vector obtained by semantic understanding to obtain the user's operation intention includes using a naive Bayes classifier to classify the sentence representation vector U to obtain the user's operation intention I.

3. The browser element positioning method based on natural language processing according to claim 2 is characterized in that: The verification of the user's operation intention I is expressed as follows: Among them, P(C j |U) is the category C under the given input U j The posterior probability, P(U|C j ) is in category C j The likelihood of the input U, P(C j ) is Category C j The prior probability of , P(U) is the marginal probability of input U.

4. The browser element positioning method based on natural language processing according to claim 2 is characterized in that: Get the context window according to the vocabulary sequence W, where W = {w1,w2,…,w i ,w j ,w n }, each word w i The context window C i Contains a fixed number of words before and after k, and then according to the context window C i Construct the vocabulary co-occurrence matrix M, which is expressed as follows: Wherein, δ(x, y) represents the Kroneckerdelta function, when x=y, δ(x, y)=1, otherwise δ(x, y)=0; C k represents the context window; n represents the number of words; l represents the current context window, and the matrix element M ij Representation word w i and w j The number of occurrences in the same context window; Then the co-occurrence matrix M is converted into a word vector using the global vector method, which is expressed as follows: Among them, v i and v j Representation word w i and w j The word vector of b i and b j represents the bias term, ∈ ij represents the error term; Calculate the central word vector v of the specific domain vocabulary according to the preset scenario domain And perform semantic enhancement to obtain the enhanced semantic vector v′i. According to the enhanced semantic vector v′i, the word vector is weighted averaged to generate the sentence vector s, which is expressed as follows: Among them, tf-idf(w i ) represents word w i TF-IDF value of The support vector machine is then used to perform intent recognition and classification on the sentence representation vector s.

5. The browser element positioning method based on natural language processing according to claim 4 is characterized in that: The objective function of the support vector machine is expressed as follows: And satisfy the constraints, expressed as follows: y i (w·s i +b)≥1-ξ i ,x i ≥0 Where w and b represent the weight and bias of the support vector machine, ξi represents the slack variable, C represents the regularization parameter, and y i represents the classification label, m represents the number of samples, s i Represents the current sentence vector.

6. The browser element positioning method based on natural language processing according to claim 1, characterized in that: In S4, the objective function of the enhanced support vector machine is expressed as follows: And satisfy the constraints: y i (w·φ(x i )+b)≥1-ξ i ,x i ≥0 Where w and b represent the weight and bias of the enhanced support vector machine, ξi represents the slack variable, C represents the regularization parameter, and w i represents the importance weight of the click event, λ represents the feature interaction regularization coefficient, γ represents the L1 regularization coefficient, φ(x i ) represents the feature interaction function.

7. The browser element positioning method based on natural language processing according to claim 6 is characterized in that: During the training process, the sensitivity to the target click event is increased by combining the specific behavior patterns and context information in the preset scenario through the weighted enhanced support vector machine method, where w in the objective function i The calculation formula is as follows: w i =1+αf(x i )+βg(t i ) Among them, α and β are adjustment parameters, f(x i ) is a weighting function based on element features, g(t i ) is a weighting function based on time characteristics; According to the high-order interaction between features, the feature interaction function φ(x i ), which is expressed as follows: in, Represents the outer product operation of features.

8. The browser element positioning method based on natural language processing according to claim 7 is characterized in that: In the computational geometry method, the precise bounding box of the element is generated to enhance the element information. The bounding box calculation is expressed as follows: BoundingBox(x,y,width,height)=(x,y,x+width,y+height) Semantic enhancement vector calculation formula: v enhanced =v element +λ·v context Among them, λ is the semantic enhancement coefficient, v enhanced represents the enhanced semantic vector, v element represents an element vector, v context Represents a sentence vector.

9. The browser element positioning method based on natural language processing according to claim 1, characterized in that: The step of jointly representing the comprehensive feature vectors to obtain the final feature representation includes: The fused feature vector H is passed through the first fully connected layer and expressed as follows: F1=ReLU(W1H+b1) Among them, W1 represents the weight matrix of the first fully connected layer, b1 represents the bias term, and ReLU represents the activation function; Then pass F1 through the second fully connected layer, expressed as follows: F2=ReLU(W2F1+b2) Among them, W2 represents the weight matrix of the second fully connected layer, b2 represents the bias term, and ReLU represents the activation function; Finally, the final feature representation F is generated, which is expressed as follows: <h2 style=";text-align:left;direction:ltr">F = ReLU(W3F2+b3+λ∥F2∥<h2 style=";text-align:left;direction:ltr"> 2 <h2 style=";text-align:left;direction:ltr"> +γ∥W3∥) Among them, W3 represents the weight matrix of the third fully connected layer, b3 represents the bias term, ReLU represents the activation function, λ∥F2∥ 2 is the L2 regularization term, γ∥W3∥ represents the weight attenuation term; The target element location is obtained by using the target detection network to locate the target element in the webpage screenshot according to the final feature representation, which is expressed as follows: L = Faster R-CNN (S, F) Among them, L represents the location of the target element, Faster R-CNN represents the target detection network, and S represents the web page screenshot.

10. A browser element positioning system based on natural language processing, characterized in that: The system comprises: A natural language acquisition unit is used to acquire and preprocess the natural language description input by the user, perform feature extraction and semantic understanding on the preprocessed natural language description, and perform intent recognition based on the representation vector obtained by the semantic understanding to acquire the user's operation intention; A natural language analysis unit is used to analyze and semantically understand the preprocessed natural language description and the context window using natural language processing technology, convert the natural language description input by the user into a structured information representation, obtain the user's intention and classify it; The browser initialization unit is used to start the WebDriver to start the specified browser, and build a JavaScript script for locating web page elements, inject and execute the JavaScript script through the WebDriver interface, obtain the location information of the target element, verify whether the obtained XPath is correct, and output it in the console, and pass the obtained element information to the target element processing unit for processing; The target element processing unit is used to highlight the elements that meet the description according to the result of the browser initialization unit, capture the user click event through the browser event listener and classify the click event by using the enhanced support vector machine, filter out non-target click events, and enhance the accuracy of the click event by using context information and user behavior patterns, and further generate the precise bounding box of the element by using the computational geometry method to enhance the element information, and then let the user confirm whether the selected target element information is correct, and pass the target element information selected and confirmed by the user to the target element positioning unit for processing; The target element positioning unit is used to fuse the user's natural language description and web page screenshot as multimodal input data to obtain a natural language description word vector, then extract the feature vectors of the natural language description and the web page screenshot based on the natural language description word vector, and based on the attention mechanism, multimodally fuse the feature vectors of the natural language description and the web page screenshot to obtain a fused comprehensive feature vector, jointly represent the comprehensive feature vector to obtain a final feature representation, and use a target detection network to locate the target element of the web page screenshot based on the final feature representation to obtain the final target element location.

Images