Page element positioning method and device and electronic equipment

Abstract

The invention discloses a page element positioning method and device and electronic equipment, and the method comprises the steps: recognizing structural features, semantic features and visual features of each initial page element in a to-be-detected page, generating initial multi-dimensional features of each initial page element, and storing the initial multi-dimensional features in a preset feature library; if the initial page element needs to be positioned after the to-be-detected page is updated, identifying structural features, semantic features and visual features of each target page element of the updated to-be-detected page, and generating target multi-dimensional features of each target element; and matching the target multi-dimensional features of the target page elements with the initial multi-dimensional features of the initial page elements in the preset feature library, and determining the initial page elements corresponding to the target page elements according to a matching result.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision, and in particular to a page element positioning method and device and electronic equipment. BACKGROUND

[0002] With the version update of an application system, the position and structure of page elements can also change, which can cause the element positioning information obtained through design and debugging in the early stage to lose effectiveness. Therefore, how to efficiently cope with the dynamic changes of application element positioning has become a problem to be solved at present. SUMMARY

[0003] The present application provides a page element positioning method, device and electronic equipment to at least solve the above technical problems in the prior art.

[0004] In one aspect, the present application provides a page element positioning method, which comprises:

[0005] Identifying the structural features, semantic features and visual features of each initial page element in a to-be-detected page, generating initial multi-dimensional features of each initial page element and storing them in a preset feature library;

[0006] If the initial page elements need to be positioned after the to-be-detected page is updated, identifying the structural features, semantic features and visual features of each target page element of the updated to-be-detected page, and generating target multi-dimensional features of each target element;

[0007] Matching the target multi-dimensional features of the target page elements with the initial multi-dimensional features of the initial page elements in the preset feature library, and determining the initial page elements corresponding to the target page elements according to the matching results.

[0008] The structural features are obtained in the following manner:

[0009] Obtaining page structure information;

[0010] Recursively traversing the page structure information to generate path information of the page elements;

[0011] Performing a hash operation on the path information to generate the structural features.

[0012] The semantic features are obtained in the following manner:

[0013] Extracting text information of the page elements from the page structure information, the text information including visible text and / or supplementary text;

[0014] Performing semantic extension on the text information using a preset keyword library, and performing vector encoding on the extended text information to generate semantic features.

[0015] The visual feature is obtained by the following manner:

[0016] Obtaining a page image;

[0017] Cutting an element region image of a page element from the page image;

[0018] Converting the element region image into an HSV space, and extracting a hue channel of the HSV space;

[0019] Calculating a histogram of the hue channel, and converting the histogram into an array of a preset dimension to obtain the visual feature.

[0020] The matching of the target multi-dimensional feature of the target page element with the initial multi-dimensional feature of the initial page element in the preset feature library, and the determination of the corresponding initial page element of the target page element according to the matching result, comprises:

[0021] For the target page element, calculating a matching score of the target multi-dimensional feature with the initial multi-dimensional feature of the initial page element in each dimension feature in the preset feature library;

[0022] According to a preset weight of each dimension feature, performing a weighted summation on the matching scores in each dimension feature to obtain a total matching score;

[0023] If the total matching score is not less than a preset threshold, determining that the currently matched initial page element is the corresponding element of the target page element;

[0024] If the total matching score of the currently matched target page element with all the initial page elements is less than the preset threshold, determining that the positioning of the currently matched target page element fails.

[0025] The method further comprises:

[0026] For the matched target page element, according to the matching scores in each dimension feature and the preset weight of each dimension feature, calculating a contribution degree of each dimension feature;

[0027] According to the contribution degree of each dimension feature, using a weight adjustment algorithm to update the preset weight of each dimension feature.

[0028] The calculation of the matching score of the target multi-dimensional feature with the initial multi-dimensional feature of the initial page element in each dimension feature, comprises:

[0029] Performing a hash value comparison on the structural features of the target multi-dimensional feature and the initial multi-dimensional feature to obtain a matching score of the structural features;

[0030] perform vector similarity calculation on the semantic features of the target multi-dimensional features and the semantic features of the initial multi-dimensional features to obtain a matching score of the semantic features;

[0031] perform histogram similarity calculation on the visual features of the target multi-dimensional features and the visual features of the initial multi-dimensional features by using a similarity calculation function to obtain a matching score of the visual features.

[0032] The method further includes:

[0033] For the target page element for which the matching is successful, replace the corresponding initial multi-dimensional feature in the preset feature library with the target multi-dimensional feature thereof.

[0034] For the target page element for which the positioning fails, add the target multi-dimensional feature thereof as a new initial multi-dimensional feature to the preset feature library.

[0035] The second aspect of the present application provides a page element positioning device, which includes:

[0036] The first identification module is configured to identify the structural features, semantic features and visual features of each initial page element in the page to be detected, generate initial multi-dimensional features of each page element and store the initial multi-dimensional features in a preset feature library.

[0037] The second identification module is configured to identify the structural features, semantic features and visual features of each target page element in the updated page to be detected in response to detecting that the page to be detected is updated, and generate target multi-dimensional features of each target element.

[0038] The matching module is configured to match the target multi-dimensional features of the target page element with the initial multi-dimensional features of the initial page element in the preset feature library, and determine the corresponding initial page element of the target page element according to the matching result.

[0039] The third aspect of the present application provides an electronic device, which includes

[0040] a processor;

[0041] a memory for storing executable instructions of the processor;

[0042] The processor is configured to read the executable instructions from the memory and execute the instructions to implement the above-described page element positioning method.

[0043] The fourth aspect of the present application provides a computer readable storage medium, which stores a computer program for executing the above-described page element positioning method.

[0044] It should be understood that all the contents described in this part are not intended to identify key or important features of the embodiments of the present application, and are not used to limit the scope of the present application. Other features of the present application will become apparent through the following description. BRIEF DESCRIPTION OF DRAWINGS

[0045] Figure 1 A flow chart of a page element positioning method according to one embodiment of the present application is shown;

[0046] Figure 2 A flow chart of a page element positioning method according to another embodiment of the present application is shown;

[0047] Figure 3 A flow chart of a page element positioning method according to another embodiment of the present application is shown;

[0048] Figure 4 A flow chart of a page element positioning method according to another embodiment of the present application is shown;

[0049] Figure 5 A flow chart of a page element positioning method according to another embodiment of the present application is shown;

[0050] Figure 6 A flow chart of a page element positioning method according to another embodiment of the present application is shown;

[0051] Figure 7 A structural schematic diagram of a page element positioning apparatus according to one embodiment of the present application is shown. DETAILED DESCRIPTION

[0052] In order to make the purposes, features and advantages of the present application more obvious and easy to understand, the technical solutions in the embodiments of the present application will be described clearly and completely in combination with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0053] For various application systems such as Web (World Wide Web) applications, mobile applications, and enterprise information systems, accurate positioning of page elements is a basic prerequisite for implementing core functions such as automated testing, intelligent crawling, and user behavior analysis. Therefore, in order to ensure the smooth progress of application system tasks, it is necessary to accurately position the page elements. The current page element positioning scheme mainly requires developers to query the element identifiers of the page elements one by one and obtain the element positioning information of the page elements based on the element identifiers, but this approach has problems such as high maintenance cost and low efficiency. Although there are existing automatic positioning schemes based on single semantics or image features, the positioning accuracy is low due to insufficient feature expression capability. In order to improve the positioning efficiency and accuracy, the embodiments of the present application provide a page element positioning method as shown in Figure 1 The method comprises:

[0054] Step 101, identify the structural features, semantic features, and visual features of each initial page element in the page to be detected, generate the initial multi-dimensional features of each initial page element, and store them in a preset feature library.

[0055] The page to be detected refers to a terminal display page containing various page elements such as buttons, text, and images. In order to ensure the smooth progress of system tasks, accurate positioning of the page elements of the system is required. First, all page elements contained in the page to be detected, such as buttons, text, and images, are obtained, and then the structural features, semantic features, and visual features of all initial page elements of the obtained page to be detected are identified to generate the initial multi-dimensional features of each initial page element and save them to a preset feature library.

[0056] Structural features, semantic features, and visual features usually show the characteristics of page elements from different dimensions. The structural features can be considered as the layout, hierarchical relationship, nesting situation, and relative position of the page elements. The semantic features can be considered as the text content and label attributes of the page elements. The visual features can be considered as the appearance style, color matching, size, shape outline, and animation effect of the page elements. Structural features, semantic features, and visual features can usually be obtained based on existing multi-modal large models.

[0057] For the obtained structural features, semantic features, and visual features, they can be preprocessed according to actual needs, such as data cleaning, data normalization, etc. After preprocessing, the independent structural features, semantic features, and visual features after preprocessing can be directly stored as the initial multi-dimensional features of the initial page elements, or the preprocessed features can be fused based on a specific algorithm or model to generate a more comprehensive and comprehensive fused feature, and the fused feature is stored as the initial multi-dimensional feature of the initial page element.

[0058] Step 102, if the initial page element needs to be positioned after the update of the to-be-detected page, the structural features, semantic features and visual features of each target page element of the updated to-be-detected page are identified, and target multi-dimensional features of each target element are generated.

[0059] After the update of the to-be-detected page, the position and structure of the page element may change, so when the initial page element needs to be positioned, the target page element of the updated to-be-detected page needs to be re-identified, and the structural features, semantic features and visual features of each target page element are extracted to generate target multi-dimensional features of each target page element for subsequent matching with the initial page element in the preset feature library.

[0060] Wherein, the need to position the initial page element can be understood as that after the update of the to-be-detected page, any initial page element is detected to have position offset or structure reorganization, or after the update of the to-be-detected page, the system subsequent task (such as automatic testing, data crawling) needs to call the initial page element.

[0061] Step 103, matching the target multi-dimensional features of the target page element with the initial multi-dimensional features of the initial page element in the preset feature library, and determining the initial page element corresponding to the target page element according to the matching result.

[0062] For each target page element, its target multi-dimensional features are matched with the initial multi-dimensional features of the initial page element in the preset feature library. When the matching result shows matching, it is determined that the current target page element and the matched initial page element are the same page element. If the matching result shows no matching, the target page element is continuously matched with other initial page elements until each target page element is matched.

[0063] In the above scheme, by extracting and matching the multi-dimensional features of the structural features, semantic features and visual features of the page elements of the to-be-detected page before and after the update, the positioning of the page elements is performed, which avoids the traditional way of relying on manual positioning of the page elements, improves the positioning efficiency, and at the same time avoids the problem of inaccurate positioning caused by relying on a single feature for positioning of the page elements, and improves the accuracy of positioning of the page elements.

[0064] In an embodiment of the present application, as shown in Figure 2 The structural features of the initial page element and the target page element are obtained by the following method:

[0065] Step 201, obtaining page structure information.

[0066] First, the page structure information of the to-be-detected page is obtained.

[0067] The page structure information is used to show the organization manner and mutual relationship of all page elements in an HTML (HyperText Markup Language) document of the page, and describes the hierarchical nesting and arrangement order of various page elements (such as buttons, pictures, texts, etc.) in the page. The page structure information can be shown in the form of a DOM (Document Object Model) tree, and the DOM tree refers to a tree-shaped data structure, each node in the tree corresponds to an HTML element, that is, a page element, and the hierarchical relationship between the nodes represents the nesting relationship of the page elements in the page.

[0068] In step 202, the page structure information is recursively traversed to generate path information of the page elements.

[0069] The DOM tree is traversed by using a traversal algorithm, and for each page element, the path information thereof is obtained, where the traversal algorithm can be a DFS (Depth-First Search) algorithm.

[0070] In the process of obtaining the path information of the page elements, since some page elements have an ID (identifier) attribute and some page elements do not have the ID attribute, the way of obtaining the path information of the page elements can be divided into the following two kinds:

[0071] (1) For the page elements with the ID, an XPath (XML Path Language) expression is directly generated based on the ID to realize fast positioning by using the uniqueness of the ID. The XPath expression refers to the path information of the page element.

[0072] For example, the id of a page element tag div is header, that is, <div id＝"header">then can be directly based on <div id＝"header">The path information of the label div is generated as follows: @xpath= ' / / *[@id=' 'header' ']

[0073] (2) For a page element without an ID, its parent node is found by recursion, and the number of page elements with the same tag name in the parent node is counted to determine the index of the page element among the sibling elements. Then, the path elements of the page element are constructed based on the index, and the complete XPath expression is obtained by splicing all the path elements.

[0074] For example, a button element button is located at the third position among all sibling button elements in its parent node, and the index is 3. The path element button[3] of the button element button can be generated based on the index 3. After obtaining all the path elements of the button element button, the complete XPath expression is obtained by splicing the path elements, and the path information of the button element button is obtained.

[0075] In step 203, the path information is subjected to a hash operation to generate a structural feature.

[0076] In order to ensure the uniqueness and fixed length of the structural feature, after obtaining the XPath expression of the page element, the XPath expression is subjected to SHA-256 hash operation to generate a 64-bit hexadecimal hash value as the final structural feature.

[0077] In an embodiment of the present application, as shown in Figure 3 The semantic features of the initial page element and the target page element are obtained in the following manner:

[0078] In step 301, the text information of the page element is extracted from the page structure information, and the text information includes visible text and / or supplementary text.

[0079] Not all page elements have visible text. For page elements with visible text, the visible text can be directly extracted based on the DOM tree. The visible text refers to the text content directly displayed to the user on the page, such as the "submit" text on the button. The visible text can be extracted by accessing the textContent attribute or similar API of the page element. For example, in JavaScript, element.textContent can be used to obtain the visible text of the page element.

[0080] For page elements without visible text, in order to recognize their semantic features, text expansion is required to extract their supplementary text. Text expansion of page elements can be considered as supplementing the page elements with ARIA (Accessible Rich Internet Applications) attributes.

[0081] Specifically, in the DOM tree, ARIA attributes exist as attributes of element nodes. You can retrieve the supplementary text of a page element by accessing the getAttribute method of the corresponding page element. For example, in JavaScript, you can use element.getAttribute('aria-label') to retrieve the aria-label attribute value of a page element as the supplementary text of the page element.

[0082] Among them, for page elements with visible text, their supplementary text can also be extracted synchronously, so as to obtain the text information of the page element more comprehensively by combining the visible text and the supplementary text.

[0083] Step 302 : semantically expand the text information using a preset keyword library, and perform vector encoding on the expanded text information to generate semantic features.

[0084] To accommodate multilingual scenarios, a pre-built keyword library is also built for page elements. This pre-built keyword library predefines semantic tags (keywords) for page elements, covering both Chinese and English, as well as industry data. For example, for the login button, the pre-built keyword library contains multiple keywords for it, including "login," "signin," and "登录" (login).

[0085] Semantic expansion of text information can be thought of as matching the extracted text information with a preset keyword library and determining the matching degree of multiple matching keywords. For example, if the text is "login", which completely matches the keyword "login", the matching degree f_login of the keyword "login" is directly assigned to 1, that is, f_login = 1.0.

[0086] After obtaining the keywords for a page element, we perform vector encoding on features such as keyword matching and text length to generate an n-dimensional vector, which represents the semantic features of the page element. For example, the semantic features of a page element v can be expressed as: v = [f_login, f_signin, length], where f_login represents the matching degree of the text with the keyword "login", f_signin represents the matching degree of the text with the keyword "signin", and length represents the length of the text in characters.

[0087] In one embodiment of the present application, Figure 4 As shown, the visual features of the initial page element and the target page element are obtained in the following way:

[0088] Step 401, obtaining a page image.

[0089] The page image refers to the page image of the terminal where the page to be detected is located.

[0090] Step 402, cutting an element region image of a page element from the page image.

[0091] For each page element, its corresponding element region image is cut from the page image.

[0092] Step 403, converting the element region image into HSV space and extracting the hue channel of the HSV space.

[0093] The HSV (Hue, Saturation, Value) refers to a direct color representation method, the hue (H) represents the type of color (such as red, yellow, green), the saturation (Saturation, S) represents the purity of the color (0% is gray, 100% is pure color), and the value (Value, V) represents the brightness of the color (0% is black, 100% is the brightest).

[0094] For each page element, in order to facilitate color-based analysis, its element region image is also converted into HSV space which can separate color properties, so as to separate the hue, saturation and value channels of the image. The conversion of the HSV space can be performed through the OpenCV library.

[0095] Step 404, calculating the histogram of the hue channel and converting the histogram into an array of a preset dimension to obtain the visual feature.

[0096] After the element region image is converted into HSV space, the separated H channel is extracted, and a 256-bin histogram (such as 30% blue and 50% green) for statistical hue distribution is calculated through the OpenCV library.

[0097] Further, in order to realize the unification of the dimension, the histogram is also converted into an array of a preset dimension, and the array of the preset dimension represents the visual feature. The preset dimension can be configured according to actual conditions, and is usually configured as 1.

[0098] In an embodiment of the present application, as shown, the target multi-dimensional feature of the target page element is matched with the initial multi-dimensional feature of the initial page element in the preset feature library, and the initial page element corresponding to the target page element is determined according to the matching result, including: Figure 5

[0099] ​Step 501, for the target page element, calculate the matching score of each dimension feature between the target multi-dimensional feature and the initial multi-dimensional feature of the initial page element in the preset feature library.

[0100] For each target page element, match its structure feature, semantic feature and visual feature with the structure feature, semantic feature and visual feature of the initial page element respectively to obtain the matching scores of the structure feature, semantic feature and visual feature.

[0101] In an embodiment of the present application, the matching score of each dimension feature between the target multi-dimensional feature and the initial multi-dimensional feature of the initial page element in the preset feature library is calculated, including: performing hash value comparison on the structure feature of the target multi-dimensional feature and the structure feature of the initial multi-dimensional feature to obtain the matching score of the structure feature; performing vector similarity calculation on the semantic feature of the target multi-dimensional feature and the semantic feature of the initial multi-dimensional feature to obtain the matching score of the semantic feature; and performing histogram similarity calculation on the visual feature of the target multi-dimensional feature and the visual feature of the initial multi-dimensional feature by using a similarity calculation function to obtain the matching score of the visual feature.

[0102] Specifically, referring to Figures 3 to 5 For the description of the structure feature, semantic feature and visual feature acquisition process, the structure feature can be represented as a hash value, the semantic feature can be represented as a vector, and the visual feature can be represented as a histogram.

[0103] The matching score of the structure feature can be directly obtained based on the hash value comparison. For the hash values that are the same, the matching score is 1, and for the hash values that are different, the matching score is 0.

[0104] The vector similarity calculation of the semantic feature can be performed based on the following cosine similarity calculation formula:

[0105]

[0106] wherein, and respectively represent the vectors of the page elements, A i and B i respectively represent the components of the corresponding vectors, and respectively represent the lengths of the corresponding vectors.

[0107] For example, assuming that the vector A is [1, 0, 5] (text "Login") and the vector B is [0, 1, 7] (text "SignIn"), first calculate Then calculate and Finally, the parameters are brought into the cosine similarity calculation formula above, and the matching score of the vector is obtained 35 / (5.1x7.07)≈0.97.

[0108] The histogram similarity calculation of the visual feature is performed by using a similarity calculation function. The similarity calculation function can be a function (org.opencv.imgproc.Imgproc.compareHist()) in OpenCV for comparing the similarity of two histograms, i.e., a Pearson correlation coefficient calculation method:

[0109]

[0110] where x and y represent the corresponding histograms, x i and y i represent the i-th observation value of the corresponding histogram, and represent the mean value of the corresponding histogram.

[0111] In step 502, the matching scores on each dimension feature are weighted and summed according to the preset weight of each dimension feature, to obtain a total matching score.

[0112] Since the importance of structure, semantic and visual features in page element positioning is different, the preset weight of each dimension feature is configured based on the importance of each dimension feature. In order to maintain the standardization and calculability of the weight system, the sum of the weights of the three dimension features should be equal to 1 when configuring the preset weight.

[0113] After obtaining the matching scores of each dimension feature, the matching scores on each dimension feature are weighted and summed by using the preset weight of each dimension feature, to obtain a total matching score.

[0114] In step 503, if the total matching score is not less than a preset threshold, the initial page element of the current matching is determined as the corresponding element of the target page element.

[0115] The preset threshold is used to show the judgment basis of the matching success, and can be configured according to actual needs. For example, for a scene with high positioning accuracy requirement, the threshold can be set to a high value to ensure the matching accuracy, and for a scene pursuing matching efficiency, the threshold can be appropriately reduced to improve the processing speed.

[0116] When the total score of multi-dimensional feature matching reaches or exceeds the preset threshold, it is determined that the current initial page element and the target page element meet the positioning consistency requirement, and the initial page element of the current matching is determined as the corresponding element of the target page element.

[0117] Step 504, if the total matching score of the current matching target page element and all initial page elements is less than the preset threshold, it is determined that the positioning of the current matching target page element fails.

[0118] When the total matching score of the current target page element and all initial page elements is lower than the preset threshold, it indicates that there is no corresponding relationship between the target page element and the initial page element in the existing preset feature library that meets the positioning requirements, and it is determined that the positioning of the current matching target page element fails.

[0119] In an embodiment of the present application, with reference to Figure 6 For the matching successful target page element, a weight optimization mechanism is also configured, specifically including:

[0120] Step 601, for the matching successful target page element, the contribution degree of each dimension feature is calculated according to the matching score on the corresponding dimension feature and the preset dimension feature weight.

[0121] For the matching successful target page element, the contribution degree of each dimension feature is calculated based on the matching score on the corresponding dimension feature and the preset dimension feature weight. Specifically, the contribution degree of each dimension feature can be calculated by the following formula:

[0122]

[0123] Wherein, C i represents the contribution degree, S i represents the matching score of the i-th feature, W i is the current use weight of the i-th feature, and n is the total number of features.

[0124] Step 602, according to the contribution degree of each dimension feature, the preset dimension feature weight is updated by using the weight adjustment algorithm.

[0125] Based on the contribution degree of each dimension feature, the weight adjustment algorithm is used to update the dimension feature weight. Specifically, updating the preset dimension feature weight by using the weight adjustment algorithm can be regarded as updating the preset dimension feature weight by the following weight adjustment formula:

[0126] W i new = W i old ×(1-α)+C i ×α

[0127] Wherein, W i old represents the old weight of the i-th feature, α represents the learning rate, and C i is the contribution degree of the i-th feature.

[0128] In an embodiment of the present application, the preset feature library is further configured with an updating mechanism, specifically including:

[0129] For the target page element with a successful match, the corresponding initial multi-dimensional feature in the preset feature library is replaced with the target multi-dimensional feature thereof;

[0130] For the target page element with a failed positioning, the target multi-dimensional feature thereof is added to the preset feature library as a new initial multi-dimensional feature.

[0131] Specifically, for the target page element with a successful match, the corresponding initial multi-dimensional feature in the preset feature library is replaced with the target multi-dimensional feature thereof for subsequent positioning. For the target page element with a failed positioning, it can be a page element generated by redesign after updating, and the target multi-dimensional feature can be directly stored in the preset feature library as a new initial page element.

[0132] To implement the above-mentioned page element positioning method, as shown in Figure 7 The present application also provides a page element positioning device, which includes:

[0133] The first identification module 701 is configured to identify the structural feature, semantic feature and visual feature of each initial page element in the page to be detected, generate the initial multi-dimensional feature of each page element and store it in the preset feature library;

[0134] The second identification module 702 is configured to identify the structural feature, semantic feature and visual feature of each target page element in the updated page to be detected in response to detecting that the page to be detected is updated, and generate the target multi-dimensional feature of each target element;

[0135] The matching module 703 is configured to match the target multi-dimensional feature of the target page element with the initial multi-dimensional feature of the initial page element in the preset feature library, and determine the corresponding initial page element of the target page element according to the matching result.

[0136] In an embodiment of the present application, the matching module 703 includes:

[0137] The first calculation sub-module is configured to calculate the matching score of the target multi-dimensional feature of the target page element with the initial multi-dimensional feature of the initial page element in the preset feature library in each dimensional feature;

[0138] The second calculation sub-module is configured to weight and sum the matching scores in each dimensional feature according to the preset weight of each dimensional feature to obtain a total matching score;

[0139] The first determining sub-module is configured to determine that the initial page element in the current matching is a corresponding element of the target page element if the matching total score is not less than the preset threshold value.

[0140] The second determining sub-module is configured to determine that the positioning of the target page element in the current matching fails if the matching total score of the target page element in the current matching with all the initial page elements is less than the preset threshold value.

[0141] In an embodiment of the present application, the device further comprises:

[0142] The calculating module is configured to calculate the contribution degree of each dimension feature according to the matching score of each dimension feature of the target page element in the matching success and the preset dimension feature weight.

[0143] The updating module is configured to update the preset dimension feature weight by using a weight adjustment algorithm according to the contribution degree of each dimension feature.

[0144] In an embodiment of the present application, the first calculating sub-module comprises:

[0145] The comparison unit is configured to compare the hash values of the structural features of the target multi-dimensional feature and the structural features of the initial multi-dimensional feature to obtain the matching score of the structural features.

[0146] The first calculating unit is configured to calculate the vector similarity of the semantic features of the target multi-dimensional feature and the semantic features of the initial multi-dimensional feature to obtain the matching score of the semantic features.

[0147] The second calculating unit is configured to calculate the histogram similarity of the visual features of the target multi-dimensional feature and the visual features of the initial multi-dimensional feature by using a similarity calculation function to obtain the matching score of the visual features.

[0148] In an embodiment of the present application, the device further comprises:

[0149] The replacing module is configured to replace the corresponding initial multi-dimensional feature in the preset feature library with the target multi-dimensional feature of the target page element in the matching success.

[0150] The adding module is configured to add the target multi-dimensional feature of the target page element in the positioning failure as a new initial multi-dimensional feature to the preset feature library.

[0151] Exemplarily, the present application further provides an electronic device comprising:

[0152] a processor;

[0153] a memory for storing instructions executable by the processor;

[0154] The processor is configured to read the executable instructions from the memory and execute the instructions to implement the page element positioning method.

[0155] Exemplarily, the present application also provides a computer readable storage medium, which stores a computer program for executing the page element positioning method.

[0156] In addition to the method and the device described above, the embodiments of the present application can also be a computer program product, which includes computer program instructions, which, when executed by a processor, cause the processor to perform the steps of the methods according to various embodiments of the present application described in the above "Exemplary Methods” section of the present specification.

[0157] The computer program product can be written in any combination of one or more programming languages, including an object-oriented programming language such as Java, C++, etc., and conventional procedural programming languages, such as the "C” programming language or similar programming languages. The program code can execute entirely on the user's computing device, partly on the user's device, as a stand-alone software package, partly on the user's computing device and partly on a remote computing device or entirely on the remote computing device or server.

[0158] In addition, the embodiments of the present application can also be a computer readable storage medium, which stores computer program instructions, which, when executed by a processor, cause the processor to perform the steps of the methods according to various embodiments of the present application described in the above "Exemplary Methods” section of the present specification.

[0159] The computer readable storage medium can take any combination of one or more of the following readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can include, for example, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or apparatus, or any combination of the above. More specific examples (a non-exhaustive list) of the readable storage medium include an electrical connection having one or more wires, a portable disc, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0160] The above describes the basic principles of the present application in combination with specific embodiments, but it needs to be pointed out that the advantages, benefits, effects and the like mentioned in the present application are only examples and are not limiting, and these advantages, benefits, effects and the like cannot be considered as necessary for each embodiment of the present application. In addition, the above specific details disclosed are only for the purpose of example and understanding, and are not limiting, and the above details do not limit the present application to be necessarily implemented with the above specific details.

[0161] The block diagrams of the devices, apparatuses, equipment, systems involved in the present application are only illustrative examples and are not intended to require or imply the connection, arrangement, configuration shown in the block diagram. As those skilled in the art will recognize, these devices, apparatuses, equipment, systems can be connected, arranged, configured in any manner. Words such as "include", "contain", "have" and the like are open-ended words, mean "including but not limited to", and can be used interchangeably. The words "or" and "and" used herein mean "and / or", and can be used interchangeably unless the context clearly indicates otherwise. The word "such as" used herein means the phrase "such as but not limited to", and can be used interchangeably.

[0162] It also needs to be pointed out that in the devices, equipment and methods of the present application, each component or each step can be decomposed and / or recombined. These decompositions and / or recombinations should be considered as equivalent solutions of the present application.

[0163] The above description of the disclosed aspects is provided so that any person skilled in the art can make or use the present application. Various modifications to these aspects will be apparent to those skilled in the art, and the general principles defined herein can be applied to other aspects without departing from the scope of the present application. Therefore, the present application is not intended to be limited to the aspects shown herein, but is intended to be accorded the widest scope consistent with the principles and novel features disclosed herein.

[0164] The above description has been given for the purpose of illustration and description. Furthermore, this description is not intended to limit the embodiments of the present application to the forms disclosed herein. Although a number of example aspects and embodiments have been discussed above, those skilled in the art will recognize certain modifications, alterations, changes, additions and sub-combinations thereof.

Claims

1. A method for locating a page element, the method comprising: Identify the structural features, semantic features, and visual features of each initial page element in the page to be detected, generate initial multi-dimensional features of each initial page element, and store them in a preset feature library; If the initial page element needs to be located after the page to be detected is updated, the structural features, semantic features, and visual features of each target page element of the updated page to be detected are identified, and target multi-dimensional features of each target element are generated; The target multi-dimensional features of the target page element are matched with the initial multi-dimensional features of the initial page element in the preset feature library, and the initial page element corresponding to the target page element is determined according to the matching result.

2. The page element locating method according to claim 1, wherein the structural features are obtained by: Get page structure information; Recursively traverse the page structure information to generate the path information of the page elements; A hash operation is performed on the path information to generate a structural feature.

3. The page element locating method according to claim 1, wherein the semantic features are obtained by: Extracting text information of page elements from the page structure information, wherein the text information includes visible text and / or supplementary text; The text information is semantically expanded using a preset keyword library, and the expanded text information is vector-encoded to generate semantic features.

4. The page element locating method according to claim 1, wherein the visual features are obtained by: Get the page image; Cut out the element area image of the page element from the page image; Convert the element area image into HSV space and extract the hue channel of the HSV space; Calculate the histogram of the hue channel and convert the histogram into an array of preset dimensions to obtain visual features.

5. The page element locating method according to claim 1, wherein the step of matching the target multidimensional features of the target page element with the initial multidimensional features of the initial page element in the preset feature library, and determining the initial page element corresponding to the target page element based on the matching results, comprises: For the target page element, calculating the matching score of its target multidimensional feature with the initial multidimensional feature of the initial page element in the preset feature library in each dimensional feature; According to the preset weights of each dimension feature, the matching scores of each dimension feature are weighted and summed to obtain the total matching score; If the total matching score is not less than a preset threshold, determining that the currently matched initial page element is the corresponding element of the target page element; If the total matching scores of the current matching target page element and all the initial page elements are less than a preset threshold, it is determined that the positioning of the current matching target page element has failed.

6. The page element locating method according to claim 5, further comprising: For the successfully matched target page elements, the contribution of each dimensional feature is calculated based on the matching score of each dimensional feature and the preset weight of each dimensional feature; According to the contribution of each dimensional feature, the weight adjustment algorithm is used to update the preset weights of each dimensional feature.

7. The page element location method according to claim 5, wherein the step of calculating a matching score between the target multidimensional feature and the initial multidimensional feature of the initial page element in the preset feature library in each dimensional feature comprises: Compare the hash values ​​of the structural features of the target multidimensional features with those of the initial multidimensional features to obtain a matching score of the structural features; Calculate the vector similarity between the semantic features of the target multidimensional features and the semantic features of the initial multidimensional features to obtain the matching score of the semantic features; The similarity calculation function is used to calculate the histogram similarity between the visual features of the target multidimensional features and the visual features of the initial multidimensional features to obtain the matching score of the visual features.

8. The page element locating method according to claim 5, further comprising: For the successfully matched target page element, the corresponding initial multidimensional feature in the preset feature library is replaced with its target multidimensional feature; For the target page element that fails to be located, its target multidimensional feature is added to the preset feature library as a new initial multidimensional feature.

9. A device for locating page elements, the device comprising: The first recognition module is used to identify the structural features, semantic features and visual features of each initial page element in the page to be detected, generate initial multi-dimensional features of each page element and store them in a preset feature library; a second recognition module for, in response to detecting that the page to be detected is updated, identifying the structural features, semantic features, and visual features of each target page element of the updated page to be detected, and generating a target multi-dimensional feature of each target element; The matching module is used to match the target multidimensional features of the target page element with the initial multidimensional features of the initial page element in the preset feature library, and determine the initial page element corresponding to the target page element according to the matching result.

10. An electronic device comprising processor; a memory for storing instructions executable by the processor; The processor is configured to read the executable instructions from the memory and execute the instructions to implement the page element locating method according to any one of claims 1 to 8.

Images